Unsuccessful attempt

README.md

@@ -9,24 +9,13 @@ tags:
 ---
 
 > [!TIP]
-> **Credits:**
-> 
-> Made with love by [**@Lewdiculous**](https://huggingface.co/Lewdiculous) with the handy contributions by [**@SolidSnacke**](https://huggingface.co/SolidSnacke). <br>
-> If this proves useful for you, feel free to credit and share the repository and authors.
+> **Credits:** <br>
+> Made with love by [**@Lewdiculous**](https://huggingface.co/Lewdiculous). <br>
+> *If this proves useful for you, feel free to credit and share the repository and authors.*
 
 > [!WARNING]
-> **[Important] Llama-3:**
-> 
-> For those converting LLama-3 BPE models, you might need have to read [**llama.cpp/#6920**](https://github.com/ggerganov/llama.cpp/pull/6920#issue-2265280504) for more context. <br>
-> Try and if you have issues try the tips bwllow.
->
-> Basically, make sure you're in the latest llama.cpp repo commit, then run the new `convert-hf-to-gguf-update.py` script inside the repo (you will need to provide a huggingface-read-token, and you need to have access to the Meta-Llama-3 repositories – [here](https://huggingface.co/meta-llama/Meta-Llama-3-8B) and [here](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) – to be sure, so fill the access request forms right away to be able to fetch the necessary files, you also might need to refresh the tokens if it stops working after some time), afterwards you need to manually copy the config files from `llama.cpp\models\tokenizers\llama-bpe` into your downloaded **model** folder, replacing the existing ones. <br>
-> Try again and the conversion procress should work as expected.
-
-> [!WARNING]
-> **Experimental:** <br>
-> There is a new experimental script added, `gguf-imat-lossless-for-BF16.py`, which performs the conversions directly from a BF16 GGUF to hopefully generate lossless, or as close to that for now, Llama-3 model quantizations avoiding the recent talked about issues on that topic, it is more resource intensive and will generate more writes in the drive as there's a whole additional conversion step that isn't performed in the previous version. This should only be necessary until we have GPU support for BF16 to run directly without conversion.
-
+> **Warning:** <br>
+> For **Llama-3** models that don't follow the ChatML, Alpaca, Vicuna and other conventional formats, at the moment, you have to use `gguf-imat-llama-3.py` and replace the config files with the ones in the [**ChaoticNeutrals/Llama3-Corrections**](https://huggingface.co/ChaoticNeutrals/Llama3-Corrections/tree/main) repository to properly quant and generate the imatrix data.
 
 Pull Requests with your own features and improvements to this script are always welcome.
 
@@ -34,7 +23,7 @@ Pull Requests with your own features and improvements to this script are always
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65ddabb9bbffb280f4b45d8e/vwlPdqxrSdILCHM24n_M2.png)
 
-Simple python script (`gguf-imat.py` - I recommend using the specific "for-FP16" or "for-BF16" scripts) to generate various GGUF-IQ-Imatrix quantizations from a Hugging Face `author/model` input, for Windows and NVIDIA hardware.
+Simple python script (`gguf-imat.py`) to generate various GGUF-IQ-Imatrix quantizations from a Hugging Face `author/model` input, for Windows and NVIDIA hardware.
 
 This is setup for a Windows machine with 8GB of VRAM, assuming use with an NVIDIA GPU. If you want to change the `-ngl` (number of GPU layers) amount, you can do so at [**line 124**](https://huggingface.co/FantasiaFoundry/GGUF-Quantization-Script/blob/main/gguf-imat.py#L124). This is only relevant during the `--imatrix` data generation. If you don't have enough VRAM you can decrease the `-ngl` amount or set it to 0 to only use your System RAM instead for all layers, this will make the imatrix data generation take longer, so it's a good idea to find the number that gives your own machine the best results.
 
@@ -52,7 +41,6 @@ Adjust `quantization_options` in [**line 138**](https://huggingface.co/FantasiaF
 - 32GB of system RAM.
 
 **Software Requirements:**
-- Windows 10/11
 - Git
 - Python 3.11
   - `pip install huggingface_hub`

gguf-imat-for-FP16.py → gguf-imat-llama-3.py

@@ -70,7 +70,6 @@ def download_cudart_if_necessary(latest_release_tag):
 def download_model_repo():
     base_dir = os.path.dirname(os.path.abspath(__file__))
     models_dir = os.path.join(base_dir, "models")
-
     if not os.path.exists(models_dir):
         os.makedirs(models_dir)
 
@@ -78,30 +77,30 @@ def download_model_repo():
     model_name = model_id.split("/")[-1]
     model_dir = os.path.join(models_dir, model_name)
 
-    gguf_dir = os.path.join(base_dir, "models", f"{model_name}-GGUF")
-    gguf_model_path = os.path.join(gguf_dir, f"{model_name}-F16.gguf")
-    imatrix_file_name = input("Enter the name of the imatrix.txt file (default: imatrix.txt): ").strip() or "imatrix.txt"
-    delete_model_dir = input("Remove HF model folder after converting original model to GGUF? (yes/no) (default: no): ").strip().lower()
+    if os.path.exists(model_dir):
+        print("Model repository already exists. Using existing repository.")
 
-    if os.path.exists(gguf_model_path):
-        create_imatrix(base_dir, gguf_dir, gguf_model_path, model_name, imatrix_file_name)
-    else: 
-        if os.path.exists(model_dir):
-            print("Model repository already exists. Using existing repository.")  
+        delete_model_dir = input("Remove HF model folder after converting original model to GGUF? (yes/no) (default: no): ").strip().lower()
 
-            convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
+        imatrix_file_name = input("Enter the name of the imatrix.txt file (default: imatrix.txt): ").strip() or "imatrix.txt"
 
-        else:
-            revision = input("Enter the revision (branch, tag, or commit) to download (default: main): ") or "main"
+        convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
+
+    else:
+        revision = input("Enter the revision (branch, tag, or commit) to download (default: main): ") or "main"
+
+        delete_model_dir = input("Remove HF model folder after converting original model to GGUF? (yes/no) (default: no): ").strip().lower()
+
+        print("Downloading model repository...")
+        snapshot_download(repo_id=model_id, local_dir=model_dir, revision=revision)
+        print("Model repository downloaded successfully.")
 
-            print("Downloading model repository...")
-            snapshot_download(repo_id=model_id, local_dir=model_dir, revision=revision)
-            print("Model repository downloaded successfully.")
+        imatrix_file_name = input("Enter the name of the imatrix.txt file (default: imatrix.txt): ").strip() or "imatrix.txt"
 
-            convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
+        convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
 
 def convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name):
-    convert_script = os.path.join(base_dir, "llama.cpp", "convert_hf_to_gguf.py")
+    convert_script = os.path.join(base_dir, "llama.cpp", "convert.py")
     gguf_dir = os.path.join(base_dir, "models", f"{model_name}-GGUF")
     gguf_model_path = os.path.join(gguf_dir, f"{model_name}-F16.gguf")
 
@@ -109,19 +108,15 @@ def convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir,
         os.makedirs(gguf_dir)
 
     if not os.path.exists(gguf_model_path):
-        subprocess.run(["python", convert_script, model_dir, "--outfile", gguf_model_path, "--outtype", "f16"])
+        subprocess.run(["python", convert_script, model_dir, "--outfile", gguf_model_path, "--outtype", "f16", "--vocab-type", "bpe"])
 
         if delete_model_dir == 'yes' or delete_model_dir == 'y':
             shutil.rmtree(model_dir)
             print(f"Original model directory '{model_dir}' deleted.")
         else:
             print(f"Original model directory '{model_dir}' was not deleted. You can remove it manually.")
-        
-    
-    create_imatrix(base_dir, gguf_dir, gguf_model_path, model_name, imatrix_file_name)
 
-def create_imatrix(base_dir, gguf_dir, gguf_model_path, model_name, imatrix_file_name):
-    imatrix_exe = os.path.join(base_dir, "bin", "llama-imatrix.exe")
+    imatrix_exe = os.path.join(base_dir, "bin", "imatrix.exe")
     imatrix_output_src = os.path.join(gguf_dir, "imatrix.dat")
     imatrix_output_dst = os.path.join(gguf_dir, "imatrix.dat")
     if not os.path.exists(imatrix_output_dst):
@@ -142,7 +137,7 @@ def quantize_models(base_dir, model_name):
 
     quantization_options = [
         "IQ3_M", "IQ3_XXS",
-        "Q4_K_M", "Q4_K_S", "IQ4_XS",
+        "Q4_K_M", "Q4_K_S", "IQ4_NL", "IQ4_XS",
         "Q5_K_M", "Q5_K_S",
         "Q6_K",
         "Q8_0"
@@ -151,7 +146,7 @@ def quantize_models(base_dir, model_name):
     for quant_option in quantization_options:
         quantized_gguf_name = f"{model_name}-{quant_option}-imat.gguf"
         quantized_gguf_path = os.path.join(gguf_dir, quantized_gguf_name)
-        quantize_command = os.path.join(base_dir, "bin", "llama-quantize.exe")
+        quantize_command = os.path.join(base_dir, "bin", "quantize.exe")
         imatrix_path = os.path.join(gguf_dir, "imatrix.dat")
 
         subprocess.run([quantize_command, "--imatrix", imatrix_path, 
@@ -166,4 +161,4 @@ def main():
     print("Finished preparing resources.")
 
 if __name__ == "__main__":
-    main()
+    main()

gguf-imat-lossless-for-BF16.py

@@ -1,181 +0,0 @@
-import os
-import requests
-import zipfile
-import subprocess
-import shutil
-from huggingface_hub import snapshot_download
-
-def clone_or_update_llama_cpp():
-    print("Preparing...")
-    base_dir = os.path.dirname(os.path.abspath(__file__))
-    os.chdir(base_dir)
-    if not os.path.exists("llama.cpp"):
-        subprocess.run(["git", "clone", "--depth", "1", "https://github.com/ggerganov/llama.cpp"])
-    else:
-        os.chdir("llama.cpp")
-        subprocess.run(["git", "pull"])
-    os.chdir(base_dir)
-    print("The 'llama.cpp' repository is ready.")
-
-def download_llama_release():
-    base_dir = os.path.dirname(os.path.abspath(__file__))
-    dl_dir = os.path.join(base_dir, "bin", "dl")
-    if not os.path.exists(dl_dir):
-        os.makedirs(dl_dir)
-
-    os.chdir(dl_dir)
-    latest_release_url = "https://github.com/ggerganov/llama.cpp/releases/latest"
-    response = requests.get(latest_release_url)
-    if response.status_code == 200:
-        latest_release_tag = response.url.split("/")[-1]
-        download_url = f"https://github.com/ggerganov/llama.cpp/releases/download/{latest_release_tag}/llama-{latest_release_tag}-bin-win-cuda-cu12.2.0-x64.zip"
-        response = requests.get(download_url)
-        if response.status_code == 200:
-            with open(f"llama-{latest_release_tag}-bin-win-cuda-cu12.2.0-x64.zip", "wb") as f:
-                f.write(response.content)
-            with zipfile.ZipFile(f"llama-{latest_release_tag}-bin-win-cuda-cu12.2.0-x64.zip", "r") as zip_ref:
-                zip_ref.extractall(os.path.join(base_dir, "bin"))
-            print("Downloading latest 'llama.cpp' prebuilt Windows binaries...")
-            print("Download and extraction completed successfully.")
-            return latest_release_tag
-        else:
-            print("Failed to download the release file.")
-    else:
-        print("Failed to fetch the latest release information.")
-
-def download_cudart_if_necessary(latest_release_tag):
-    base_dir = os.path.dirname(os.path.abspath(__file__))
-    cudart_dl_dir = os.path.join(base_dir, "bin", "dl")
-    if not os.path.exists(cudart_dl_dir):
-        os.makedirs(cudart_dl_dir)
-
-    cudart_zip_file = os.path.join(cudart_dl_dir, "cudart-llama-bin-win-cu12.2.0-x64.zip")
-    cudart_extracted_files = ["cublas64_12.dll", "cublasLt64_12.dll", "cudart64_12.dll"]
-
-    if all(os.path.exists(os.path.join(base_dir, "bin", file)) for file in cudart_extracted_files):
-        print("Cuda resources already exist. Skipping download.")
-    else:
-        cudart_download_url = f"https://github.com/ggerganov/llama.cpp/releases/download/{latest_release_tag}/cudart-llama-bin-win-cu12.2.0-x64.zip"
-        response = requests.get(cudart_download_url)
-        if response.status_code == 200:
-            with open(cudart_zip_file, "wb") as f:
-                f.write(response.content)
-            with zipfile.ZipFile(cudart_zip_file, "r") as zip_ref:
-                zip_ref.extractall(os.path.join(base_dir, "bin"))
-            print("Preparing 'cuda' resources...")
-            print("Download and extraction of cudart completed successfully.")
-        else:
-            print("Failed to download the cudart release file.")
-
-def download_model_repo():
-    base_dir = os.path.dirname(os.path.abspath(__file__))
-    models_dir = os.path.join(base_dir, "models")
-
-    if not os.path.exists(models_dir):
-        os.makedirs(models_dir)
-
-    model_id = input("Enter the model ID to download (e.g., huggingface/transformers): ")
-    model_name = model_id.split("/")[-1]
-    model_dir = os.path.join(models_dir, model_name)
-
-    gguf_dir = os.path.join(base_dir, "models", f"{model_name}-GGUF")
-    gguf_model_path = os.path.join(gguf_dir, f"{model_name}-F16.gguf")
-    imatrix_file_name = input("Enter the name of the imatrix.txt file (default: imatrix.txt): ").strip() or "imatrix.txt"
-    delete_model_dir = input("Remove HF model folder after converting original model to GGUF? (yes/no) (default: no): ").strip().lower()
-
-    if os.path.exists(gguf_model_path):
-        create_imatrix(base_dir, gguf_dir, gguf_model_path, model_name, imatrix_file_name)
-    else: 
-        if os.path.exists(model_dir):
-            print("Model repository already exists. Using existing repository.")  
-
-            convert_model_to_gguf_bf16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
-
-        else:
-            revision = input("Enter the revision (branch, tag, or commit) to download (default: main): ") or "main"
-
-            print("Downloading model repository...")
-            snapshot_download(repo_id=model_id, local_dir=model_dir, revision=revision)
-            print("Model repository downloaded successfully.")
-
-            convert_model_to_gguf_bf16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
-
-def convert_model_to_gguf_bf16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name):
-    convert_script = os.path.join(base_dir, "llama.cpp", "convert_hf_to_gguf.py")
-    gguf_dir = os.path.join(base_dir, "models", f"{model_name}-GGUF")
-    gguf_model_path = os.path.join(gguf_dir, f"{model_name}-BF16.gguf")
-
-    if not os.path.exists(gguf_dir):
-        os.makedirs(gguf_dir)
-
-    if not os.path.exists(gguf_model_path):
-        subprocess.run(["python", convert_script, model_dir, "--outfile", gguf_model_path, "--outtype", "bf16"])
-
-    convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
-
-def convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name):
-    convert_script = os.path.join(base_dir, "llama.cpp", "convert_hf_to_gguf.py")
-    gguf_dir = os.path.join(base_dir, "models", f"{model_name}-GGUF")
-    gguf_model_path = os.path.join(gguf_dir, f"{model_name}-F16.gguf")
-
-    if not os.path.exists(gguf_dir):
-        os.makedirs(gguf_dir)
-
-    if not os.path.exists(gguf_model_path):
-        subprocess.run(["python", convert_script, model_dir, "--outfile", gguf_model_path, "--outtype", "f16"])
-
-        if delete_model_dir == 'yes' or delete_model_dir == 'y':
-            shutil.rmtree(model_dir)
-            print(f"Original model directory '{model_dir}' deleted.")
-        else:
-            print(f"Original model directory '{model_dir}' was not deleted. You can remove it manually.")
-        
-    create_imatrix(base_dir, gguf_dir, gguf_model_path, model_name, imatrix_file_name)
-
-def create_imatrix(base_dir, gguf_dir, gguf_model_path, model_name, imatrix_file_name):
-    imatrix_exe = os.path.join(base_dir, "bin", "llama-imatrix.exe")
-    imatrix_output_src = os.path.join(gguf_dir, "imatrix.dat")
-    imatrix_output_dst = os.path.join(gguf_dir, "imatrix.dat")
-    if not os.path.exists(imatrix_output_dst):
-        try:
-            subprocess.run([imatrix_exe, "-m", gguf_model_path, "-f", os.path.join(base_dir, "imatrix", imatrix_file_name), "-ngl", "7"], cwd=gguf_dir)
-            shutil.move(imatrix_output_src, imatrix_output_dst)
-            print("imatrix.dat moved successfully.")
-        except Exception as e:
-            print("Error occurred while moving imatrix.dat:", e)
-    else:
-        print("imatrix.dat already exists in the GGUF folder.")
-
-    quantize_models(base_dir, model_name)
-
-def quantize_models(base_dir, model_name):
-    gguf_dir = os.path.join(base_dir, "models", f"{model_name}-GGUF")
-    bf16_gguf_path = os.path.join(gguf_dir, f"{model_name}-BF16.gguf")
-
-    quantization_options = [
-        "IQ3_M", "IQ3_XXS",
-        "Q4_K_M", "Q4_K_S", "IQ4_XS",
-        "Q5_K_M", "Q5_K_S",
-        "Q6_K",
-        "Q8_0"
-    ]
-
-    for quant_option in quantization_options:
-        quantized_gguf_name = f"{model_name}-{quant_option}-imat.gguf"
-        quantized_gguf_path = os.path.join(gguf_dir, quantized_gguf_name)
-        quantize_command = os.path.join(base_dir, "bin", "llama-quantize.exe")
-        imatrix_path = os.path.join(gguf_dir, "imatrix.dat")
-
-        subprocess.run([quantize_command, "--imatrix", imatrix_path, 
-                        bf16_gguf_path, quantized_gguf_path, quant_option], cwd=gguf_dir)
-        print(f"Model quantized with {quant_option} option.")
-
-def main():
-    clone_or_update_llama_cpp()
-    latest_release_tag = download_llama_release()
-    download_cudart_if_necessary(latest_release_tag)
-    download_model_repo()
-    print("Finished preparing resources.")
-
-if __name__ == "__main__":
-    main()

gguf-imat.py

@@ -70,7 +70,6 @@ def download_cudart_if_necessary(latest_release_tag):
 def download_model_repo():
     base_dir = os.path.dirname(os.path.abspath(__file__))
     models_dir = os.path.join(base_dir, "models")
-
     if not os.path.exists(models_dir):
         os.makedirs(models_dir)
 
@@ -78,27 +77,27 @@ def download_model_repo():
     model_name = model_id.split("/")[-1]
     model_dir = os.path.join(models_dir, model_name)
 
-    gguf_dir = os.path.join(base_dir, "models", f"{model_name}-GGUF")
-    gguf_model_path = os.path.join(gguf_dir, f"{model_name}-F16.gguf")
-    imatrix_file_name = input("Enter the name of the imatrix.txt file (default: imatrix.txt): ").strip() or "imatrix.txt"
-    delete_model_dir = input("Remove HF model folder after converting original model to GGUF? (yes/no) (default: no): ").strip().lower()
+    if os.path.exists(model_dir):
+        print("Model repository already exists. Using existing repository.")
 
-    if os.path.exists(gguf_model_path):
-        create_imatrix(base_dir, gguf_dir, gguf_model_path, model_name, imatrix_file_name)
-    else: 
-        if os.path.exists(model_dir):
-            print("Model repository already exists. Using existing repository.")  
+        delete_model_dir = input("Remove HF model folder after converting original model to GGUF? (yes/no) (default: no): ").strip().lower()
 
-            convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
+        imatrix_file_name = input("Enter the name of the imatrix.txt file (default: imatrix.txt): ").strip() or "imatrix.txt"
 
-        else:
-            revision = input("Enter the revision (branch, tag, or commit) to download (default: main): ") or "main"
+        convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
+
+    else:
+        revision = input("Enter the revision (branch, tag, or commit) to download (default: main): ") or "main"
+
+        delete_model_dir = input("Remove HF model folder after converting original model to GGUF? (yes/no) (default: no): ").strip().lower()
+
+        print("Downloading model repository...")
+        snapshot_download(repo_id=model_id, local_dir=model_dir, revision=revision)
+        print("Model repository downloaded successfully.")
 
-            print("Downloading model repository...")
-            snapshot_download(repo_id=model_id, local_dir=model_dir, revision=revision)
-            print("Model repository downloaded successfully.")
+        imatrix_file_name = input("Enter the name of the imatrix.txt file (default: imatrix.txt): ").strip() or "imatrix.txt"
 
-            convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
+        convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name)
 
 def convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir, imatrix_file_name):
     convert_script = os.path.join(base_dir, "llama.cpp", "convert.py")
@@ -116,17 +115,13 @@ def convert_model_to_gguf_f16(base_dir, model_dir, model_name, delete_model_dir,
             print(f"Original model directory '{model_dir}' deleted.")
         else:
             print(f"Original model directory '{model_dir}' was not deleted. You can remove it manually.")
-        
-    
-    create_imatrix(base_dir, gguf_dir, gguf_model_path, model_name, imatrix_file_name)
 
-def create_imatrix(base_dir, gguf_dir, gguf_model_path, model_name, imatrix_file_name):
     imatrix_exe = os.path.join(base_dir, "bin", "imatrix.exe")
     imatrix_output_src = os.path.join(gguf_dir, "imatrix.dat")
     imatrix_output_dst = os.path.join(gguf_dir, "imatrix.dat")
     if not os.path.exists(imatrix_output_dst):
         try:
-            subprocess.run([imatrix_exe, "-m", gguf_model_path, "-f", os.path.join(base_dir, "imatrix", imatrix_file_name), "-ngl", "7"], cwd=gguf_dir)
+            subprocess.run([imatrix_exe, "-m", gguf_model_path, "-f", os.path.join(base_dir, "imatrix", imatrix_file_name), "-ngl", "8"], cwd=gguf_dir)
             shutil.move(imatrix_output_src, imatrix_output_dst)
             print("imatrix.dat moved successfully.")
         except Exception as e:

requirements.txt

@@ -1,4 +0,0 @@
--r ./llama.cpp/requirements/requirements-convert-legacy-llama.txt
--r ./llama.cpp/requirements/requirements-convert-hf-to-gguf.txt
--r ./llama.cpp/requirements/requirements-convert-hf-to-gguf-update.txt
--r ./llama.cpp/requirements/requirements-convert-llama-ggml-to-gguf.txt