Repository: ggml-org/llama.cpp
Pull request #12511: quantize: Handle user-defined quantization levels for additional tensors
Author: login: EAddario
Labels: name: examples
Created: 2025-03-22T08:21:22Z | Merged: 2025-04-13T18:29:28Z

This PR adds the ability to quantize other tensors, beyond token-embedding and output-tensor. It handles most of the supported architectures. ~~except Mamba, RWKV6, RWKV6QWEN2 and T5 to avoid having too many command options, but can add as well if maintainers request it.~~

For full background on the PR, please see: [Squeezing Tensor Bits: the quest for smaller LLMs](https://medium.com/@eaddario/squeezing-tensor-bits-the-quest-for-smaller-llms-86b23bd052ca)


Diff:
diff --git a/examples/quantize/quantize.cpp b/examples/quantize/quantize.cpp
index a4468b16987..0355311dc5c 100644
--- a/examples/quantize/quantize.cpp
+++ b/examples/quantize/quantize.cpp
@@ -9,6 +9,7 @@
 #include <fstream>
 #include <cmath>
 #include <cctype>
+#include <algorithm>
 
 struct quant_option {
     std::string name;
@@ -16,7 +17,7 @@ struct quant_option {
     std::string desc;
 };
 
-static const std::vector<struct quant_option> QUANT_OPTIONS = {
+static const std::vector<quant_option> QUANT_OPTIONS = {
     { "Q4_0",     LLAMA_FTYPE_MOSTLY_Q4_0,     " 4.34G, +0.4685 ppl @ Llama-3-8B",  },
     { "Q4_1",     LLAMA_FTYPE_MOSTLY_Q4_1,     " 4.78G, +0.4511 ppl @ Llama-3-8B",  },
     { "Q5_0",     LLAMA_FTYPE_MOSTLY_Q5_0,     " 5.21G, +0.1316 ppl @ Llama-3-8B",  },
@@ -105,7 +106,8 @@ static bool try_parse_ftype(const std::string & ftype_str_in, llama_ftype & ftyp
 //
 [[noreturn]]
 static void usage(const char * executable) {
-    printf("usage: %s [--help] [--allow-requantize] [--leave-output-tensor] [--pure] [--imatrix] [--include-weights] [--exclude-weights] [--output-tensor-type] [--token-embedding-type] [--override-kv] model-f32.gguf [model-quant.gguf] type [nthreads]\n\n", executable);
+    printf("usage: %s [--help] [--allow-requantize] [--leave-output-tensor] [--pure] [--imatrix] [--include-weights] [--exclude-weights] [--output-tensor-type]\n", executable);
+    printf("       [--token-embedding-type] [--tensor-type] [--keep-split] [--override-kv] model-f32.gguf [model-quant.gguf] type [nthreads]\n\n");
     printf("  --allow-requantize: Allows requantizing tensors that have already been quantized. Warning: This can severely reduce quality compared to quantizing from 16bit or 32bit\n");
     printf("  --leave-output-tensor: Will leave output.weight un(re)quantized. Increases model size but may also increase quality, especially when requantizing\n");
     printf("  --pure: Disable k-quant mixtures and quantize all tensors to the same type\n");
@@ -114,6 +116,8 @@ static void usage(const char * executable) {
     printf("  --exclude-weights tensor_name: use importance matrix for this/these tensor(s)\n");
     printf("  --output-tensor-type ggml_type: use this ggml_type for the output.weight tensor\n");
     printf("  --token-embedding-type ggml_type: use this ggml_type for the token embeddings tensor\n");
+    printf("  --tensor-type TENSOR=TYPE: quantize this tensor to this ggml_type. example: --tensor-type attn_q=q8_0\n");
+    printf("      Advanced option to selectively quantize tensors. May be specified multiple times.\n");
     printf("  --keep-split: will generate quantized model in the same shards as input\n");
     printf("  --override-kv KEY=TYPE:VALUE\n");
     printf("      Advanced option to override model metadata by key in the quantized model. May be specified multiple times.\n");
@@ -244,6 +248,107 @@ static ggml_type parse_ggml_type(const char * arg) {
     return GGML_TYPE_COUNT;
 }
 
+// Allowed tensors for arbitrary quantization with --tensor-type option
+static const std::vector<std::string> ALLOWED_TENSOR_TYPE = {
+    "attn_k",
+    "attn_kv_a_mqa",
+    "attn_kv_b",
+    "attn_o",
+    "attn_output",
+    "attn_q",
+    "attn_q_a",
+    "attn_q_b",
+    "attn_qkv",
+    "attn_v",
+    "channel_mix_key",
+    "channel_mix_receptance",
+    "channel_mix_value",
+    "cls",
+    "cls.output",
+    "cross_attn_k",
+    "cross_attn_o",
+    "cross_attn_q",
+    "cross_attn_v",
+    "ffn_act",
+    "ffn_down",
+    "ffn_down_exps",
+    "ffn_down_shexp",
+    "ffn_gate",
+    "ffn_gate_exps",
+    "ffn_gate_shexp",
+    "ffn_up",
+    "ffn_up_exps",
+    "ffn_up_shexp",
+    "ssm_in",
+    "ssm_out",
+    "time_mix_gate",
+    "time_mix_key",
+    "time_mix_output",
+    "time_mix_receptance",
+    "time_mix_value",
+};
+
+// changes to this struct must be replicated in llama-quant.cpp
+struct tensor_quantization {
+    std::string name;
+    ggml_type quant = GGML_TYPE_COUNT;
+};
+
+static bool parse_tensor_type(const char * data, std::vector<tensor_quantization> & tensor_type) {
+    const char * sep = strchr(data, '=');
+    if (sep == nullptr) {
+        printf("\n%s: malformed tensor type '%s'\n\n", __func__, data);
+        return false;
+    }
+
+    const size_t tn_len = sep - data;
+    if (tn_len == 0) {
+        printf("\n%s: missing tensor name\n\n", __func__);
+        return false;
+    }
+
+    if (const size_t qt_len = strlen(sep); qt_len == 1) {
+        printf("\n%s: missing quantization type\n\n", __func__);
+        return false;
+    }
+
+    std::string tn(data, tn_len);
+    std::transform(tn.begin(), tn.end(), tn.begin(), tolower);
+    sep++;
+    const std::string qt(sep);
+
+    bool found = false;
+    for (const auto & allowed : ALLOWED_TENSOR_TYPE) {
+        std::string tensor;
+        tensor = tn.rfind('.') != std::string::npos ? tn.substr(tn.rfind('.') + 1) : tn;
+        // handle special case of cls.output
+        std::string cls_output = "cls.output";
+        if (tn.find(cls_output) != std::string::npos) {
+            tensor = "cls.output";
+        }
+        // check if an allowed tensor exists and it's at the end of the kv string
+        if (tensor == allowed) {
+            found = true;
+            break;
+        }
+    }
+    if (!found) {
+        printf("\n%s: invalid tensor name '%s'\n\n", __func__, tn.c_str());
+        return false;
+    }
+
+    if (parse_ggml_type(qt.c_str()) == GGML_TYPE_COUNT) {
+        printf("\n%s: invalid quantization type '%s'\n\n", __func__, qt.c_str());
+        return false;
+    }
+
+    tensor_quantization tqz;
+    tqz.name = tn;
+    tqz.quant = parse_ggml_type(qt.c_str());
+    tensor_type.emplace_back(std::move(tqz));
+    return true;
+}
+
 int main(int argc, char ** argv) {
     if (argc < 3) {
         usage(argv[0]);
@@ -255,6 +360,7 @@ int main(int argc, char ** argv) {
     std::string imatrix_file;
     std::vector<std::string> included_weights, excluded_weights;
     std::vector<llama_model_kv_override> kv_overrides;
+    std::vector<tensor_quantization> tensor_types;
 
     for (; arg_idx < argc && strncmp(argv[arg_idx], "--", 2) == 0; arg_idx++) {
         if (strcmp(argv[arg_idx], "--leave-output-tensor") == 0) {
@@ -277,6 +383,10 @@ int main(int argc, char ** argv) {
             } else {
                 usage(argv[0]);
             }
+        } else if (strcmp(argv[arg_idx], "--tensor-type") == 0) {
+            if (arg_idx == argc-1 || !parse_tensor_type(argv[++arg_idx], tensor_types)) {
+                usage(argv[0]);
+            }
         } else if (strcmp(argv[arg_idx], "--override-kv") == 0) {
             if (arg_idx == argc-1 || !string_parse_kv_override(argv[++arg_idx], kv_overrides)) {
                 usage(argv[0]);
@@ -361,6 +471,9 @@ int main(int argc, char ** argv) {
         kv_overrides.back().key[0] = 0;
         params.kv_overrides = &kv_overrides;
     }
+    if (!tensor_types.empty()) {
+        params.tensor_types = &tensor_types;
+    }
 
     llama_backend_init();
 
diff --git a/include/llama.h b/include/llama.h
index fca2b034ba2..c102525b17b 100644
--- a/include/llama.h
+++ b/include/llama.h
@@ -366,17 +366,18 @@ extern "C" {
 
     // model quantization parameters
     typedef struct llama_model_quantize_params {
-        int32_t nthread;                     // number of threads to use for quantizing, if <=0 will use std::thread::hardware_concurrency()
-        enum llama_ftype ftype;              // quantize to this llama_ftype
-        enum ggml_type output_tensor_type;   // output tensor type
-        enum ggml_type token_embedding_type; // token embeddings tensor type
-        bool allow_requantize;               // allow quantizing non-f32/f16 tensors
-        bool quantize_output_tensor;         // quantize output.weight
-        bool only_copy;                      // only copy tensors - ftype, allow_requantize and quantize_output_tensor are ignored
-        bool pure;                           // quantize all tensors to the default type
-        bool keep_split;                     // quantize to the same number of shards
-        void * imatrix;                      // pointer to importance matrix data
-        void * kv_overrides;                 // pointer to vector containing overrides
+        int32_t nthread;                      // number of threads to use for quantizing, if <=0 will use std::thread::hardware_concurrency()
+        enum llama_ftype ftype;               // quantize to this llama_ftype
+        enum ggml_type output_tensor_type;    // output tensor type
+        enum ggml_type token_embedding_type;  // token embeddings tensor type
+        bool allow_requantize;                // allow quantizing non-f32/f16 tensors
+        bool quantize_output_tensor;          // quantize output.weight
+        bool only_copy;                       // only copy tensors - ftype, allow_requantize and quantize_output_tensor are ignored
+        bool pure;                            // quantize all tensors to the default type
+        bool keep_split;                      // quantize to the same number of shards
+        void * imatrix;                       // pointer to importance matrix data
+        void * kv_overrides;                  // pointer to vector containing overrides
+        void * tensor_types;                  // pointer to vector containing tensor types
     } llama_model_quantize_params;
 
     typedef struct llama_logit_bias {
diff --git a/src/llama-quant.cpp b/src/llama-quant.cpp
index e3e10fa6cf7..7dc54227631 100644
--- a/src/llama-quant.cpp
+++ b/src/llama-quant.cpp
@@ -10,6 +10,7 @@
 #include <cinttypes>
 #include <fstream>
 #include <mutex>
+#include <regex>
 #include <thread>
 #include <unordered_map>
 
@@ -47,8 +48,14 @@ struct quantize_state_impl {
         {}
 };
 
+// changes to this struct must be replicated in quantize.cpp
+struct tensor_quantization {
+    std::string name;
+    ggml_type quant = GGML_TYPE_COUNT;
+};
+
 static void llama_tensor_dequantize_impl(
-    struct ggml_tensor * tensor, std::vector<no_init<float>> & output, std::vector<std::thread> & workers,
+    ggml_tensor * tensor, std::vector<no_init<float>> & output, std::vector<std::thread> & workers,
     const size_t nelements, const int nthread
 ) {
     if (output.size() < nelements) {
@@ -536,7 +543,7 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
     model.load_hparams(ml);
     model.load_stats  (ml);
 
-    struct quantize_state_impl qs(model, params);
+    quantize_state_impl qs(model, params);
 
     if (params->only_copy) {
         ftype = ml.ftype;
@@ -661,7 +668,7 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
     // populate the original tensors so we get an initial meta data
     for (const auto * it : tensors) {
         uint16_t i_split = params->keep_split ? it->idx : 0;
-        struct ggml_tensor * tensor = it->tensor;
+        ggml_tensor * tensor = it->tensor;
         if (!ctx_outs[i_split]) {
             ctx_outs[i_split].reset(gguf_init_empty());
         }
@@ -710,7 +717,7 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
     new_ofstream(0);
     for (const auto * it : tensors) {
         const auto & weight = *it;
-        struct ggml_tensor * tensor = weight.tensor;
+        ggml_tensor * tensor = weight.tensor;
         if (weight.idx != cur_split && params->keep_split) {
             close_ofstream();
             new_ofstream(weight.idx);
@@ -776,7 +783,7 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
         // do not quantize relative position bias (T5)
         quantize &= name.find("attn_rel_b.weight") == std::string::npos;
 
-        enum ggml_type new_type;
+        ggml_type new_type;
         void * new_data;
         size_t new_size;
 
@@ -786,6 +793,19 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
             // get more optimal quantization type based on the tensor shape, layer, etc.
             if (!params->pure && ggml_is_quantized(default_type)) {
                 new_type = llama_tensor_get_type(qs, new_type, tensor, ftype);
+                // unless the user specifies a type
+                if (params->tensor_types) {
+                    const std::vector<tensor_quantization> & tensor_types = *static_cast<const std::vector<tensor_quantization> *>(params->tensor_types);
+                    for (const auto & [tname, qtype] : tensor_types) {
+                        if (std::regex pattern(tname); std::regex_search(tensor->name, pattern)) {
+                            if (qtype != new_type) {
+                                LLAMA_LOG_DEBUG("(overriding %s -> %s), ", ggml_type_name(new_type), ggml_type_name(qtype));
+                            }
+                            new_type = qtype;
+                            break;
+                        }
+                    }
+                }
             }
             if (params->token_embedding_type < GGML_TYPE_COUNT && strcmp(tensor->name, "token_embd.weight") == 0) {
                 new_type = params->token_embedding_type;
@@ -910,8 +930,8 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
 // interface implementation
 //
 
-struct llama_model_quantize_params llama_model_quantize_default_params() {
-    struct llama_model_quantize_params result = {
+llama_model_quantize_params llama_model_quantize_default_params() {
+    llama_model_quantize_params result = {
         /*.nthread                     =*/ 0,
         /*.ftype                       =*/ LLAMA_FTYPE_MOSTLY_Q5_1,
         /*.output_tensor_type          =*/ GGML_TYPE_COUNT,
@@ -923,6 +943,7 @@ struct llama_model_quantize_params llama_model_quantize_default_params() {
         /*.keep_split                  =*/ false,
         /*.imatrix                     =*/ nullptr,
         /*.kv_overrides                =*/ nullptr,
+        /*.tensor_type                 =*/ nullptr,
     };
 
     return result;


Reviews:
user: login: ngxson
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T14:12:46Z
user: login: ddh0
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T02:39:52Z
user: login: EAddario
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T07:33:23Z
user: login: joseph777111
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-05T03:38:50Z
user: login: ggerganov
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-08T07:16:02Z
user: login: EAddario
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T21:00:43Z
user: login: slaren
body: This is a bit too hacky for my preference, but I suppose if people are already creating custom mixes by modifying the code it is better to at least have a tool to do it.

I would prefer if the allowed tensor check is removed, it doesn't really work as a reliable check, and it will prevent some legitimate uses.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-11T11:16:19Z

Review comments:
diff_hunk: @@ -244,6 +247,103 @@ static ggml_type parse_ggml_type(const char * arg) {
     return GGML_TYPE_COUNT;
 }
 
+// Allowed tensors for arbitrary quantization with --tensor-type option
+static const std::vector<std::string> ALLOWED_TENSOR_TYPE = {
path: examples/quantize/quantize.cpp
user: login: ngxson
body: I'm rethinking about this, maybe we can simplify this functionality by adding just 2 flags:
- `--dump-mapping` to get the list of tensors and the target quantized type. User can then modify the target quant directly
- `--mapping FILE` user can then specify a custom mapping file given from step above
created_at: 2025-04-02T14:12:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -244,6 +247,103 @@ static ggml_type parse_ggml_type(const char * arg) {
     return GGML_TYPE_COUNT;
 }
 
+// Allowed tensors for arbitrary quantization with --tensor-type option
+static const std::vector<std::string> ALLOWED_TENSOR_TYPE = {
path: examples/quantize/quantize.cpp
user: login: ddh0
body: I think it makes sense to only allow certain tensors to be quantized, otherwise users will lobotomize their model and then complain that llama.cpp is broken
created_at: 2025-04-03T02:39:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -244,6 +247,103 @@ static ggml_type parse_ggml_type(const char * arg) {
     return GGML_TYPE_COUNT;
 }
 
+// Allowed tensors for arbitrary quantization with --tensor-type option
+static const std::vector<std::string> ALLOWED_TENSOR_TYPE = {
path: examples/quantize/quantize.cpp
user: login: EAddario
body: Agree with @ddh0 although I can see how down the line something similar to what @ngxson is suggesting may be useful: I'm testing the layer-wise quant using the modified llama-imatrix for guidance, and whilst I'm getting some really encouraging results (I'll publish the full model in my [HF repo](https://huggingface.co/eaddario) over the weekend), the process is overly manual and the regexes can get unwieldy (e.g. `--tensor-type "(1[3-9]|2[0-9]|30)\.attn_v=q6_k" --tensor-type "([0-9]|[1-2][0-9]|30|31)\.ffn_down=q3_k" --tensor-type "(10|1[3-9]|2[0-9]|30)\.attn_q=q5_k" ...`

I think it would be nice to have a way to AutoMagically generate optimum regexes to be fed into llama-quantize!
created_at: 2025-04-03T07:33:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -244,6 +247,103 @@ static ggml_type parse_ggml_type(const char * arg) {
     return GGML_TYPE_COUNT;
 }
 
+// Allowed tensors for arbitrary quantization with --tensor-type option
+static const std::vector<std::string> ALLOWED_TENSOR_TYPE = {
path: examples/quantize/quantize.cpp
user: login: joseph777111
body: > I think it makes sense to only allow certain tensors to be quantized, otherwise users will lobotomize their model and then complain that llama.cpp is broken

This is the case of full granular-control vs guided hand-holding to competence. And, we can have the best of both worlds. What we will need is a brief, informative How-To Guide, which introduces and explains the concepts of per-tensor and per-layer quantization, and then gives concrete examples, which they can base their quantization decisions on. Adding to this: the guide should further educate the User on which tensors/weights are targets for quantization (Embeddings, ATTN_K, ATTN_Q, ATTN_V, ATTN_Output, FFN_Down, FFN_Gate, FFN_Up, and, Output) and which are more likely not to be good targets for quantization (FFN_NORM, etc)  and why. And, then for the more dense of mankind: a brief disclaimer stating that any and all modifications that they make to their custom quantized model is their business and responsibility and thereby waiving ggml-org or any of you guys from liability. 🤔 

Just because some choose not to read and learn, doesn't mean we should have to suffer a loss of "power-user" features, because those who aren't paying attention will lobotomize their quantized models. This is all a fun game of trial-error and experimentation. If the users have made it this far, they will have to learn. 

created_at: 2025-04-05T03:38:50Z
author_association: NONE
diff_hunk: @@ -366,17 +366,18 @@ extern "C" {
 
     // model quantization parameters
     typedef struct llama_model_quantize_params {
-        int32_t nthread;                     // number of threads to use for quantizing, if <=0 will use std::thread::hardware_concurrency()
-        enum llama_ftype ftype;              // quantize to this llama_ftype
-        enum ggml_type output_tensor_type;   // output tensor type
-        enum ggml_type token_embedding_type; // token embeddings tensor type
-        bool allow_requantize;               // allow quantizing non-f32/f16 tensors
-        bool quantize_output_tensor;         // quantize output.weight
-        bool only_copy;                      // only copy tensors - ftype, allow_requantize and quantize_output_tensor are ignored
-        bool pure;                           // quantize all tensors to the default type
-        bool keep_split;                     // quantize to the same number of shards
-        void * imatrix;                      // pointer to importance matrix data
-        void * kv_overrides;                 // pointer to vector containing overrides
+        int32_t nthread;                      // number of threads to use for quantizing, if <=0 will use std::thread::hardware_concurrency()
+        enum llama_ftype ftype;               // quantize to this llama_ftype
+        enum ggml_type output_tensor_type;    // output tensor type
+        enum ggml_type token_embedding_type;  // token embeddings tensor type
+        bool allow_requantize;                // allow quantizing non-f32/f16 tensors
+        bool quantize_output_tensor;          // quantize output.weight
+        bool only_copy;                       // only copy tensors - ftype, allow_requantize and quantize_output_tensor are ignored
+        bool pure;                            // quantize all tensors to the default type
+        bool keep_split;                      // quantize to the same number of shards
+        void * imatrix;                       // pointer to importance matrix data
+        void * kv_overrides;                  // pointer to vector containing overrides
+        void * tensor_types;                  // pointer to vector containing tensor types
     } llama_model_quantize_params;
path: include/llama.h
user: login: ggerganov
body: This changes the public interface, so add a comment in https://github.com/ggml-org/llama.cpp/issues/9289.

Note that passing C++ objects here is not correct and we eventually have to fix this API to not do that. It hasn't become a problem yet because the quantization functions are likely not used frequently by 3rd party applications.

@EAddario If you are interested, you can give it a shot in another PR and fix these structs to become C compatible.
created_at: 2025-04-08T07:14:03Z
author_association: MEMBER
diff_hunk: @@ -786,6 +793,19 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
             // get more optimal quantization type based on the tensor shape, layer, etc.
             if (!params->pure && ggml_is_quantized(default_type)) {
                 new_type = llama_tensor_get_type(qs, new_type, tensor, ftype);
+                // unless the user specifies a type
+                if (params->tensor_types) {
+                    const std::vector<tensor_quantization> & tensor_types = *static_cast<const std::vector<tensor_quantization> *>(params->tensor_types);
+                    for (const auto & [tname, qtype] : tensor_types) {
+                        if (std::regex pattern(tname); std::regex_search(tensor->name, pattern)) {
+                            if (qtype != new_type) {
+                                LLAMA_LOG_DEBUG("(overriding %s -> %s), ", ggml_type_name(new_type), ggml_type_name(qtype));
path: src/llama-quant.cpp
user: login: ggerganov
body: ~Think these logs would be more readable if printed on separate lines:~

Nevermind.


created_at: 2025-04-08T07:15:38Z
author_association: MEMBER
diff_hunk: @@ -366,17 +366,18 @@ extern "C" {
 
     // model quantization parameters
     typedef struct llama_model_quantize_params {
-        int32_t nthread;                     // number of threads to use for quantizing, if <=0 will use std::thread::hardware_concurrency()
-        enum llama_ftype ftype;              // quantize to this llama_ftype
-        enum ggml_type output_tensor_type;   // output tensor type
-        enum ggml_type token_embedding_type; // token embeddings tensor type
-        bool allow_requantize;               // allow quantizing non-f32/f16 tensors
-        bool quantize_output_tensor;         // quantize output.weight
-        bool only_copy;                      // only copy tensors - ftype, allow_requantize and quantize_output_tensor are ignored
-        bool pure;                           // quantize all tensors to the default type
-        bool keep_split;                     // quantize to the same number of shards
-        void * imatrix;                      // pointer to importance matrix data
-        void * kv_overrides;                 // pointer to vector containing overrides
+        int32_t nthread;                      // number of threads to use for quantizing, if <=0 will use std::thread::hardware_concurrency()
+        enum llama_ftype ftype;               // quantize to this llama_ftype
+        enum ggml_type output_tensor_type;    // output tensor type
+        enum ggml_type token_embedding_type;  // token embeddings tensor type
+        bool allow_requantize;                // allow quantizing non-f32/f16 tensors
+        bool quantize_output_tensor;          // quantize output.weight
+        bool only_copy;                       // only copy tensors - ftype, allow_requantize and quantize_output_tensor are ignored
+        bool pure;                            // quantize all tensors to the default type
+        bool keep_split;                      // quantize to the same number of shards
+        void * imatrix;                       // pointer to importance matrix data
+        void * kv_overrides;                  // pointer to vector containing overrides
+        void * tensor_types;                  // pointer to vector containing tensor types
     } llama_model_quantize_params;
path: include/llama.h
user: login: EAddario
body: Thanks @ggerganov, happy to
created_at: 2025-04-08T21:00:43Z
author_association: CONTRIBUTOR

PR comments:
user: login: max-krasnyansky
created_at: 2025-03-25T03:19:30Z
body: How about we add a more generic `--tensor-type tensor_name_pattern=type`. 
@slaren has the PR #11397 that overrides backend mapping per tensor. 
Let's make this one similar (same patters, etc). That way we'll be able to override specific layers (if needed). 
author_association: MEMBER
user: login: EAddario
created_at: 2025-03-25T09:24:05Z
body: That's an excellent idea! and it'll allow to add all supported tensor types (50+) without creating a mess of parameters. Plus, it will give me something to do over the weekend 😆
author_association: CONTRIBUTOR
user: login: jukofyork
created_at: 2025-03-27T13:05:51Z
body: > How about we add a more generic `--tensor-type tensor_name_pattern=type`. @slaren has the PR #11397 that overrides backend mapping per tensor. Let's make this one similar (same patters, etc). That way we'll be able to override specific layers (if needed).

Yeah, I think this is definitely the way to go - the regex support of that PR gives really good flexibility.
author_association: COLLABORATOR
user: login: EAddario
created_at: 2025-03-29T14:30:20Z
body: Adding the regex matching would be a relatively straightforward change, and happy to do it, but wanted to check how useful the feature would really be.

@slaren's use case is about choosing which tensors and/or which layers get processed where. For example, the user could opt to run all expert tensors in CPU and the shared experts in the GPU, like so: `-ot "_exp=CPU" -ot "_sexp=GPU"`, or like in the PR's example, to keep experts of layers 20-99 in the CPU: `-ot "[2-9][0-9]\.ffn_.*_exps\.=CPU"`

In my use case, I had assumed the user would want to choose which tensors get quantized at which level, excluding the ones already handled by the program (--leave-output-tensor, --output-tensor-type, --token-embedding-type and --pure), with said quantization applied to the selected tensor/s across all layers.

The changes I have just pushed implement exactly that, but with @max-krasnyansky suggestion. A couple of things to note:
- Implemented the --tensor-type logic very similar to --override-kv to keep the code consistent
- _ALLOWED_TENSOR_TYPE_ in `quantize.cpp` controls which tensors can be selected. I have excluded 1D tensors like norm, gamma, lerp, etc. and others that are usually too small to benefit from quantizing, but happy to change if I got it wrong
- Rather than adding to `llama.h`, I'm duplicating the _tensor_quantization_  in `quantize.cpp` and `llama-quant.cpp`. I think is better than polluting the main header file with such a trivial and program-specific struct, but can change if that's not the recommended approach

Having said that, @jukofyork raises an interesting possibility reminiscing of Razvan-Gabriel Dumitru et al [Layer-Wise Quantization](https://arxiv.org/abs/2406.17415) and Binrui Zeng et al  [Layer-Specific Adaptive Quantization](https://arxiv.org/abs/2412.18135v1). I think is worth exploring further, but suspect the amount of testing to ensure nothing gets broken will be significant. Either way, one for another weekend 😁

In the meantime, will keep 🤞 for this PR to be merged.


author_association: CONTRIBUTOR
user: login: ddh0
created_at: 2025-03-30T04:15:31Z
body: This is super neat! I hope it gets merged in
author_association: CONTRIBUTOR
user: login: ddh0
created_at: 2025-03-30T04:23:36Z
body: Unfortunely the build failed for me on macOS Metal:

```
In file included from /Users/dylan/Documents/AI/llama.cpp/tests/test-c.c:1:
/Users/dylan/Documents/AI/llama.cpp/src/../include/llama.h:360:9: error: must use 'enum' tag to refer to type 'llama_ftype'
  360 |         llama_ftype ftype;                    // quantize to this llama_ftype
      |         ^
      |         enum 
/Users/dylan/Documents/AI/llama.cpp/src/../include/llama.h:361:9: error: must use 'enum' tag to refer to type 'ggml_type'
  361 |         ggml_type output_tensor_type;         // output tensor type
      |         ^
      |         enum 
/Users/dylan/Documents/AI/llama.cpp/src/../include/llama.h:362:9: error: must use 'enum' tag to refer to type 'ggml_type'
  362 |         ggml_type token_embedding_type;       // token embeddings tensor type
      |         ^
      |         enum 
3 errors generated.
make[2]: *** [tests/CMakeFiles/test-c.dir/test-c.c.o] Error 1
make[1]: *** [tests/CMakeFiles/test-c.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
```

The command I used to build was `cmake -B build; cmake --build build --config Release -j`, which is how I always build llama.cpp.

---

**EDIT 1:** It looks like lines 360 - 362 in llama.h are just missing enums, they should be changed to:

```cpp
        enum llama_ftype ftype;               // quantize to this llama_ftype
        enum ggml_type output_tensor_type;    // output tensor type
        enum ggml_type token_embedding_type;  // token embeddings tensor type
```

---

**EDIT 2:** Yep, successfully built after the above changes! 🎉 Do you think we still need the `--output-tensor-type` and `--token-embedding-type` CLI options? Currently they are still present in the --help output.
author_association: CONTRIBUTOR
user: login: EAddario
created_at: 2025-03-30T07:00:55Z
body: My bad. Thanks for flagging the bug @ddh0!

As for the cli flags, I would keep them as they are. I'm sure there are automated pipelines out there that rely on those options. 
author_association: CONTRIBUTOR
user: login: joseph777111
created_at: 2025-03-30T16:40:51Z
body: @ggerganov can you look at this, please? Having this PR merged for per tensor quantization would be awesome. 😋 
author_association: NONE
user: login: ddh0
created_at: 2025-03-30T18:40:46Z
body: In the article, you say:

> 1. embed_tokens: refers to the embedding layer, essentially a large lookup table. It stores a pre-learned vector representation (an “embedding”) for each word or token in the model’s vocabulary. The embedding layer has a relatively large number of parameters, especially with a large vocabulary, representing a significant portion of the overall model size. However, each embedding vector represents a distinct token, and even with some quantization error, the relative distances between embeddings are often preserved well enough. **The embedding layer is more about lookup than complex computation; hence more robust to quantization errors.**

and also

> 7. lm_head: is the final, crucial component, that transforms the model’s internal representations into probabilities for the next word in a sequence. Its function is to predict the next word. Although seemingly simple (it is only one linear layer), **it’s extremely sensitive to quantization**. This layer maps the final hidden state to the vocabulary logits. Small changes in these logits can drastically alter the probability distribution over the vocabulary, leading to incorrect word predictions.

(emphasis mine)

However I notice that in your preferred quantization schemes ([this table](https://miro.medium.com/v2/resize:fit:1400/format:webp/1*X4HwtBP9-ytKKZPGfEN3uA.png)) you never have the output tensor at anything higher than Q3_K, and in fact it's always the same as the token embeddings.

I understand that these are just your personal preferences and I'm glad they're working well for you. I'm just curious what the reason is for the discrepancy between the explanations of embedding and output tensors and the values you use in practice. Thanks! @EAddario 
author_association: CONTRIBUTOR
user: login: ddh0
created_at: 2025-03-31T03:00:45Z
body: Sorry for so many messages here, but the build also fails for me on Linux CUDA:

```
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp: In function ‘size_t llama_tensor_quantize_impl(ggml_type, const float*, void*, int64_t, int64_t, int64_t, const float*, std::vector<std::thread>&, int)’:
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:426:10: error: ‘mutex’ is not a member of ‘std’
  426 |     std::mutex mutex;
      |          ^~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:12:1: note: ‘std::mutex’ is defined in header ‘<mutex>’; did you forget to ‘#include <mutex>’?
   11 | #include <thread>
  +++ |+#include <mutex>
   12 | #include <unordered_map>
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:430:22: error: ‘mutex’ was not declared in this scope
  430 |     auto compute = [&mutex, &counter, &new_size, &valid, new_type, f32_data, new_data, chunk_size,
      |                      ^~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp: In lambda function:
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:435:18: error: ‘unique_lock’ is not a member of ‘std’
  435 |             std::unique_lock<std::mutex> lock(mutex);
      |                  ^~~~~~~~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:435:18: note: ‘std::unique_lock’ is defined in header ‘<mutex>’; did you forget to ‘#include <mutex>’?
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:435:35: error: ‘mutex’ is not a member of ‘std’
  435 |             std::unique_lock<std::mutex> lock(mutex);
      |                                   ^~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:435:35: note: ‘std::mutex’ is defined in header ‘<mutex>’; did you forget to ‘#include <mutex>’?
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:435:47: error: ‘mutex’ is not captured
  435 |             std::unique_lock<std::mutex> lock(mutex);
      |                                               ^~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:431:38: note: the lambda has no capture-default
  431 |             nrows, n_per_row, imatrix]() {
      |                                      ^
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:430:22: note: ‘<typeprefixerror>mutex’ declared here
  430 |     auto compute = [&mutex, &counter, &new_size, &valid, new_type, f32_data, new_data, chunk_size,
      |                      ^~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:435:42: error: ‘lock’ was not declared in this scope; did you mean ‘clock’?
  435 |             std::unique_lock<std::mutex> lock(mutex);
      |                                          ^~~~
      |                                          clock
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:452:22: error: ‘unique_lock’ is not a member of ‘std’
  452 |                 std::unique_lock<std::mutex> lock(mutex);
      |                      ^~~~~~~~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:452:22: note: ‘std::unique_lock’ is defined in header ‘<mutex>’; did you forget to ‘#include <mutex>’?
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:452:39: error: ‘mutex’ is not a member of ‘std’
  452 |                 std::unique_lock<std::mutex> lock(mutex);
      |                                       ^~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:452:39: note: ‘std::mutex’ is defined in header ‘<mutex>’; did you forget to ‘#include <mutex>’?
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:452:51: error: ‘mutex’ is not captured
  452 |                 std::unique_lock<std::mutex> lock(mutex);
      |                                                   ^~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:431:38: note: the lambda has no capture-default
  431 |             nrows, n_per_row, imatrix]() {
      |                                      ^
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:430:22: note: ‘<typeprefixerror>mutex’ declared here
  430 |     auto compute = [&mutex, &counter, &new_size, &valid, new_type, f32_data, new_data, chunk_size,
      |                      ^~~~~
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp: In function ‘void llama_model_quantize_impl(const std::string&, const std::string&, const llama_model_quantize_params*)’:
/home/dylan/Documents/AI/llama.cpp/src/llama-quant.cpp:557:31: error: ‘isfinite’ is not a member of ‘std’
  557 |                     if (!std::isfinite(f)) {
      |                               ^~~~~~~~
gmake[2]: *** [src/CMakeFiles/llama.dir/build.make:303: src/CMakeFiles/llama.dir/llama-quant.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:2273: src/CMakeFiles/llama.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2
```

Again using the same build command I normally use, in this case `cmake -B build -DGGML_CUDA=ON; cmake --build build --config Release -j`. I'm trying to build https://github.com/EAddario/llama.cpp/tree/quantize as of commit `b3c7db572c44867746249ecb49527ae9b49f24f7 - Fix implied type bug`.
author_association: CONTRIBUTOR
user: login: EAddario
created_at: 2025-03-31T07:24:36Z
body: No need to be sorry @ddh0, quite the contrary! **Thank you** for testing on a different platform (something I should have done before pushing 😖). At risk of jinxing it, you should now be able to compile and run on Win, *nix and Mac.

Regarding your question, that [discrepancy](https://huggingface.co/posts/eaddario/832567461491467) is exactly what got me to modify llama-quantize in the first place, as the results I was getting did not quite stack up with my understanding of how transformers work. I'm still not 100% sure, but the working theory is that current* (2024 onwards?) models are more robust to quantization.

The PPL and test scores of models quantized as per that article seem to support this: [DeepSeek-R1-Distill-Llama-8B](https://huggingface.co/eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF), [Llama-Guard-3-8B](https://huggingface.co/eaddario/Llama-Guard-3-8B-GGUF), [Hammer2.1-7b](https://huggingface.co/eaddario/Hammer2.1-7b-GGUF), [Watt-Tool-8B](https://huggingface.co/eaddario/Watt-Tool-8B-GGUF), etc. Both, the embedding and output tensors, seem to be fairly robust to quantization, up to Q3_K.

Next weekend I'll push one more change to enable regex capability to interleave quantization per layer (e.g. Q4_K for even and Q3_K for odd numbered layers), just as in @slaren's PR (pretty cool idea BTW!)

(* just an assumption based on quantizing older models)


author_association: CONTRIBUTOR
user: login: EAddario
created_at: 2025-04-02T11:10:22Z
body: @max-krasnyansky, @jukofyork, @ddh0 and @joseph777111: found some time to add the regex support. The way it's implemented requires the user to still provide the full tensor name with the regex applying to the preceding characters. The reason for this approach is twofold:
- I'd prefer not to give the user the ability to bypass existing error checks and shoot themselves in the foot (i.e. --tensor-type ".*=xxx_yy")
- I think being able to quantize a tensor at different levels per layer opens the door to interesting possibilities (layer-wise quantization!)

I've also modified llama-imatrix to display importance score statistics, which will help in deciding which tensor/layer number to quantize. Will create PR when a get some free time.
author_association: CONTRIBUTOR
user: login: jukofyork
created_at: 2025-04-02T12:28:34Z
body:  > I've also modified llama-imatrix to display importance score statistics, which will help in deciding which tensor/layer number to quantize. Will create PR when a get some free time.

This sounds interesting - look forward to seeing it!
author_association: COLLABORATOR
user: login: ddh0
created_at: 2025-04-03T02:42:46Z
body: Currently quantize.cpp is missing an `#include <algorithm>` which causes the build to fail on linux CUDA:

```
/home/dylan/Documents/AI/llama.cpp/examples/quantize/quantize.cpp: In function ‘bool string_parse_tensor_type(const char*, std::vector<tensor_quantization>&)’:
/home/dylan/Documents/AI/llama.cpp/examples/quantize/quantize.cpp:318:10: error: ‘transform’ is not a member of ‘std’
  318 |     std::transform(tn.begin(), tn.end(), tn.begin(), tolower);
      |          ^~~~~~~~~
gmake[2]: *** [examples/quantize/CMakeFiles/llama-quantize.dir/build.make:79: examples/quantize/CMakeFiles/llama-quantize.dir/quantize.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:4067: examples/quantize/CMakeFiles/llama-quantize.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
```

Adding the missing include lets me build successfully
author_association: CONTRIBUTOR
user: login: ddh0
created_at: 2025-04-03T20:06:38Z
body: I started a discussion thread if anyone's interested, so we don't clog this PR: https://github.com/ggml-org/llama.cpp/discussions/12741
author_association: CONTRIBUTOR
user: login: EAddario
created_at: 2025-04-03T22:33:14Z
body: I'll add results of my weekend testing there as well
author_association: CONTRIBUTOR
user: login: ubergarm
created_at: 2025-04-04T16:23:01Z
body: fwiw I've been doing this with [ik_llama.cpp llama-quantize --custom-q](https://github.com/ikawrakow/ik_llama.cpp/pull/244) feature with good success. Just in case there is any desire to keep this PRs syntax compatible or not.

Specifying exact quants per tensor becomes more important now as `-ot` is merged, and the `MLA` PR is in the works here. This allows trading off quality and performance specific to target hardware configurations (e.g. how much VRAM to leave available for MLA kv-cache with using `-ot exps=CPU` etc).

I have a couple custom quants up on huggingface [ubergarm/DeepSeek-V3-0324-GGUF](https://huggingface.co/ubergarm/DeepSeek-V3-0324-GGUF) that use this technique.

Here is an example bash script recipe for an experimental CPU only speed blend:

<details>

<summary>CPU only quant performance blend V3-0324 recipe</summary>

*NOTE*: mainline llama.cpp doesn't have all these quants, but you can see how regex tensor<->quant mappings via `--custom-q` allows easy testing and maintenance of recipe scripts.

```bash
#!/usr/bin/env bash

# CPU only inference blend

# Notes:
# https://github.com/ikawrakow/ik_llama.cpp/issues/296#issuecomment-2765210993
# https://github.com/ikawrakow/ik_llama.cpp/issues/296#issuecomment-2768567062
custom="
# Token embedding and output tensors
# note token_embd cannot be repacked quant type
token_embd\.weight=iq6_k
output\.weight=iq5_k_r4
output_norm\.weight=iq5_k_r4

# First 3 dense layers (0-3)
blk\.[0-2]\.attn_k_b.*=q6_0_r4
blk\.[0-2]\.attn_.*=iq5_k_r4
blk\.[0-2]\..*=iq5_k_r4

# All attention, norm weights, and bias tensors for MoE layers (3-60)
# Except blk.*.attn_k_b.weight is not divisible by 256 and no iq6_k so go with q6_0_r4 for CPU only speed boost
blk\.[3-9]\.attn_k_b.*=q6_0_r4
blk\.[1-5][0-9]\.attn_k_b.*=q6_0_r4
blk\.60\.attn_k_b.*=q6_0_r4

blk\.[3-9]\.attn_.*=iq5_k_r4
blk\.[1-5][0-9]\.attn_.*=iq5_k_r4
blk\.60\.attn_.*=iq5_k_r4

blk\.[3-9]\.ffn_norm\.weight=iq5_k_r4
blk\.[1-5][0-9]\.ffn_norm\.weight=iq5_k_r4
blk\.60\.ffn_norm\.weight=iq5_k_r4

blk\.[3-9]\.exp_probs_b\.bias=iq5_k_r4
blk\.[1-5][0-9]\.exp_probs_b\.bias=iq5_k_r4
blk\.60\.exp_probs_b\.bias=iq5_k_r4

# Shared Experts (3-60)
blk\.[3-9]\.ffn_down_shexp\.weight=iq5_k_r4
blk\.[1-5][0-9]\.ffn_down_shexp\.weight=iq5_k_r4
blk\.60\.ffn_down_shexp\.weight=iq5_k_r4

blk\.[3-9]\.ffn_(gate|up)_shexp\.weight=iq5_k_r4
blk\.[1-5][0-9]\.ffn_(gate|up)_shexp\.weight=iq5_k_r4
blk\.60\.ffn_(gate|up)_shexp\.weight=iq5_k_r4

# Routed Experts (3-60)
# First 16 layers are more sensitive so keep larger
blk\.[3-9]\.ffn_down_exps\.weight=iq5_k_r4
blk\.[1][0-9]\.ffn_down_exps\.weight=iq5_k_r4
blk\.[2-5][0-9]\.ffn_down_exps\.weight=iq4_k_r4
blk\.60\.ffn_down_exps\.weight=iq4_k_r4

blk\.[3-9]\.ffn_(gate|up)_exps\.weight=iq4_k_r4
blk\.[1][0-9]\.ffn_(gate|up)_exps\.weight=iq4_k_r4
blk\.[2-5][0-9]\.ffn_(gate|up)_exps\.weight=iq3_k_r4
blk\.60\.ffn_(gate|up)_exps\.weight=iq3_k_r4
"
custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)

./build/bin/llama-quantize \
    --imatrix /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324.imatrix \
    --token-embedding-type iq6_k \
    --output-tensor-type iq5_k_r4 \
    --custom-q "$custom" \
    /mnt/raid/models/deepseek-ai/DeepSeek-V3-0324-bf16-GGUF/DeepSeek-256x21B-V3-0324-BF16-00001-of-00030.gguf \
    /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-CPU-IQ3_K.gguf \
    IQ3_K \
    24
```

</details>
author_association: CONTRIBUTOR
user: login: David-AU-github
created_at: 2025-04-06T07:24:29Z
body: Super stoked about this, especially option to adjust weights for "shared expert" weights.
Its playtime.
author_association: NONE
user: login: EAddario
created_at: 2025-04-07T18:16:18Z
body: **TL;DR:** A combination of Tensor-Wise Quantization (TWQ) and Layer-Wise Quantization (LWQ) is useful to generate custom models. Using _DeepSeek-R1-Distill-Llama-8B-Q4_K_M_ as an example, LWQ yields a 10.4% smaller model with only a 0.83%  𝜌PPL penalty compared to the naive model.

More info [here](https://github.com/ggml-org/llama.cpp/discussions/12741)

### Test results

| Model                               | Naive (GB) | TWQ (GB) | Reduction | LWQ (GB) | Reduction | Naive 𝜌PPL |   TWQ 𝜌PPL |   LWQ 𝜌PPL |
| :---------------------------------- | ---------: | -------: | --------: | -------: | --------: | ---------: | ---------: | ---------: |
| DeepSeek-R1-Distill-Llama-8B-IQ3_M  |       3.78 |     3.48 |      7.9% |     3.67 |      2.9% |     93.64% |     91.75% | **94.84%** |
| DeepSeek-R1-Distill-Llama-8B-IQ3_S  |       3.68 |     3.24 |     12.0% |     3.56 |      3.3% |     93.71% |     91.50% | **93.48%** |
| DeepSeek-R1-Distill-Llama-8B-IQ4_NL |       4.68 |      4.3 |      8.1% |      4.4 |      6.0% |     98.82% | **96.44%** |     95.87% |
| DeepSeek-R1-Distill-Llama-8B-Q3_K_L |       4.32 |     3.45 |     20.1% |     3.88 |     10.2% |     97.25% |     92.60% | **94.83%** |
| DeepSeek-R1-Distill-Llama-8B-Q3_K_M |       4.02 |     3.37 |     16.2% |     3.57 |     11.2% |     96.92% |     91.45% | **94.63%** |
| DeepSeek-R1-Distill-Llama-8B-Q3_K_S |       3.66 |     3.28 |     10.4% |     3.43 |      6.3% |     94.59% |     90.73% | **92.46%** |
| DeepSeek-R1-Distill-Llama-8B-Q4_K_M |       4.92 |     4.44 |      9.8% |     4.41 |     10.4% |     98.85% |     98.02% | **98.04%** |
| DeepSeek-R1-Distill-Llama-8B-Q4_K_S |       4.69 |     4.31 |      8.1% |     4.33 |      7.7% |     99.01% | **97.97%** |     97.57% |
| DeepSeek-R1-Distill-Llama-8B-Q5_K_M |       5.73 |     5.35 |      6.6% |     5.38 |      6.1% |     99.09% |     98.83% | **98.95%** |
| DeepSeek-R1-Distill-Llama-8B-Q5_K_S |        5.6 |     5.19 |      7.3% |      5.3 |      5.4% |     99.00% |     98.82% |     98.82% |
| DeepSeek-R1-Distill-Llama-8B-Q6_K   |        6.6 |     6.17 |      6.5% |     6.51 |      1.4% |     99.47% |     98.91% | **99.18%** |
| DeepSeek-R1-Distill-Llama-8B-Q8_0   |       8.54 |     7.84 |      8.2% |     7.47 |     12.5% |     99.93% | **98.99%** |     98.97% |

author_association: CONTRIBUTOR
user: login: David-AU-github
created_at: 2025-04-07T23:46:02Z
body: > **TL;DR:** A combination of Tensor-Wise Quantization (TWQ) and Layer-Wise Quantization (LWQ) is useful to generate custom models. Using _DeepSeek-R1-Distill-Llama-8B-Q4_K_M_ as an example, LWQ yields a 10.4% smaller model with only a 0.83% 𝜌PPL penalty compared to the naive model.
> 
> More info [here](https://github.com/ggml-org/llama.cpp/discussions/12741)
> ### Test results
> Model 	Naive (GB) 	TWQ (GB) 	Reduction 	LWQ (GB) 	Reduction 	Naive 𝜌PPL 	TWQ 𝜌PPL 	LWQ 𝜌PPL
> DeepSeek-R1-Distill-Llama-8B-IQ3_M 	3.78 	3.48 	7.9% 	3.67 	2.9% 	93.64% 	91.75% 	**94.84%**
> DeepSeek-R1-Distill-Llama-8B-IQ3_S 	3.68 	3.24 	12.0% 	3.56 	3.3% 	93.71% 	91.50% 	**93.48%**
> DeepSeek-R1-Distill-Llama-8B-IQ4_NL 	4.68 	4.3 	8.1% 	4.4 	6.0% 	98.82% 	**96.44%** 	95.87%
> DeepSeek-R1-Distill-Llama-8B-Q3_K_L 	4.32 	3.45 	20.1% 	3.88 	10.2% 	97.25% 	92.60% 	**94.83%**
> DeepSeek-R1-Distill-Llama-8B-Q3_K_M 	4.02 	3.37 	16.2% 	3.57 	11.2% 	96.92% 	91.45% 	**94.63%**
> DeepSeek-R1-Distill-Llama-8B-Q3_K_S 	3.66 	3.28 	10.4% 	3.43 	6.3% 	94.59% 	90.73% 	**92.46%**
> DeepSeek-R1-Distill-Llama-8B-Q4_K_M 	4.92 	4.44 	9.8% 	4.41 	10.4% 	98.85% 	98.02% 	**98.04%**
> DeepSeek-R1-Distill-Llama-8B-Q4_K_S 	4.69 	4.31 	8.1% 	4.33 	7.7% 	99.01% 	**97.97%** 	97.57%
> DeepSeek-R1-Distill-Llama-8B-Q5_K_M 	5.73 	5.35 	6.6% 	5.38 	6.1% 	99.09% 	98.83% 	**98.95%**
> DeepSeek-R1-Distill-Llama-8B-Q5_K_S 	5.6 	5.19 	7.3% 	5.3 	5.4% 	99.00% 	98.82% 	98.82%
> DeepSeek-R1-Distill-Llama-8B-Q6_K 	6.6 	6.17 	6.5% 	6.51 	1.4% 	99.47% 	98.91% 	**99.18%**
> DeepSeek-R1-Distill-Llama-8B-Q8_0 	8.54 	7.84 	8.2% 	7.47 	12.5% 	99.93% 	**98.99%** 	98.97%

@EAddario 
Just a heads up:
Distill (and preview models) are sensitive in layers 0-7 and 28-31.
You could give these more bits, and lower the other layers to maintain or augment function.
author_association: NONE
user: login: EAddario
created_at: 2025-04-12T11:26:46Z
body: Thanks for approving @slaren. Any particular use case you have in mind it will prevent? Maybe I can work it into the logic.
author_association: CONTRIBUTOR
user: login: EAddario
created_at: 2025-04-12T14:43:24Z
body: Got a better quality LWQ mix using the stats from the modified [llama-imatrix](https://github.com/ggml-org/llama.cpp/pull/12718). More info [here](https://github.com/ggml-org/llama.cpp/discussions/12741)

# Test results

| Model                               | Naive (GB) | TWQ (GB) | Reduction | LWQ (GB) | Reduction | Naive 𝜌PPL |   TWQ 𝜌PPL |   LWQ 𝜌PPL |
| :---------------------------------- | ---------: | -------: | --------: | -------: | --------: | ---------: | ---------: | ---------: |
| DeepSeek-R1-Distill-Llama-8B-IQ3_M  |       3.78 |     3.48 |      7.9% |     3.69 |      2.5% |     93.64% |     91.75% | **94.24%** |
| DeepSeek-R1-Distill-Llama-8B-IQ3_S  |       3.68 |     3.24 |     12.0% |     3.43 |      6.8% |     93.71% |     91.50% | **92.97%** |
| DeepSeek-R1-Distill-Llama-8B-IQ4_NL |       4.68 |     4.30 |      8.1% |     4.39 |      6.1% |     98.82% | **96.44%** |     96.12% |
| DeepSeek-R1-Distill-Llama-8B-Q3_K_L |       4.32 |     3.45 |     20.1% |     3.76 |     13.0% |     97.25% |     92.60% | **94.79%** |
| DeepSeek-R1-Distill-Llama-8B-Q3_K_M |       4.02 |     3.37 |     16.2% |     3.56 |     11.3% |     96.92% |     91.45% | **94.45%** |
| DeepSeek-R1-Distill-Llama-8B-Q3_K_S |       3.66 |     3.28 |     10.4% |     3.31 |      9.7% |     94.59% |     90.73% | **92.23%** |
| DeepSeek-R1-Distill-Llama-8B-Q4_K_M |       4.92 |     4.44 |      9.8% |     4.41 |     10.5% |     98.85% |     98.02% | **98.03%** |
| DeepSeek-R1-Distill-Llama-8B-Q4_K_S |       4.69 |     4.31 |      8.1% |     4.28 |      8.8% |     99.01% | **97.97%** |     97.72% |
| DeepSeek-R1-Distill-Llama-8B-Q5_K_M |       5.73 |     5.35 |      6.6% |     5.38 |      6.2% |     99.09% |     98.83% | **98.94%** |
| DeepSeek-R1-Distill-Llama-8B-Q5_K_S |       5.60 |     5.19 |      7.3% |     5.24 |      6.4% |     99.00% |     98.82% | **98.85%** |
| DeepSeek-R1-Distill-Llama-8B-Q6_K   |       6.60 |     6.17 |      6.5% |     6.57 |      0.5% |     99.47% |     98.91% | **99.19%** |
| DeepSeek-R1-Distill-Llama-8B-Q8_0   |       8.54 |     7.84 |      8.2% |     7.73 |      9.4% |     99.93% |     98.99% | **99.26%** |
author_association: CONTRIBUTOR
user: login: slaren
created_at: 2025-04-12T14:57:09Z
body: > Any particular use case you have in mind it will prevent? Maybe I can work it into the logic.

For example, using `ffn` as the pattern to set the type of all ffn tensors, or `attn` of all attention tensors, without having to specify each one individually.
author_association: MEMBER
user: login: EAddario
created_at: 2025-04-13T10:01:24Z
body: I see what you mean. The choice of approach was a trade-off between ensuring the program continues to work exactly as before (backwards compatibility), not introducing new options that are already available (--pure, --output-tensor-type and --token-embedding-type), and adding new capabilities in a way that's consistent with existing error checking logic.

By restricting the tensors, users won't be able to do things that clearly are not useful, like trying to quantize norms, lerps, ropes, etc., but you're right in that users wanting to quantize all attn tensors would need to pass three options (--tensor-type attn_q=q4_k --tensor-type attn_k=q4_k --tensor-type attn_v=q4_k) instead of just one (--tensor-type attn=q4_k).

Once the changes are merged, I'll open a new PR to address this, within the tensor checking logic to avoid matching instances like attn_norm, ffn_norm, etc., plus implementing @ggerganov's recommendation to make the struct C compatible.
author_association: CONTRIBUTOR
user: login: acbits
created_at: 2025-04-13T20:43:40Z
body: > By restricting the tensors, users won't be able to do things that clearly are not useful, like trying to quantize norms, lerps, ropes, etc., but you're right in that users wanting to quantize all attn tensors would need to pass three options (--tensor-type attn_q=q4_k --tensor-type attn_k=q4_k --tensor-type attn_v=q4_k) instead of just one (--tensor-type attn=q4_k).
> 

Late to this conversation, but isn't this case already handled by a regex that uses grouping?

`--tensor-type 'attn_(q|k)'=q4_k` could be applied to attn_q and attn_k?
author_association: NONE
user: login: EAddario
created_at: 2025-04-14T06:29:15Z
body: Not quite @acbits. For the reasons described above, the program requires the full tensor name, with the regex applying only to preceding characters.

I'll improve this behaviour in the next PR.
author_association: CONTRIBUTOR
user: login: joseph777111
created_at: 2025-04-14T16:27:58Z
body: @EAddario Congrats! 🚀 
author_association: NONE
user: login: Djip007
created_at: 2025-04-19T13:44:53Z
body: Really late on that.  And nice PR.

1 idea can we define all logic in more general way, for example using json format? and possibly read it from a file for most advance case?
author_association: CONTRIBUTOR
user: login: EAddario
created_at: 2025-04-20T09:31:18Z
body: Thanks @Djip007. @ngxson had a similar suggestion, and it's in my to-do list.

The way I'm thinking about it is for llama-imatrix (#12718) to generate a file with "recommended" quants, based on relevant statistics, which can then be processed by llama-quantize. The file can of course be changed/created by hand.

I don't know exactly what "recommended" means yet so open to suggestions.
author_association: CONTRIBUTOR
user: login: Djip007
created_at: 2025-04-20T13:23:35Z
body: > I don't know exactly what "recommended" means yet so open to suggestions.

I'll think about it... if I have any ideas, I'll try to share them.
Is there a better place than this closed PR to make suggestions?
author_association: CONTRIBUTOR
user: login: EAddario
created_at: 2025-04-21T07:30:41Z
body: Feel free to comment on #12718
author_association: CONTRIBUTOR