BackSlash: Rate Constrained Optimized Training of Large Language Models
Authors: Jun Wu
Jiangtao Wen
Yuxing Han
Venue: ICML 2025 poster (2025)
Keywords: Model Compression
Rate-Distortion Optimization
Entropy Encoding
Primary area: deep_learning->large_language_models

Abstract: The rapid advancement of large-language models (LLMs) has driven extensive research into parameter compression after training has been completed, yet compression during the training phase remains largely unexplored. In this work, we introduce Rate-Constrained Training (BackSlash), a novel training-time compression approach based on rate-distortion optimization (RDO). BackSlash enables a flexible trade-off between model accuracy and complexity, significantly reducing parameter redundancy while preserving performance. Experiments in various architectures and tasks demonstrate that BackSlash can reduce memory usage by 60\% - 90\% without accuracy loss and provides significant compression gain compared to compression after training. Moreover, BackSlash proves to be highly versatile: it enhances generalization with small Lagrange multipliers, improves model robustness to pruning (maintaining accuracy even at 80\% pruning rates), and enables network simplification for accelerated inference on edge devices.

BackSlash:
Rate Constrained Optimized Training of Large Language Models
Jun Wu
Jiangtao Wen
Yuxing Han
Abstract
Table 1. Parameter scale and growth rate of GPTs as example over recent years.
The rapid advancement of large-language models (LLMs) has driven extensive research into parame- ter compression after training has been completed, yet compression during the training phase remains largely unexplored. In this work, we introduce Rate-Conee Sate-Constrained Training (BackSlash), a novel training-timi compression approach based on rate- distortion optimization (RDO). BackSlash en- ables a flexible trade-off between model accuracy and complexity, significantly reducing parameter redundancy while preserving performance. Exper- iments in various architectures and tasks demon- strate that BackSlash can reduce memory usage by 60% 90% without accuracy loss and pro- vides significant compression gain compared to compression after training. Moreover, BackSlash proves to be highly versatile: it enhances general- ization with small Lagrange multipliers, improves model robustness to pruning (maintaining accu- racy even at 80% pruning rates), and enables net- work simplification for accelerated inference on edge devices.
Model name
Time
Parameter size
Growth rate
GPT-1
2018.06
117M
GPT-2
2019.02
1.5B
12.8x
GPT-3
2020.06
175B
116.7x
GPT-4
2024.11
1.8T
1200.0x
The ever-increasing size of LLM parameters leads to in- creased computational costs, inference latency, and network distribution overhead during model deployment. enable efficient inference of I.L.Ss on edge devices, extensive re- search has focused on model compression using techniques such parameter quantization, model pruning, and low- rank matrix decomposition. However, while there have been extensive studies on LLM redundancy at the microscopic level, such as precision and structural inefficiencies, the overall parameter distribution has received little attention. In addition, most existing compression techniques are ap- plied after training, as opposed to being integrated into the L.LM parameter training process proactively achieve opti- mized trade-off's between parameter precision, model size, and model performance. Finally, existing studies assume that model parameters follow the Gaussian distribution and, therefore employ Huffman coding designed using empirical statistics for compression. Both the probabilistic model and the entropy coding technique have room for improvement.
1. Introduction
As the foundation of modern artificial intelligence, genera- tive large language models (LLMs) such as Llama (Touvron et al., 2023), GPT (Brown et al., 2020), and Qwen (Bai et al., 2023) exhibit remarkable self-lcarning and non-linear modeling capabilities. With continuous advancements in deep learning, the parameter scales of LLMs have grown at unprecedented rate, shown in Table 1. Looking ahead, it is expected that the parameter scale of neural networks will continue to expand rapidly, driving further progress in Al development.
In this paper, introduce rate-constrained optimized ap- proach to LLM training. By incorporating model parameter size into the training process through rate (R) and distor- tion (D) joint optimization, the proposed rate-constrained training (BackSlash) approach capable of producing the optimal performing model for given parameter set size, that is, producing the best-fit model given by its end appli- cation (hardware constrained parameter size).
'Shenzhen International Graduate School, Tsinghua Univer- sity "Computer Science, New York University. Correspondence Jiangtao Wen
The main contributions of this paper follows:
to: Yuxing Han <yuxinghan@e www.tsinghua.cuu.cm. Jiangta (project lead) (ww9663@nyu.cduu
1. Instead of the widely used Gaussian model for LLM parameter distribution, we found through extensive experiments that the generalized Gaussian (GG) distri- bution with the shape parameter less than 2 is better
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

BackSlash:
Rate-Constrained Optimized Training of Large Language Models
model.
tivations. Liu al. (2021) simultaneously maintained the representational power of non-uniform quantization and the efficiency of uniform quantization.
2. We propose to use exp-Golomb (EG) codes for entropy coding of LLM parameters, whose distribution can be well-modeled by GG distributions. It has been shown (Wen & Villasenor, 1999) that for GG sources, EG codes can achieve coding efficiency very close to the entropy limit, well over 90% in many cases. We also find the optimal EG code with k=0 implementation can accommodate many applications.
Low-rank decomposition (Jaderberg et al., 2014; Masana et al., 2017), parameter sharing (Wang et al., 2017; Kos- saifi et al., 2019), and knowledge distillation (Xu et al., 2018; Chen et al., 2017) have demonstrated significant ef- fectiveness in applications. Additionally, with large-scale distributed deep learning training systems, communication- efficient gradient compression techniques were proposed (Lin et al., 2017). These approaches collectively enhance the efficiency of deep learning model training and deployment.
3. Based on the GG distribution observation and using EG codes as entropy codes, we proposed a discretized generalized Gaussian information rate (DGGR) to mea- sure the model information rate and an BackSlash algorithm that jointly optimizes the information rate and performance during the training phase of LLMs. Experiments with different LLMs and different deep- learning tasks show significant savings in model size as compared with both unconstrained training and un- constrained training followed by entropy coding.
2.2. Rate Distortion Optimization
Information theory (Shannon, 1948; Cover, 1999) mathemat- ically quantified the efficiency with which information can be transmitted, stored, and processed, where rate-distortion function defines the minimal distortion that can be achieved while entropy coding a system to a given bitrate (Davisson, 1972; Berger, 2003).
In practical applications, rate-distortion optimization (RDO) has found extensive adoption in video coding (Luttrell et al., 2000; Itu-T & Jtc, 2010; Wien, 2015; Brand et al., 2022; Chen et al., 2023; Guo et al., 2023; Chiang et al., 2023; Xia et al., 2023b; Zhang et al., 2024), etc. are also continuing to deepen the application of RDO in video and images.
2. Related Work
2.1. LLMs Compression
To achieve low-cost distribution, deployment, and inference, many compression strategies for LLMs have been proposed.
Pruning reduces computational and storage overheads by removing unimportant weights or neurons from the model. Unstructured pruning achieves compression by removing redundant connections, c.g., Han et al. (2015b) and Han et al. (2015a) proposed a pruning method based on weight paradigms. Because unstructured pruning may lead irreg- ular network structures, structured pruning of filters chan- nels was proposed. Li et al. (2016) proposed pruning based on filter, while Luo et al. (2017) proposed Thinet that min- imized the reconstruction error. Hardware constraints (He et al., 2018; Wang et al., (0) such as energy consumption and delay were also introduced into the pruning process to optimize the model performance in resource-constrained environments. Prune continues to be an important direction for model optimization, as evidenced by recent publica- tions such as Dynamic Structure Pruning (Park et al., 2023), LAPP (Zhai et al., 2023), and Turbo- VBll (Xia et al., 2023a).
In recent years, RDO has also been introduced for the com- pression of neural networks. For example, Gao et al. (2018) investigated the fundamental limits of model compression and proposed a compression framework for pruning, quanti- zation, and other techniques. Isik et al. (2021) proposes a new pruning strategy based on RDO to approach the com- pression limits of neural networks. In both cases, RDO is applied after models have been trained to further pruning and quantization, as opposed to being integral to the training process itself.
3. Generalized Gaussian Model of LLM Parameters
Most research assumes that LLM parameters follow the Gaussian distribution in the initialization and rarely dis- cussed the distribution after training. For example, Gaussian distribution was used by both Xavier and He for random parameter initialization (Glorot & Bengio, 2010; He et al., 2015). However, through extensive experiments, we found that the more broad generalized Gaussian (GG) distribu- tion family with shape parameter less than 2 might be a better model for LLM models, especially considering that different regulations during training may impact parameter distribution. The distribution usually also changes during training as the model converges. In practice, the parameter
Quantization can speed up training and inference by reduc- ing the precision of weights and activation values. Binary weights (Courbariaux et al., 2015; Rastegari et al., 2016), triple weights (Li & Liu, 2016; Zhu et al., 2016), cluster quantization (Gong et al., 2014; Choi et al., 2016), and mixed bit-width quantization (Zhou et al., 2016; Wang et al., 2018b) were examples of quantization techniques. Long et al. (2020) used shift operation to replace the costly full- precision operation by quantizing low-bit weights and ac-
2

---

BackSlash: Rate-Constrained Optimized Training of Large Language Models
distribution tends develop heavier tails during the training process(ortuin al., 2021).
4. Rate-Constraing Training (BackSlash) of LLMs
Mathematically, the probability density function (pdf) of generalized Gaussian distribution is defined as
4.1. Overview and loss function definition
In contrast to traditional non-constrained training, the target (1) loss function for optimization in BackSlash considers both model performance and model size is
f(x) = Cie- -Calaa
Cie
Calarl
where
vy
C1 =
C2 = 7²"
(2)
J= D
(6)
21(1/D)
where D denotes the distortion of the fitting data, i.e. the de- viation between the model predictions and the ground truth. R denotes the rate of the model parameters, indicating the complexity of the model itself. À is the Lagrange multiplier. The selection of distortion D varies depending on the deep- learning task. For example, we often use the categorical cross-entropy loss function in classification problems and the mean squared error loss in regression tasks. Methods like KI. divergence or logarithmic loss are also utilized in some specific tasks. We can choose the most suitable empir- ical loss function for specific tasks, which does not affect the BackSlash results.
1 I(3/v)
7 0 T(1/v))
r(a) = /-00000 aa-le dt,
and a > O.
It is easy to see that when V = 1, the GG distribution is the Laplacian distribution, while when v = 2. it is the Gaussian distribution. Varying the shape parameter of GG distribution allows for better match between the probabilistic model to better match LLM while using the same mathematical formulation.
The rate R is expressed by the average information content of the parameters, defined using DGGR. Combining (5) and (6) we get
The GG distribution in (1) is a continuous distribution, while in reality, LLM parameters all have fixed-length and limited precision. Therefore, we treat LLM parameters as quan- tized GG distribution. Assuming the quantization step size of the parameters 8, the probability of a parameter 0, is
N Σ 10,1%
J ] +
(7)
N
p(0.) = { 8.,6
f(z)dz.
(3)
where X and Y denote the training ground truth, f and 0 denote the forward propagation function and parameter set of the neural network.
9.
As is typically small, we approximate p(O.) Ri (0.) =
(4)
The shape parameter v of DGGR is not a constant during training and needs to be dynamically estimated before each batch of gradient descent. A well-known method (Sharifi & Leon-Garcia, 1995) for estimating the shape parameter is by introducing comparison function:
The validity of this assumption could be verified with exist- ing LLMss For instance, BERT-base (1 (110M) can be well- modeled by GG with shape parameter of 1.36, the shape parameter for GPT2 (774M) is 1.54, or 1.26 for Llama3 (IB), or 0.85 for DeepSeek (7B), and their distributions are shown in Fig. 1. These shape parameter values are, although different, all smaller than 2. The corresponding pdfs show higher peaks and longer tails than the Gaussian distribution.
plv) = I(1/v) r(3/v)
(8)
(2(2/D)
E 1210T
Specifically, the estimation process of can be organized as follows:
Denote the size of LLM as Ny. the mean of the infor- mation content of the parameters can be calculated as R(0)- - n Σizr log, p(0.). Neglecting constant fac- tors and terms, we define discretized generalized Gaussian rate (DGGR) as follows
1. obtain the model parameters 0 and compute E[A?] and [²[00]]
2. estimate p(v) by (8) based on E[[2]] and IE22 [02
N₂ R(0) = N₂
3. Find v using p(v).
8 V
(5)
Additionally, notice that when 0 < v < and 0 - O, V (())) + co. This causes severe oscillations in the pa- rameters during gradient descent, which prevents the model
and use R(O) as a measure of the information complexity of the model.
3

---

BackSlash: Rate-Constraince Optimized Training of Large Language Models
1.4
14
1,4
1,4
GGD
GCD
GGD
GGD
Gb
GD
GD
1.2
12-
1.2
1.2
1.0
1,0
1.0
1.0-
Relatve Prqquncyy
Relatie Prequencyy10)
Relative FaqquncyS5
Relative Fnsquency00
0.8
0.8
0.8
0.8
0.6-
0.6-
0.6-
0.5-
0.4
0.4
0.4-
0.4-
0.2
0.2
0.2
0.2
0.0
-6.2
-0.1
0.0
0.0
-02
-6.1
0.0
0.1
0.0
-0.2
-6.1
0.0
0.1
0.0
-0.1
0.0
0.1
0.2
Model Porameter Value
Model Parameter Valuo
Model Parameter Value
Model Puramoter Value
BERT
(b) GPT2
(c) Llama3
(d) DeepSeek
Figure 1. Parameter distributions fitting by generalized Gaussian distribution (GGD) and Gaussian distribution (GD) under different LLMs. GGD fits the boundaries of the parameter distributions better than GD does.
from converging. To address this, we introduce a trick to optimize the gradient descent of R()) by adding a constant e (e to control the gradient size and modify the infor- mation rate formula as R(0) N, E (10,) + e)". We refer to this method of gradient suppression as soft gradient clipping.
degenerates into L1 (Fitriani et al., 2022) regularization (N. E Nr (0:) and T.2 (Hoerl & Kennard, 1970) regular- ization (,) Σnn (0,3)) respectively. This means, that Lı and L2 regularization are special cases of DGGR when the model parameter distribution is the Laplace and the Gaus- sian distributions respectively.
4.2. BackSlash algorithm description
4.3. Entropy coding of LLM using EG codes
The overall BackSlash algorithmic can be summarized as follows:
The complexity constraint in BackSlash defined using DGGR. In practice, the parameters will have to be entropy coded using practical entropy codes, whose rate can not exactly match the DGGR.
Algorithm 1 CoateCoonttained Training (BackSlash)
1: Require: Model f, learning rate loss function 4,
Due to its simplicity. Huffiman codes have been used for en- tropy coding of LLM parameters. For example, in Han et al. (201 5a) achieved 20%-30% size reduction using Huffman coding. However, using Huffman coding in BackSlash or compression of LLM in general has several drawbacks.
Lagrange multiplier x, and clipping coefficient
2: for each epoch r do
3: for each batch (I,, y1) do
4: Retrieve model parameters 0.
5: Estimate comparison function p(v): p(v)
E|0"1 ETM
First of all, Huffman code tables designed using plicit distributions calculated from LLM parameters. The mismatch between the distribution of the parameters and the distribution for which the Huffman code designed may lead severe coding efficiency loss. On the other hand, the large parameter size and non non-parallelizatee building process of Huffman code may bring prohibitively high complexity to BackSlash. This is also why used the theoretical DGGR as opposed to coded bits in the loss function.
6: Find shape parameter v using p(v).
7: Forward propagation and calculate RD Cost J:
J + L(2,, yr f.0) + A N, ( e)"
8:
Backward propagation and optimize parameters: ( + X N. (10) + 11)
table
0 0-0
9: end for
10: end for
11: Until convergence or max iterations.
Secondly, Huffman tables designed for different LLMs are different, while a practical implementation may often need to accommodate multiple models in the same system (e.g.
should be noted that if we set the shape parameter y in DGGR to be and 2, we find that DGGR

---

BackSlash: Rate-Constrained Optimized Training of Large Language Models
Table Structure of exp-Golomb code with different parameter which from to an example. general. EG codes with smaller parameter & encode better for GG sources with low shape parameters.
Parameter (k)
0
1
2
3
4
5
6
7
8
9
0
010
011
00100
00101
00110
00111
0001000
0001001
0001010
k 1
10
11
0100
0101
0110
0111
001000
001001
001010
001011
A 2
100
101
110
111
01000
01001
01010
01011
01100
01101
3
1000
1001
1010
1011
1100
1101
1110
1111
010000
010001
10000
10001
10010
10011
10100
10101
10110
10111
11000
11001
k 5
100000
100001
100010
100011
100100
100101
100110
100111
101000
101001
on the same chip).
Table 3. Average code lengths for several models with different EG parameters. With the EG parameter increasing, the average code length of the model parameters also increases and the EG code gradually converges to the fixed-length code.
Thirdly, the Huffman table designed based on empirical distributions usually is not well-structured, leading more complicated encoder/decoder implementation.
Fourthly, we observe the Huffman code can only pro- vide minimal efficiency gains over EG code on BackSlash- trained models in all subsequent experiments.
Model
h 0
k 1
k 2
k
3
4
BERT
2.64
3.16
3.74
4.42
5.17
GPT-2
2.46
3.05
3.70
4.40
5.19
Llama-3
1.72
2.47
3.26
4.12
5.03
As noted in Section 3, LLM parameters can be modeled well by quantized GG distributions with shape parameters less than 2. In Wen & Villasenor (1999), Wen and Villasenor studied and proposed using exp-Golomb (EG) codes to tropy coding quantized GG sources. The structure of EG code is shown in Table 2. The advantages of EG codes can be summarized as the following:
Gemma-2
1.16
2.10
3.06
4.02
5.01
In addition, we note that after BackSlash, most model pa- rameters are zero, while the non-zero values are extremely sparse, usually accounting for few percent of all possible values. For example, the number of code words occupied by BERT with BackSlash after quantization with a step 2-8 is 2695 but only 641 are actually used. And except for a small number of quantized parameters near the others are highly disordered. For example, the quantized parameter - 177 of BERT is only ranked 609 by frequency but actually it takes up 142nd. Therefore, prior to entropy coding of model parameters, we first rank the number of occurrences of all parameter values and map each value that model pa- rameters might actually take an index. The index, instead of the parameter value, is then entropy coded. This process can be formally summarized as follows:
1. the efficiency of EG codes is consistently within few percentage points of the entropy limit and almost identical to the Huffman code specifically designed for each quantized GG source,
2. the performance of EG codes is robust with regard to parameter mismatch, and as a result, adaptive coding is not needed when parameters of the quantized GG source change.
3. EG codes contain an infinite number of codewords, and can therefore be used for LLM of any size,
4. EG codes nicely structured, and allow for highly optimized encoder/decoder.
Algorithm 2 Parameter Entropy Encoding
1: Require: Model parameter set θ, quantization step size
2-", encoding strategy Enc.
Therefore, in our experiments, we used EG codes for the actual entropy coding rate (as opposed the theoretical rate of DGGR in BackSlash) for both entropy coding of param- eters after traditional, unconstrained L.M training, and in comparison, BackSlash. In our experiment, we tested EG with different parameters on several models with BackSlash which is shown Table 3, and the EG code that we found was optimal was EG with parameter k:
2: Quantize the parameters: Q + round .
3: Sort by frequency, build the code table by quantized pa-
rameter and sorted index: C + {(g,, q) | q: E Qa c
C.}
Map quantized parameters codewords and encode it bitstream: Bstream <- Enc(C(Q))
5: Output: Bitstream B Batream and code table C
5

---

BackSlash: Cate-Constrainee Optimized Training of Large Language Models
1m1
Am32
2m10
RD Cost RatetRRan)
3=300
2m100)
Aw2000
Soupe Dinsmegge
1.5
Lo
1.5
as
as
0.6
230
900
750
1100
1500
1001
1230
3000
200
1000
120
1500
1700
2000
Trining Step(n)
Training Step(n))
Figure 2. RD cost rate changes in training with different La- grange multiplier (A).
Figure 3. Shape parameter changes in training with different Lagrange multiplier (A).
10:02
10.00
1000
1010
32/0
10:60
00055
Average Code Lenugha(
FaoehSezati GapeGalmmh Clole
Than Dutuent
733
5.00
Hiffrean Oede
Accuracy (%
Tivt Dotwert
5.47
545
90.95
1000
Lagrange Multiplier())
Lagrango MultipFoer())
Figure Impact Lagrange multipliers on average code Figure 5. Impact of Lagrange multipliers on accuracy of test and train Dataset.
length of various encoding algorithms.
The distribution of indices still follows a generalized Gaus- sian, though the value mapping induces slight deviations compared to the original parameters. For example, the shape of parameters and index of BERT under normal training is 1.36 and 1.47, while under BackSlash they become 0.26 and 0.30, respectively. Nevertheless, these minor shape variations negligibly impact entropy coding efficiency, as EG codes maintain robustness across the entire family of generalized Gaussian sources.
performances of BackSlash by classification accuracy, and generation tasks on DeepSeek evaluated by next token accu- racy. We mainly use classification tasks on BERT to analyze the effects of BackSlash when it trained and deployed, as classification accuracy one of the most intuitive quanti- tative metrics of model performance and BERT model has better performance on classification tasks. In addition, we examined the entropy coding efficiency of EG codes as compared with Huffman (HM) coding and fixed-length (FL) coding with value mapping for all EG, HM and FL codes.
The sparsity of the values model parameters the reason that in BackSlash, we use DGGR, as opposed EG code length directly in BackSlash even though the same EG code may be used throughout BackSlash, the parameter value that the codeword is mapped to changes.
5.1. Performance
Taking the sentiment analysis task of the BERT model as an example, we tested BackSlash using different Lagrange multiplier settings. Fig. shows how the loss changes in training under different Lagrange multipliers. As RD Costs may vary significantly with different Lagrange multipliers, for visual clarity, we used (RAD = log BJmin) B = 0.995) as the y-axis. As can be seen from the figure, the larger the Lagrange multiplier, the steeper the curve, reflecting the fact that the Lagrange multiplier controls the training speed of the BackSlash.
The mapping (termed "Value Mapping") between the quan- tized parameter set Q, and the codeword set C, defined as C ((1,, (4) gi E Qs, C C.).
5. Experiments
We perform various classification tasks popular LLMs including BERT, GPT, Llama, and Gemma to evaluate the
6

---

BackSlash: Cate-Constrained Optimized Training of Large Language Models
Square of Froquency (109)
10
Square of Frequency (10*)
1
Square of Frsquency (10*)
0.2
0.1
0.0
0.1
0.2
0.2
0.1
0.0
0.1
0.2
0.2
0.1
0.0
0.1
0.2
Model Parameter Value
Model Paremeter Value (0)
Model Parameter Value (0)
(a) Lagrange Multiplier 0
(b) Lagrange Multiplier 10
(c) Lagrange Multiplier 1000
Figure Parameters distribution under different Lagrange multiplier training. With the Lagrange multipliers increasing, parameter distributions become more concentrated and have higher peaks and lower tails.
Table 4. Compression performance of BackSlash with different model architectures and parameter scales.
Model
Param Size
Method
FL (bits)
EG (bits)
HM (bits)
EG Compress
HM Compress
Accuracy
10.00
7.31
5.47
27%
45%
93.63 %
BERT
110M
BackSlash
10.00
2.64
2.42
74%
76%
92.59%
11.00
7.78
5.73
29%
48%
85.92%
GPT
774M
BackSlash
11.00
2.46
2.25
78%
80%
88.73 %
10.00
5.49
4.43
45%
56%
86.09%
Llama
BackSlash
10.00
1.72
1.66
83%
83%
86.93 %
11.00
4.45
3,95
60%
64%
86.95 %
Gemma
BackSlash
11.00
1.16
1.15
89%
90%
85.86%
Fig. 3 illustrates how the shape parameter of the parameter distribution varies during training. For different values, the shape parameter
the training set and 1.90% on the test set as compared with normal training (i.e. X 0). was observed that model accuracy was monotonic with regard to X, i.e. there is optimal À value, the setting of which is topic under investigation.
to identical initial value but converged to different values, reflecting how À in the BackSlash led to different model distributions.
In Fig. 4, after quantizing the model parameters with the quantization step 2 we use EG code, HM code and FL code to encode the model parameters and compute the av- erage code length respectively. When applying EG coding and HM coding after unconstrained training, the model size was compressed to 73% and 55% of the size of FL coding, corresponding to 27% and 45% saving. Whereas BackSlash with EG coding and À of 1000, reduced the model size to 26% and 24%. Even though Huffman coding leads to a very small gain in coding efficiency, different Huffman table and the corresponding encoder/decoder will have to be designed and implemented for each LLM. In contrast, the same EG table could be used across models and sizes (e.g. DeepSeek 7B and 170B).
Fig. 6 shows the impact of on model parameter distribu- tion. As can be seen clearly, as À increases, i.c. we give more weights the rate in the accuracy-rate trade-off, the model trained by BackSlash would become more sparse.
In the current study, the setting of À was still through trials- and-errors. For example, when we set set of values for A and train BERT model using BackSlash until convergence, we found that the model trained with 2000 achieved the best overall trade-off with 2.52% loss in accuracy and only 13% of the size. Moreover, our extensive experiments, consistently achieves similar and remarkable effectiveness across various models and tasks.
5.2. Generalization Analysis
Fig. 5 demonstrates the effect BackSlash on model accu- racy. We can find that BackSlash with reasonable did not have significant effect on accuracy. For the model with 1000, model performance decreased by only 0.02%
Model architectures and training tasks of great signifi- cance to both the process and the performance of the model and tend to affect the final model obtained from training

---

BackSlash: Rate-Constrained Optimized Training of Large Language Models
Table 5. Compression performance of BackSlash under different deep learning tasks.
Task
Dataset
Method
FL (bits)
EG (bits)
HM (bits)
EG Compress
HM Compress
Accuracy
10.00
7.31
5.47
27%
45%
93.63%
Sentiment
IMDB
BackSlash
10.00
2.64
2.42
74%
76%
92.59%
10.00
7.31
5.47
27%
45%
99.65%
Spam
Enron-Spam
BackSlash
10.00
2.42
2.19
76%
78%
98.96%
10.00
7.31
5.47
27%
45%
70.78%
Topic
20 Newsgroups
BackSlash
10.00
3.61
3.18
64%
68%
69.36%
11.00
5.95
4.70
46%
57%
99.97%
Q-A
SQuAD
BackSlash
11.00
2.90
2.81
74%
74%
99.97%
11.00
6.10
4.70
45%
57%
99.96%
Translation
WMT-19 19
BackSlash
11.00
3.10
3.00
72%
73%
99.95%
Accurme (56)
Acctrey (96)
100
000
01
Nomoll Tiitilag
Neraal Thxinn Mark
Resttihh Modd
2:21
02
24
0.5
Quantization Step
Prening Rate
Figure Quantization using different quantization steps for BackSlash model and normal training model.
Figure Pruning using different pruning rates for BackSlash model and normal training model.
heavily. It is worth discussing whether BackSlash has the same effects in other models and tasks besides the sentiment analysis of BERT.
der normal training and BackSlash. The "Sentiment" and "Spam" both binary-classification tasks and the "Topic" is 20-classclassiffcation tasks, which evaluated by classification accuracy. The "Q-A" and "Translation" are both text generation tasks, which are evaluated by next token accuracy. These tasks can achieve satisfactory com- pression performance without compromising model accu- racy, BackSlash achieves approximately 70% compression rate compared to the original size both classification and generation tasks, demonstrating its strong generalization capability across different task types.
In Table 4, we perform the sentiment analysis task on BERT, GPT, Llama, and Gemma under normal training and Back- Slash, respectively. These models are chosen because the differences structure and parameter size among them are large enough to reflect the wide utility of BackSlash. Al- though different model structures introduce some variability in the results, BackSlash performs similarly for parameter compression. For all the models, BackSlash compresses them by more than 75%, with the highest being 90% for Gemma. Such similar performance comes from the insen- sitivity of BackSlash to network structure and parameter size. addition, it can be seen that in GPT and Llama, the accuracy of using BackSlash is instead slightly higher than that of normal training, which analyze originating from the regularization effect attached to BackSlash.
5.3. Deployment
Deployment and inference for edge devices are always the central problem and primary purpose of model compres- sion, and quantization and pruning are the main means to deploy the fine-tuned LLMs in edge devices. Therefore, it is necessary to discuss whether the generalization ability of BackSlash models can be maintained quantization and pruning.
In Table 5, we perform more classification tasks on the BERT model and generation tasks on DeepSeek model un-
8

---

BackSlash: Rate-Constrained Optimized Training of Large Language Models
Table Compression performance of BackSlash under different regularization terms.
DGGG
Convergence Shape
Accuracy
FL (bits)
FG (bits)
HM (bits)
EG Compression
HM Compression
DGGR
0.13
91.18%
10.00
1.37
1.32
86%
87%
Lo.s
0.22
91.88%
10.00
2.90
2.01
71%
80%
L1
0.15
90.65%
10.00
1.52
1.46
85%
85%
L/
0.10
88.29%
10.00
1.16
1.14
88%
89%
Fig. 7 illustrates how the accuracy of the BERT model varies with the quantization steps under normal training and Back- Slash. When the quantization step is taken 2-1, the gen- cralization ability of both normal training and BackSlash models is completely destroyed. When the quantization step not less than 2- the accuracy of both models changes very smoothly. Both models show the same trend in quan- tization. This is because quantization uniformly destroys the accuracy of the parameters, so whether or not to use BackSlash does not have an additional negative impact on the quantization results. Furthermore, We performed the same experiments in GPT, Llama, and Gemma, and they all showed identical results BERT.
their impacts on model performance and code rate. As shown in Table 6. Shape-specific Lp regularization terms present notable theoretical and practical limitations.
From a theoretical perspective, the Lo.5, L1, and L2 terms implicitly assume that model parameters follow generalized Gaussian distributions with fixed shape parameters of 0.5, 1, and 2, respectively. However, the actual shape parameters of the model converge to 0.22, 0.15, and 0.10, contradicting the fixed-shape hypothesis. In contrast, DGGR's dynamic shape parameter adaptation naturally accommodates the evolving weight distribution throughout the optimization process.
From an effectiveness perspective, L2 achieves marginally better compression than DGGR but ineurs significant ac- curacy degradation, which indicates its detrimental impact model performance. L1 is inferior to DGGR in both code length and accuracy. Lo.5 demonstrates slightly bet- ter accuracy but its code length is more than twice that of DGGR, which shows its weakness in parameters compres- sion. These findings suggest that DGGR's adaptive shape parameter adjustment puts performance and code rate in better balance.
Fig. 8 illustrates how the accuracy of the BERT model varies with the pruning rates under normal training and BackSlash. We can see that the predictive ability of the conventionally trained model has begun to degrade when the pruning ratio reaches 50% and has completely lost its predictive ability when it reaches 60%. Instead, the BackSlash model continu- ally maintains its generalization accuracy when the pruning ratio reaches 80%. This is because BackSlash makes the model's parameter distribution more sparse, which increases the space for pruning. So BackSlash's model is also easier to deploy on edge-end devices through pruning and performs more efficient inference. Furthermore, we also performed pruning on GPT, Llama, and Gemma under BackSlash, and the maximum pruning rates for them maintain accuracy all 90% while the normal training models start lose their effectiveness at pruming rates less than 60%, which is similar to the BERT model.
6. Conclusion and Future Work
We propose BackSlash, a training framework for LLMs that jointly optimizes model size and performance. We found that LLM parameters can be well modeled with quantized GG sources of shape parameters less than and can be entropy coded with extremely high efficiency and robustness using EG codes. Experiments with popular LLMs show that BackSlash was capable of reducing model size by up to 80% with virtually no loss in performance.
5.4. Ablation
As discussed in Section 4.2, L1 and Le regularizations are special cases of DGGR assuming model parameters follow Laplace and Gaussian Distribution, respectively. So whether such shape-specific Lp regularization terms can effectively substitute DGGR in the BackSlash framework warrants further investigation.
Currently, we are conducting more experiments with more LLMs and tasks. The optimal setting of À is also under investigation, well as efficient hardware architecture that can take advantage of the increased sparseness of the model in more efficient training and inference.
We perform the sentiment analysis task on BERT with Back- Slash using DGGR, Lo.5. L1, L2 respectively, to evaluate
9

---

BackSlash: Rate-Constrained Optimized Training of Large Language Models
Impact Statement
Choi, Y., EI-Khamy, M., and Lee, J. Towards the limit of network quantization. ArXiv, abs/1612.01543,, 2016. URL htttp:::/ap..semanttcccoooot www.nti azg/CorppsID:11229000
This paper introduces fundamentally new approach to training large models. Instead of using standard backpropa- gation train a large model and compressing afterward, BackSlash framework integrates efficiency directly into the training process to produce small and easy-to-deploy models. This framework can significantly influence how the next-generation foundation models are trained and deployed, both in software and hardware.
Courbariaux, M., Bengio, Y., and David, J.PP. Bina- ryconnect: Training deep neural networks with bi- nary weights during propagations. In Neural Informa- tion Processing Systems, 2015. URL https: /api. s Semantisscholle org/ azggCorppsID::.. 1518844..
Cover, T. M. Elements of information theory, John Wiley & Sons, 1999.
References
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P. Lin, J., Zhou, C., and Zhou, J. Qwen- vl: A versatile vision-language model for under- standing, localization, text reading, and beyond. 2023. URL https::////// wtwppppssmaatisccch.eeeeooooeeee atg/Co=pusSD::6110I.
Davisson, L. Rate distortion theory: A mathematical basis for data compression. IEFE Transactions on Communi- cations, 20(6):1202-111) 1972.
Fitriani, S. A., Astuti, Y., and Wulandari, I. R. Least abso- lute shrinkage and selection operator (lasso) and k-nearest neighbors (k-nn) algorithm analysis based on feature selection for diamond price prediction. In 2021 Inter- national Seminar on Machine Learning, Optimization, and Data Science (ISMODE), pp. 135-139, 2022. doi: 10.1 1AL1100SOOEE33542200.....1.
Berger, T. Rate-distortion theory. Wiley Encyclopedia of Telecommunicaatios, 2003.
Brand, F. Fischer, K., Kopte, A., Windsheimer, M., and Kaup, A. Rdonet: Rate-distortion optimized learned image compression with variable depth. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1759-1763, 2022.
Fortuin, V., Garriga-Alonso, A., Wenzel, F., Rätsch, G., Turner, R. E., van der Wilk, M., and Aitchi- son, L. Bayesian neural network priors revisited. ArXiv, abs/2102.06571, 2021. URL https:
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Ka- plan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., teusz Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I,, and Amodei, D. Language mod- cls are few-shot learners. ArXiv, abs/2005.14165. 2020. URL https:/ http://app...aaaaii.o.s www.nticsccooll.........c ott//CorpusID::1891133
231918454.
Gao, W. Wang, C., and Oh, S. Rate distortion for model compression: From theory practice. In International Conference on Machine Learning, 2018. URL pattps://api... semanti wwwanticacholar.......e.aaa org/ htss//ooppIID:33110000
Glorot, X. and Bengio. Y. Understanding the difficulty of training deep feedforward neural networks. In Pro- ceedings of the thirteenth international conference on artificial intelligence and statisties, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Chen, Y., Wang. N., and Zhang, Z. Darkrank: Accelerating deep metric learning via cross sample similarities transfer. ArXiv, abs/1707.01220, 2017. URL https: //api. semanti www.nticcchllllc aar .ow..org///oo wwwp/rrrrooeeeeeoo CorpusID:: 19207026.
Gong, Y., Liu, L., Yang, M., and Bourdev, L. D. Compress- ing deep convolutional networks using vector quantiza- tion. ArXiv, abs/1412.6111,, 2014. URL https: /apii. semant iomanticsconn www.occccccaa.....c..cc.c. org/ http://orpuuSDD 6251 653.
Chen, Y., Wang, S., Ip. H., and Kwong, S. Rate distortion optimization with adaptive content modeling for random- access versatile video coding. Information Sciences, 645; 119325, 2023.
Guo, H., Zhu, C., Ye, M., Luo, L., and Yang. X. Pre- encoding based temporal dependent rate- distortion opti- mization for heve. Signal Processing: Image Communi- cation, 1 115: 1 16957, 2023.
Chiang, J.-C., Shang, H.-Y., and Qiu, J.-J Multi-exposure image compression considering rate-distortion optimiza- tion in rendered high dynamic range image. IEEE Open Journal of Signal Processing, 4:132-147, 2023.
Han, S., Mao, H., and Dally, W. J. Deep compression: Com- pressing deep neural network with pruning, trained quan- tization and huffman coding. arXiv: Computer Vision
10

---

BackSlash: Rate-Constrained Optimized Training of Large Language Models
and Putlern Recognition, 2015a. URL https://app semanti weweiiicschhlar...o wwwplllr.rrrrooooeee :ttp://Coruuut::13333::3 ott.e//aieaii...a.aroooooooorraa
Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, W. J. Deep gradient compression: Reducing the communication bandwidth for distributed training. ArXiv, abs/1712.01887, 2017. URL https: /api. s semant i cacchlccharr ar org.orrg///ooooo Cttp/CrrpuuSTD 387966293
Han, S., Pool, J., Tran, J. and Dally, W. J. Learn- ing both weights and connections for efficient neural network. In Neural Information Processing Systems, 2015b. URL https: htt.p/p...mmaaaacccc.ooooooeooo. cttp/CorpuID::22::000 2238772.
Liu, Z.. Cheng, K.-.., Huang, D., Xing, E. P., and Shen, Z. Nonuniform-to-unifort quantization: Towards ac- curate quantization via generalized straight-through es- timation. 2022 IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 4932-4942, 2021. URL https: / www.aap....aiiiicco... semanticscholar.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. 2015 IEEE International Con- ference on Computer Vision (ICCV), pp. 1026-1034, 2015. URL https:/ http://pp..eeaaaticcccacooeaeeee org/Co:pusID:1374000
Long, X., Zeng, X., Ben, Z., Zhou, D., and Zhang, M. A novel low-bit quantization strategy for compressing deep neural networks. Computational Intelligence and Neuroscience, 2020(1):7330644, 2020.
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J,, and Han, S. Ame: Automl for model compression and accel- eration on mobile devices. In European Conference on Computer Vision, 2018. URL https://appp semanticscholes org/ Sarg//orree /Corpoo 52048008..
Luo, J.-H. Wu, J., and Lin, W. Thinet: A filter level pruning method for deep neural network compression. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5068-50787 2017. URL https: /api. semant cschol ar www.aroowoooooooooooooooo
Hoerl, A. E. and Kennard, R. W. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 1970.
Luttrell, M., Wen, J., and Villasenor, J. D. Trellis-based rd optimal quantization in h. 263+. In Proceedings 2000 International Conference on Image Processing (Cat. No. 00CH37101), volume 2, pp. 352-854. IEEE, 2000.
Isik, B., No, A., and Weissman, T. Successive prun- ing for model compression via rate distortion the- ory. ArXiv, abs/1122.0839,, 2021. URL https: //api. semanticscholar. wttp:/opr. 231933836.
Masana, M., van de Weijer, J., Herranz, L., Bagdanov, A. D., and Álvarez, J. M. Domain-adaptive deep network compression. 2017 IEEE International Con- ference Computer Vision (ICCV), pp. 4299-4307, 2017. URL https: /tte:///pptt /api. semanti ierranticcchh cscholas org/ Cts/CCrpusI::IIn :11557:2111999
Iu-T and Jtc, I. Advanced video coding for generic audiovisual services. 2010. URL https://api. semanticscholod ooallar.rrggeeee :otp/oorossI: 6033566447
Jaderberg, M., Vedaldi, A., and Zisserman, A. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv: 140..386, 2014.
Park, J.-H., Kim, Y., Kim, J., Choi, J.-Y., and Lee, S. Dynamic structure pruning for compressing cnns. ArXiv, abs/2303.09736. 2023. URL https:
Kossaifi, J., Bulat, A., Tzimiropoulos, G., and Pantic, M. T-net: Parametrizing fully con- volutional nets with a single high-order tensor. 2019 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 7814-7823, 2019. URL https://api. htt.ts///pppiimmmmmiiiiooo sergiisemmnnticccch.e.
/ sendssmmmaiiccchole... org org/CorpusID: 257622926.
Rastegari, M., Ordonez, V., Redmon, J., and Farhadi,
A. Xnor-net: Imagenet classification using binary con- volutional neural networks. ArXiv, hts/1603.05579,
2016. URL https: //ap:///pppp semanticschole org/ http/CoopuID::1422000
Li, F. and Liu, B. Ternary weight networks. ICASSP
2023 2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. 1-
Shannon, C. E. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
5, 2016. URL httph:tt/////....t. /api...... Drr/CorpusID:13356113
htt.app..aaaaa.eee.eeeaeeaaeaaae semant icscholr..
Sharifi, K. and Leon-Garcia, A. Estimation of shape param- eter for generalized gaussian distributions in subband decompositions video. JEEE Trans. Circuits Syst. Video Technol, 5:52-56, 1995. URL httpt:://aa// semattiicccholaarrro/oooooooopp
Li, H. Kadav, A., Durdanovic, I. Samet, H., and Graf, H. P. Pruning filters for efficient convnets. ArXiv, abs/1608.08716, 2016. URL https://api. semant. wwcholaseee...eeoeeeee wwtpe//eeeeeerrrrrrrrrrssaaaa100

---

BackSlash: Rate-Constrained Optimized Training of Large Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Ham- bro. E., Azhar, F,, Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023. URL https:/ http:////aa.ssmaaisssssss. semanticscholl.. Tttp/ooopuIID:25720 9404.
swwanticscoollae..c.... org/ http//Cou::::2255550002
2023. URL ntttps://api.
Zhang. Z., Lu, G., Liang, H., Tang, A., Hu, Q. and Song, Efficient dynamic-nerf based volumetric video cod- ing with rate distortion optimization. arXiv preprint 2024.
Zhou, S., Ni, Z., Zhou, X., Wen, H., Wu, Y., and Zou, Dorefa-net: Training bitwidth convo- lutional neural networks with low bitwidth gradients. ArXiv, abs/1606.06160, 2016. URL https://api. semanticscholat htgp/aa..rrrrrroeeeeee
Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S. Haq:
Hardware-aware automated quantization with mixed pre- cision. 2019 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 8604-8612, 2018a. URL https: http::/:s/app...maaniiii.i. www.ssmttisccooll............. org/ http//cppSD::1025500...
Zhu, C., Han, S., Mao, H., and Dally, J. Trained ternary quantization. ArXiv, abs/1612.01064.
Wang, P., Hu, Q. Zhang, Y., Zhang, C., Liu, Y., and Cheng, J. Two-step quantization for low-bit neural networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4376-4384, 2018b. doi: 10. 1109/CVPR.20188 8.00600.
2016. URL https://aat semantiscchho.....
Deg/CorpusID:22483..
Wang, Y., Xu, C., Xu, C., and Tao, D. Beyond fil- ters: Compact feature map for portable deep model. In International Conference on Machine Learning, 2017. URL https:/ http:/::/aap...aaaa..ssssss org (tcCCoppsI:::21995000.. sem......aatccsccool.......
Wen, J. and Villasenor, J. Structured prefix codes for quan- tized low-shape-paranee parameter generalized gaussian sources. IEEE Transactions on Information Theory, 43(4):1307- 1314, 1999. doi: 10.1 109/189 7.11288..
Wien, M. High efficiency video coding, Coding Tools and specification, 24:1, 2015.
Xia, C.-G., Tsang, D. H.-K., and Lau, V. K. N. Struc- tured bayesian compression for deep neural networks based on the turbo-vbi approach. IEEE Transac- tions on Signal Processing, 71:670-685, 2023a. URL https: www.nicsccohca...........m...... CorpusID:257050722.
Xia, F,, Jin, J., Meng, L., Ding, F., and Zhang, H. Gan- based image compression with improved rdo process. In International Conference on Image and Graphics, pp. 361-372. Springer, 2023b.
Xu, D., Ouyang, W., Wang, X., and Sebe, N. Pad-net: Multi-tasks guided prediction aandiittionanddddiillatttio network for simultaneous depth estimation and scene parsing. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 675-684, 2018. URL https:::p: ap.. semanticschon org/ Corpus wwt.:oooooooooooooo.....o 21670200
Zhai, P. Guo, K., Liu, F,, Xing, X., and Xu, X. Lapp: Layer adaptive progressive pruning for com- pressing cnns from scratch. ArXiv, abs/2309.14157,
12


Reviews:
[{"review_id": "0PmSdDk3qz", "reviewer": "Reviewer_qwnZ", "summary": "The paper introduce Rate-Constrained Training (RCT), a method integrating rate-distortion optimization into the training process of Large Language Models (LLMs). RCT leverages a generalized Gaussian (GG) distribution to accurately model LLM parameters and uses exp-Golomb (EG) coding for entropy-efficient parameter encoding. The method dynamically optimizes both model performance and compression rate during training, and reduces model size (60%-90% memory reduction) while maintaining or slightly improving accuracy.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment #1: The optimal selection of the Lagrange multiplier (λ) is pretty empirical, there is no theoretical guidance or any adaptive strategies to systematically select λ across different tasks.**\n\n**Response:** We thank the reviewer the valuable feedback. Indeed the theoretical foundation for optimal setting of $\\lambda$ is critical and currently under investigation. In our extensive experiments however we found that the same empirical value could be used for different models and tasks to achieve the best results.\n\n\n**Comment #2: Shape parameter estimation and rate calculations within every training step introduces non-trivial computational overhead:**\n\n**Response:** We thank the reviewer for the comment. In our experiments, statistics for estimating the shape parameter are collected during training of the previous round, incurring little additional overhead. Overall, current implementation of RCT fine-tuning of BERT on A100 GPUs using the IMDB dataset required 8.63 min/epoch, compared with conventional training at 7.92 min/epoch. This overhead stems mainly from DGGR gradient descent, as shape parameter evaluation added negligible time. Further optimization is possible, such as storing quantized DGGR gradients and replacing explicit computations with lookup tables.\n\n\n**Comment #3:** **Although partially addressed via soft gradient clipping, instability in gradient computation remains a concern, especially for the cases when v is close to 1.**\n\n**Response:** Thanks to reviewer for highlighting potential gradient instability in soft gradient descent. Our retraining experiments with/without RCT show that soft gradient clipping effectively mitigates unstable gradients' impact. For instance, with clipping, RCT achieves **mean=1.70 (std=2.18)** versus normal training's **mean=1.89 (std=1.23)**. Extensive preliminary experiments confirm that gradients did not cause significant performance fluctuations. Should soft gradient clipping fail, other means such as fixed value gradient clipping exist.\n\n\n**Comment #4:** **Could the authors elaborate on the computational complexity introduced by the required \"value mapping\" before entropy coding? and provide insights into practical implementations.**\n\n**Response:** Value mapping converts parameters to sorted indices via hash tables, with O(N) time/space complexity and the process can be highly parallelizable. In our serial CPU implementation, the \"value mapping\" step for 110M parameters takes 0.16s, while the entire encoding process consumes 3.20s in total.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "A novel approach to integrate rate distortion optimization methodology to the LLM training.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "strengths:\n\n1. Introduces a novel rate-distortion optimization framework integrated directly into the training phase, addressing a significant gap in existing compression techniques.\n2. Experiments show that LLM parameters follow a generalized Gaussian distribution (shape parameter v < 2), significantly improving modeling accuracy over traditional Gaussian assumptions.\n3. Employs exp-Golomb codes which closely approach entropy limits, are robust to parameter distribution changes, and simplify implementation on hardware and software platforms.\n\nweakness:\n1. The optimal selection of the Lagrange multiplier (λ) is pretty  empirical, there is no theoretical guidance or any adaptive strategies to systematically select λ across different tasks.\n2. Incorporating iterative shape parameter estimation and rate calculations within every training step introduces non-trivial computational overhead, more discussions would be needed.\n3. although partially addressed via soft gradient clipping, instability in gradient computation remains a concern, especially for the cases when v is close to 1.", "other_comments_or_suggestions": "update after rebuttal: I appreciate the responses from the author. I'd like to keep my score.", "questions_for_authors": "Could the authors elaborate on the computational complexity introduced by the required \"value mapping\" before entropy coding? and provide insights into practical implementations", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JZwfmUiicB", "reviewer": "Reviewer_wovu", "summary": "The paper presented Rate-Constrained Training (RCT) for Large Language Models (LLMs), exploring model compression in the training stage. The paper showed that parameters of representative LLMs typically followed generalized Gaussian instead of vanilla Gaussian. The paper further enforced the distribution constraints (DGGR) for model training, which eased the parameter entropy encoding with exp-Golomb (EG) codes. RCT demonstrated promising compression performance with different model architectures and parameter scales.\n\n## Update after rebuttal\n\nThe reviewer appreciated the authors’ rebuttal that resolved some of the concerns, e.g. comparisons to vanilla training + vanilla huffman coding, L1/L2 regularization. The reviewer therefore raised the score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment #1: Baseline: include baselines like, vanilla training + vanilla Huffman coding, etc.**\n\n**Response:** We greatly appreciate the suggestion, and will incorporate Huffman coding as a baseline. We found that although Huffman coding shows a small advantage in efficiency over EG when applied to compressing model parameters trained using conventional training, there is no advantage of using Huffman code during RCT. At the same time, Huffman code will need to be designed for each round of training and for each model, with $O(N)$ complexity that cannot be parallelized, while EG has very simple software and hardware implementations and can be applied to all models and tasks. **In all our experiments, the same EG0 code was used.**\n\n| Model | EG   | Huffman | FL    |\n| ----- | ---- | ------- | ----- |\n| BERT  | 2.64 | 2.42    | 10.00 |\n| GPT   | 2.46 | 2.25    | 11.00 |\n| Llama | 1.72 | 1.66    | 10.00 |\n| Gemma | 1.16 | 1.15    | 11.00 |\n\n| Task      | EG   | Huffman | FL    |\n| --------- | ---- | ------- | ----- |\n| Sentiment | 2.64 | 2.42    | 10.00 |\n| Spam      | 2.42 | 2.19    | 10.00 |\n| Topic     | 3.61 | 3.18    | 10.00 |\n| Q-A       | 2.90 | 2.81    | 11.00 |\n\n**Comment #2: Ablations: dynamic shape parameter vs vanilla L1/L2 regularization.**\n\n**Response:** We are grateful for your valuable suggestion and added L0.5, L1, and L2 to BERT fine-turning on IMDB, as shown below. In principle, L0.5, L1, and L2 need to assume “the parameters obey GG shapes of 0.5, 1, and 2”, while in reality the shape parameters of the parameters were below 0.22. The latest deepseek-7B model has a shape parameter of **0.85**. Such differences call for taking shape parameter into consideration during RCT, which achieved good results.\n\n|      | Final Shape Parameter | Accuracy | EG   | FL    |\n| ---- | --------------------- | -------- | ---- | ----- |\n| DGGR | 0.13                  | 91.18%   | 1.37 | 10.00 |\n| L0.5 | 0.22                  | 91.88%   | 2.90 | 10.00 |\n| L1   | 0.15                  | 90.65%   | 1.52 | 10.00 |\n| L2   | 0.10                  | 88.29%   | 1.16 | 10.00 |\n\n**Comment #3: Architectures: The readers may wonder if RCT scales/generalizes well.**\n\n**Response:** We thank the reviewer for pointing out this. **Since submitting the paper, we conducted experiments with the latest DeepSeek model and obtained similar results**. We will include the results in the final manuscript if accepted. **Below is the result from text generation using deepseek-7B.** This further confirmed the viability of RCT. Again, Huffman coding has an advantage over EG when used to compress a model that have already been trained, at additional traning and implementation costs. Whereas for RCT training, there is virtually no difference. Additionally, the measured shape parameter (**0.85**) of deepseek-7B **further validates the necessity of shape parameter estimation.**\n\n|                     | **FL** | **EG** | **Huffman** | **EG Compress** | **Huffman Compress** | **Accuracy** |\n| ------------------- | ------ | ------ | ----------- | --------------- | -------------------- | ------------ |\n| **Normal Training** | 11.00  | 5.93   | 4.70        | 46%             | 57%                  | 99.97%       |\n| **RCT**             | 11.00  | 2.90   | 2.81        | 74%             | 74%                  | 99.97%       |\n\n**Comment #4: Presentation: The authors are encouraged to provide also the actual numbers shown in Figures.**\n\n**Response:** We thank the reviewer for the suggestion and will annotate each node in Figs. 3-4 with specific data values for clarity as shown below.\n\n| **Lagrange multiplier** | **0**  | **1**  | **10** | **100** | **1000** |\n| ----------------------- | ------ | ------ | ------ | ------- | -------- |\n| **EG Code**             | 7.31   | 6.66   | 5.78   | 4.03    | 2.64     |\n| **Huffman Code**        | 5.47   | 5.25   | 4.81   | 3.59    | 2.42     |\n| **Fixed Length Code**   | 10.00  | 10.00  | 10.00  | 10.00   | 10.00    |\n| **Train Dataset**       | 99.54% | 99.69% | 99.56% | 99.47%  | 99.52%   |\n| **Test Dataset**        | 93.63% | 93.85% | 93.93% | 93.68%  | 92.59%   |\n\n**Comment #5: L205, right column, \"difficult\" -> \"different\"?**\n\n**Response:** We thank the reviewer for catching the typo and will proof-read the paper very thoroughly if accepted.", "claims_and_evidence": "Please refer to the strengths and weaknesses", "methods_and_evaluation_criteria": "Please refer to the strengths and weaknesses", "theoretical_claims": "Please refer to the strengths and weaknesses", "experimental_designs_or_analyses": "Please refer to the strengths and weaknesses", "relation_to_broader_scientific_literature": "The literature review looks sufficient", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n\ni) The exploration of compression rate aware LLM training is interesting.\n\nii) The paper provided various analysis, e.g. shape parameter changes in training, analysis with different lambda, etc\n\niii) The paper is easy to follow\n\nWeaknesses\n\nThe main concern of the reviewer is on the evaluations\n\ni) Baselines. While RCT was claimed to consider compression during training, the parameter encoding was applied after the model was trained, similar to vanilla solutions, e.g. huffman coding. In the paper, fixed length coding was listed as the only baseline, which was insufficient. The authors are encouraged to include baselines like, vanilla training + vanilla huffman coding, etc\n\nii) Ablations.\n\nThe readers would be of interest to understand the advantages of several experimental designs, e.g. dynamic shape parameter vs vanilla L1/L2 regularization\n\niii) Model architectures.\n\nMost of the experiments/analysis were performed on BERT and classification tasks. The readers may wonder if RCT scales/generalizes well\n\niv) Presentation\n\nProviding diagrams like Figure 2-5, while helping with the interpretation, is not sufficient. For example, it wont be easy for followup works to do the comparison. The authors are encouraged to provide also the actual numbers shown in Figures.", "other_comments_or_suggestions": "L205, right column, \"difficult\"  -> \"different\" ?", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gKrvt3RlCH", "reviewer": "Reviewer_aJi5", "summary": "The paper introduces Rate-Constrained Training (RCT), which can integrate compression during the training phase using rate-distortion optimization. The authors observed that  LLM parameters follow a generalized Gaussian distribution (GG) with shape parameters less than 2. Thus, the proposed idea is to use rate-distortion theory during training, model parameters with GG distributions, and then apply exp-Golomb coding. The authors conducted experiments on various models and downstream tasks with LLMs. They demonstrate RCT can reduce memory usage by 60-90\\% without accuracy loss and is better than post-training compression.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment #1: Analysis of \"EG codes is robust with regard to parameter mismatches.\"**\n\n**Response:** We are sincerely grateful to the reviewer for providing such valuable advice. Table. 3 demonstrates 0-order EG's optimality across models with varying shapes, confirming its robustness. Theoretical analysis can be found in Wen & Villasenor, https://ieeexplore.ieee.org/document/761289.\n\n**Comment #2: Analysis of** **\"different regulations during training may impact parameter distribution\".**\n\n**Response:** We greatly appreciate your helpful suggestion, and agree with the advice. We added L0.5, L1, and L2 to BERT fine-turning using the IMDB shown below. The original BERT has shape 1.36 and was fine-tuned to 0.22, 0.15, and 0.10 under L0.5, L1, and L2 regulations, with differences in size and performances. This shows the importance of shape parameter estimation. The shape parameter of the latest deepseek-7B model is **0.85**, also significantly different from 0.5, 1, or 2.\n\n|   |   |   |   |   |\n|---|---|---|---|---|\n||Shape|Accuracy|EG|FL|\n|DGGR|0.13|91.18%|1.37|10.00|\n|L0.5|0.22|91.88%|2.90|10.00|\n|L1|0.15|90.65%|1.52|10.00|\n|L2|0.10|88.29%|1.16|10.00|\n\n**Comment #3:** **The soft gradient clipping ($\\epsilon$) is not evaluated.**\n\n**Response:** We are sincerely grateful for your valuable suggestion regarding $\\epsilon$. $\\epsilon$ prevents gradient explosion when $\\nu<1$ and $1/\\epsilon$ is the threshold so 1 was an empirical choice. Our experiments with $\\epsilon=0.1,1,10$ show $\\epsilon=0.1$ brought unstable gradient, $\\epsilon=10$ led to slow compression and $\\epsilon=1$ was a good compromise.\n\n**Comment #4:** **Limited discussion on convergence time.**\n\n**Response:** Thank you for pointing out this problem. Number of RCT convergence’s rounds relates to the selection of $\\lambda$. We trained for 10 epochs in Fig.2 and Fig.3. $\\lambda=1000$ converges at 700 steps (3 epochs). Smaller $\\lambda$ took longer.\n\n**Comment #5:** **Run some experiments on generation tasks.**\n\n**Response:** Thank you for the suggestion. **We added a text generation experiment with the latest deepseek-7B and evaluate it by next token accuracy below**. Huffman and EG code have the same code length with RCT, but the implementation of EG code is more efficient. Furthermore, the shape parameter of deepseek-7B is **0.85.**\n\n|                     | **FL** | **EG** | **Huffman** | **EG Compress** | **Huffman Compress** | **Accuracy** |\n| ------------------- | ------ | ------ | ----------- | --------------- | -------------------- | ------------ |\n| **Normal Training** | 11.00  | 5.93   | 4.70        | 46%             | 57%                  | 99.97%       |\n| **RCT**             | 11.00  | 2.90   | 2.81        | 74%             | 74%                  | 99.97%       |\n\n**Comment #6:** **No comparison to other entropy coders.**\n\n**Response:** We found that although Huffman coding shows a small advantage in efficiency over EG when applied to compressing model parameters trained using conventional training, there is no advantage of using Huffman code during RCT. At the same time, Huffman code will need to be designed for each round of training and for each model, with $O(k\\log k)$ complexity that cannot be parallelized, while EG has very simple software and hardware implementations and can be applied to all models and tasks. **In all our experiments, the same EG0 code was used.**\n\n| Model | EG   | Huffman | FL    |\n| ----- | ---- | ------- | ----- |\n| BERT  | 2.64 | 2.42    | 10.00 |\n| GPT   | 2.46 | 2.25    | 11.00 |\n| Llama | 1.72 | 1.66    | 10.00 |\n| Gemma | 1.16 | 1.15    | 11.00 |\n\n| Task      | EG   | Huffman | FL    |\n| --------- | ---- | ------- | ----- |\n| Sentiment | 2.64 | 2.42    | 10.00 |\n| Spam      | 2.42 | 2.19    | 10.00 |\n| Topic     | 3.61 | 3.18    | 10.00 |\n| Q-A       | 2.90 | 2.81    | 11.00 |\n\n**Comment #7:** **No ablation study on EG's contribution and RCT solely.**\n\n**Response:** We thank the reviewer for the feedback. We will incorporate more comparisons with RCT using Huffman coding, conventional training (no entropy coding), L0.5, L1, L2 regulation to demonstrate the usefulness of EG.\n\n**Comment #8: (1) Contributions are too long. (2) \"other works\" misses citations. (3) tick -> trick, difficult -> different. (4) Figure 6 hard to interpret. (5) Describe the ν estimation in Algo.1. (6) why omit $1/N_p$ in Eq7? (7) Is the** $\\alpha$ **in line 320 the same as it in Eq2?**\n\n**Response:** We are extremely grateful to the reviewer for catching these issues. We will fix these typos and clarify throughout the paper if it is accepted.\n\n**Comment #9:** **The selection of $\\lambda$ may be sensitive to different tasks, but there are no experiments and analysis.**\n\n**Response:** We are sincerely grateful to the reviewer for the comment. We agree that the optimal setting of $\\lambda$ needs more research, which we are currently doing. In our extensive experiments it was set to the same empirical value to achieve the best results.", "claims_and_evidence": "The main claims are largely supported by theoretical and empirical results.\n\nWeaknesses:\n1. Authors claim \"EG codes is robust with regard to parameter mismatches.\" in lines 61-62, there is no analysis or experimental results on it.\n2. The authors mentioned that \"different regulations during training may impact parameter distribution\" in lines 110-112. I would like to see any experimental support or observations.\n3. The soft gradient clipping($\\epsilon$) is not evaluated.", "methods_and_evaluation_criteria": "Yes, but there are weaknesses:\n1. Limited discussion on convergence time\n2. Only evaluate classification tasks; it is better to run some experiments on generation tasks.", "theoretical_claims": "Equations and algorithms are sound.", "experimental_designs_or_analyses": "Weaknesses:\n1. No comparison to other entropy coders in the same setup.\n2. No ablation study on EG's contribution and RCT solely\n3. The selection of $\\lambda$ may be sensitive to different tasks, but there are no experiments and analysis.", "relation_to_broader_scientific_literature": "The paper synthesizes ideas from information theory, statistics, and deep learning into a novel framework. It can be a bridge between classical compression theory and LLMs optimization.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Comments on writing-related:\n1. The contributions are too long; it is better to summarize the main contributions in very few sentences.\n2. Line 85 (section 2.2), \"other works\" are missing citations.\n3. line 164: tick -> trick, line 206: difficult -> different?\n4. Figure 6 is hard to interpret. Please consider enlarging the critical part or normalizing the values for better visualization. \n5. I think it is better to explicitly describe the ν estimation step in Algo 1.", "questions_for_authors": "See the above weaknesses. \n\n1. why omit $1/N_p$ in Eq7?\n2. is the $\\alpha$ in line 320 the same as $\\alpha$ in Eq2? If not, please check the notions carefully.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "LK5GShLnav", "reviewer": "Reviewer_br8x", "summary": "This work proposes Rate-Constrained Training (RCT), a novel training method that allows for training LLMs in a way that allows for effective compression of their weights at the end of training. The main idea is to derive a weight regularizer by assuming a specific distribution over the model weights, the generalized Gaussian (GG) distribution (with a shape parameter that is adapted during training). The authors argue for this choice empirically by showing how the training model weights have a better fit under that distribution. This regularizer essentially becomes an $L_p^p$ norm for the weights of the network, with the $p$ being dynamically updated during training. After model training, the authors use exp-Golomb (EG) codes in order to compress the resulting weights. The argument for this choice is that the EG codes can achieve almost the entropy limit for GG sources. The authors experiment with RCT and various LLM architectures and simple classification tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment #1: Evaluation against the Gaussian model and the Laplace model.**\n\n**Response:** We thank the reviewer for valuable suggestion. We have conducted experiments with L0.5, L1, L2 regulations, the latter two correspond to the Gussian and Laplacian models. As can be seen, RCT with EG achieved the best accuracy and compression. **We also found that the** **shape parameter of the latest deepseek-7B is 0.85, also significantly different from 0.5, 1, or 2.**\n\n|| Final Shape Parameter | Accuracy | EG Code | FL Code |\n| ---- | --------------------- | -------- | ------- | ------- |\n| DGGR | 0.13| 91.18%   | 1.37    | 10.00   |\n| L0.5 | 0.22| 91.88%   | 2.90    | 10.00   |\n| L1   | 0.15| 90.65%   | 1.52    | 10.00   |\n| L2   | 0.10| 88.29%   | 1.16    | 10.00   |\n\n**Comment #2: It is unclear what the \"normal training\".**\n\n**Response:** We thank the reviewer for the comment. “Normal training” refers to \"fine-tuning models with only cross entropy\".\n\n**Comment #3: I would have expected some other tasks to be used.**\n\n**Response:** Many thanks to the reviewer for this advice. **If accepted we will also add results from text generation experiment using the latest deepseek-7B and evaluate performance by next token accuracy.** Results show 74% compression using RCT and EG0 over an already highly optimized model. In addition, EG0, the same code used for BERT, Llma, and GPT achieved nearly identical coding efficiency as Huffman, which required higher complexity. Additionally, the measured shape parameter (**0.85**) of deepseek-7B **further validates GG distribution assumption.**\n\n|| **FL** | **EG** | **Huffman** | **EG Compress** | **Huffman Compress** | **Accuracy** |\n| ------------------- | ------ | ------ | ----------- | --------------- | -------------------- | ------------ |\n| **Normal Training**|11.00|5.93| 4.70| 46%|57%|99.97%|\n| **RCT**|11.00|2.90| 2.81| 74%| 74%| 99.97%|\n\n**Comment #4: $\\delta$ can be quite large in practice.**\n\n**Response:** We appreciate the feedback. $\\delta$ represents the quantization step and Fig. 1 shows almost all parameters are concentrated in $[-0.2,0.2]$. So $\\delta$ may be much smaller (such as $2^{-8}$ or $2^{-16}$) than 2. We will clarify in the paper if accepted.\n\n**Comment #5: Is the adaptivity of the shape parameter necessary for good compression?**\n\n**Response:** We are profoundly thankful to the reviewer for the comment. As shown in Comment #1, experiments with L0.5, L1 and L2 and the DGGR show that adapting shape parameter did lead to performance and compression gains.\n\n**Comment #6: It is unclear if those indices actually follow the desired discretized GG distribution.**\n\n**Response:** Many thanks to the reviewer for this advice, it is very valuable to us. We checked the distribution of indexes and found they does follow the GG, but its shape is slightly different due to value mapping. For example, the shape of parameters and index of BERT with normal training is 1.36 and 1.47; RCT is 0.26 and 0.30. It will not affect the entropy coding because “**EG code is robust to GG sources of various shapes**”.\n\n**Comment #7: The references do not assume that weights are Gaussian after training.**\n\n**Response:** We thank the reviewer for the feedback. We revised the description in Section 3 “**Most research assumes that LLM parameters follow the Gaussian distribution in the initialization and rarely discussed the distribution after training.**”, and cited reference [1] in paper. Thank you again for such valuable argument.\n\n**Comment #8: It is unclear what the bit-width is for the experiments at Fig.7.**\n\n**Response:** Many thanks to the reviewer for this advice. We sincerely apologize for our omission and provide the bit-weight below.\n\n| **Quantization Step** | $2^{-4}$ | $2^{-8}$ | $2^{-12}$ | $2^{-16}$ | $2^{-20}$ | $2^{-24}$ | $2^{-28}$ | $2^{-32}$ | $2^{-36}$ |\n| --------------------- | -------- | -------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| **Normal Training**|7|10|13|16|19|23|26|26|26|\n| **RCT**|7|10|13|16|19|22|25|25|26|\n\n**Comment #9: One would need to retrain the (expensive) LLMs multiple times for finding the desired tradeoff.**\n\n**Response:** We greatly appreciate your helpful suggestion. Indeed the setting of $\\lambda$ needs more research. However, retraining (expensive) LLMs many times is not necessary, since $\\lambda$ can be **adjusted** in training. Specifically, during training, if the shape of the distribution remains large, $\\lambda$ can be adjusted. We also found that the value could be set to the same empirical one to achieve the best results in our experiments.\n\n**Comment #10: It would be good to show the Gaussian fit at Figure 1 for comparison purposes.**\n\n**Response:** We are profoundly thankful to the reviewer for bringing this issue to our attention and we have added the Gaussian fit and GG fit to Fig. 1.", "claims_and_evidence": "The paper claims are moderately supported by the evidence. While the authors do show improvements upon \"normal training\" with RCT, I think what is missing is a more convincing evaluation against the Gaussian model (or alternative the GG model with a shape parameter of two) and the Laplace model (i.e., the GG model with a shape parameter of one), which are two most important baselines. Furthermore, it is unclear what the \"normal training\" for the models presented at, e.g., Table 4 is, which makes the interpretation of the results more difficult. For example, since the authors assume that the Gaussian model is widely used, I would expect an evaluation against a \"normal training\" run that uses weight decay for the parameters (along with codes that are appropriate for such a distribution).", "methods_and_evaluation_criteria": "The evaluation criteria make sense, but, given that LLMs are more or less used as general purpose models, I would have expected some other tasks, besides classification, to be used (such as language modeling with next-token prediction).", "theoretical_claims": "I checked the mathematical details and they were mostly ok, albeit with some assumptions that do not necessarily hold true in practice (e.g., for the transition from Eq. 3 to Eq. 4). More specifically, the argument that $\\delta$ is typically small only makes sense when we have a large bit-width. When that is small, e.g., 2, that $\\delta$ can be quite large in practice. As a result, the approximation at Eq. 4 might be off in such cases. Why did the authors not use the GG CDF for proper discretization into a quantization grid?", "experimental_designs_or_analyses": "The experimental design is ok but this work is missing critical baselines as mentioned above. More specifically:\n\n* A \"normal training\" run where a model is trained with an L2 regularizer (i.e., weight decay) \n* A more standard \"sparse training run\" where a model is trained with an L1 regularizer \n\nBesides those results, I am also wondering a couple of other things:\n* Is the adaptivity of the shape parameter necessary for good compression? Since you train the model from scratch, the parameters can adapt to the specific $L_p^p$ penalty you employ and thus can be easily compressed via appropriate codes for that norm. \n* The authors apply entropy coding to the indices of quantized values (page 5, line ~240 second column), however it is unclear if those indices actually follow the desired discretized GG distribution. I would have expected some analysis there.", "relation_to_broader_scientific_literature": "The two main novel contributions of this work against the broader literature are\n\n* Proposing an $L_p^p$ norm as a regularizer for the weights during training of the model with a $p$ that is automatically updated during training. This is a relatively minor contribution but a novel one nonetheless.\n* Using EG codes for compression after training. This is the main novel contribution of the work relative to prior art.", "essential_references_not_discussed": "Nothing as far as I am aware of.", "other_strengths_and_weaknesses": "Overall, I found the paper interesting and the combination of the GG distribution and the EG codes in the LLM setting is novel and yielded good results in the experiments shown. The authors also considered several architectures during the evaluation, which is another bonus.\n\nOne more weakness that the authors could work on is (besides what mentioned above) the argument for the extensive application of the Gaussian distribution. It is based on old references about Gaussian initialization of neural network weights. One critical point here is that in those works the weights are only **initialized to be Gaussian** and do not assume that they are Gaussian after training. In fact, the weights after training become more heavy-tailed (which also acts a supporting argument as to why GG distributions with shape < 2 makes sense) [1]. It would be good if the authors provide more supporting evidence for this claim. \n\n[1] Fortuin et al., Bayesian neural network priors revisited, https://openreview.net/pdf?id=xkjqJYqRJy", "other_comments_or_suggestions": "* It is unclear what the bit-width is for the experiments at Fig.7; this would allow for better understanding the effects of the quantization steps. \n* While I agree that training a model for a specific RD will yield better benefits at the end, it is a bit impractical for large scale model training as one would need to retrain the (expensive) large model multiple times for finding the desired tradeoff (e.g., to pick an appropriate $\\lambda$). This is why in practice, post-training compression is more desirable.\n* It would be good to show the Gaussian fit at Figure 1 for comparison purposes. \n* What is the performance if you fix the shape parameter at the specific value you found at the end of training and then train again? This would highlight whether the adaptivity of the shape parameter is necessary.", "questions_for_authors": "No other questions besides the above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The review process identified a number of missing baselines, ablation studies and the reviewers had requested a number of clarifications in their comments. The authors have responded to these concerns in their rebuttal. I would advise the authors to incorporate this discourse in the camera-ready version of the manuscript.
Decision: Accept (poster)