SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2 presents the perplexity when various quantized LLMs are fine-tuned using QAT, PEFT (+PTQ), and PEQA. To validate our approach, PEQA, which solely fine-tunes the quantization scale as outlined in Eq. 2 while simultaneously maintaining the integer matrix in a frozen state, we use QAT as an upper bound and PEFT+PTQ as a lower bound. Note that QAT, unlike PEQA, updates all parameters including pre-trained weights as well as quantization scales. Table 2 reveals the competitive performance of PEQA compared to QAT. Furthermore, our observations indicate that PEQA consistently outperforms the combination of LoRA and OPTQ for any selected model, regardless of whether a 3-bit or 4-bit setting is employed. Such superior performance can be attributed to PEQA's method of fine-tuning quantized LLMs, which minimizes the final task loss on the full training data, a capability that OPTQ lacks. Detailed settings are in Appendix B.
Diving deeper into the comparison between QAT and PEQA, it is important to note that QAT minimizes the final task loss computed from a weight-only quantized model, as described in Eq. 1, with respect to both  $W_{0}$  and  $s_{0}$ . Note that QAT includes all pre-trained weights for training, resulting in the practical model size limitation of LLMs under investigation being capped at 13B in our experiments. Despite the fact that QAT also updates  $W_{0}$  in Eq. 1, which is one of the most simple and straightforward approach though, we observe that the performance gap between QAT and PEQA narrows when the 4-bit association is introduced, especially as the size of LLMs increases. Impressively, PEQA can even outperform QAT in a 3-bit setting, a notably low-bit setting that challenges OPTQ in terms of quantizing LLMs. These findings suggest that the approach of PEQA, solely updating quantization scales while freezing the integer quantization values of pre-trained weights, can achieve performance comparable to that of QAT.
4.2 Task-specific Adaptation with Wikitext2 and PennTreeBank Datasets
Task-specific Adaptation and Scalability. We evaluate the task-specific adaptation performance and scalability of PEQA by employing GPT-Neo, GPT-J, and LLaMA models (up to 65B) on the Wikitext2 [51] and PennTreeBank (PTB) [58] datasets. The adaptation performance of PEQA is compared with LoRA, with its configuration set to QV4. As depicted in Table 3, we note a gradual convergence of PEQA's perplexity to that of full-precision LoRA, with only marginal PPL degradation as the model size expands. Thus, Table 3 demonstrates that PEQA, compared to a prominent PEFT technique that utilizes full-precision pre-trained language model (PLM), can maintain a competitive perplexity level in LLMs while concurrently reducing DRAM usage through low-bit quantized
Table 4: Number of learnable parameters and model size of GPT-Neo, GPT-J and LLaMAs. PEQA configuration is set to 4-bit or 3-bit channel-wise quantization.
Method	GPT-Neo 2.7B	GPT-J 6B	LLaMA 7B	LLaMA 13B	LLaMA 30B	LLaMA 65B
# of Learnable Param. (M)	LoRA (QV4)	1.31	1.84	2.10	3.28	6.39	10.49
LoRA (QKVO16)	5.24	7.34	8.39	13.11	25.56	41.94
PEQA (Ours)	0.74	1.03	1.36	2.13	4.15	6.80
Model Size (GB)	LoRA (QV4)	5.30	12.10	13.48	26.03	65.06	130.57
PEQA (Ours, 4-bit)	1.53	3.65	3.77	7.01	16.92	33.45
PEQA (Ours, 3-bit)	1.21	2.94	2.96	5.42	12.90	25.35
Table 5: Multi-scale (grouping) performance with PEQA-tuned LLaMA 7B and 13B on Wikitext2 where  $g$  indicates the group size [49]. The perplexity consistently increases as PEQA take on more learnable parameters.
Model	W Bits	Channel-Wise	g256	g128	g64
LLaMA 7B	4	5.84	5.69	5.66	5.64
3	6.19	5.96	5.91	5.89
LLaMA 13B	4	5.30	5.18	5.16	5.16
3	5.54	5.40	5.37	5.34
weights. Notably, for a 3-bit quantization, PEQA experiences less performance degradation as the model size decreases due to extreme low-bit quantization compared to the combined LoRA and OPTQ. To further elucidate our findings, we have provided figures illustrating the results of 3-bit and 4-bit PEQA in the Appendix D. The comprehensive results indicate that for the deployment stage, PEQA allows models with larger parameters to operate under DRAM usage constraints, outperforming full-precision PEFT methods. For instance, under a restricted DRAM footprint, large LLaMA models can be explored using PEQA, while full-precision LoRA permits only smaller LLaMA models. Additional results with OPT [4], ranging from 1.3B to 66B models are included in the Appendix E. The detailed experimental settings are also included in the Appendix C.
Model Size and Number of Learnable Parameters. In an effort to estimate the DRAM usage necessitated by PEQA and LoRA during training and deployment, we outline the number of learnable parameters (for training) and the model size (expressed in gigabytes, GB, for deployment) in Table 4. As demonstrated in Table 4, PEQA involves fewer learnable parameters than LoRA when a quantization scale is assigned to each channel of pre-trained weights. For instance, PEQA has approximately 1.54 times fewer learnable parameters for LLaMA models than LoRA (QV4). In addition to having fewer learnable parameters, PEQA, through low-bit weight quantization, can also reduce the model size, which captures a huge amount of the DRAM footprint in fine-tuning LLMs. Remarkably, when fine-tuning LLaMA 30B using PEQA with 4-bit precision, the resulting model size is significantly smaller than that obtained by adapting 13B through LoRA, and slightly larger than the model adapted from LLaMA 7B using LoRA. Additionally, a comparison of the memory peak during training between PEQA and LoRA is provided in Appendix L.
Group-wise Quantization. Group-wise per-channel quantization [49], where weight groups in channels share quantization parameters, maintains accuracy at lower bits. In Table 5, we present that the performance incrementally improves as more learnable parameters are incorporated into PEQA. In particular, for Table 5, we examine various group sizes (denoted by  $g$ ) when quantizing the weights [49, 62]. Through relatively straightforward grouping (employed to regulate the number of learnable parameters for PEQA), the perplexity incrementally decreases as more learnable parameters are utilized. Detailed settings are in Appendix G.
Table 6: Common-sense reasoning and in-context learning performance of parameter-efficient instruction-tuned LLaMAs [6] using Alpaca datasets. LoRA configuration is set to QKVO16. Quantization precision of PEQA is set to 4-bit per-channel without group size. Note that ARC-C, ARC-E and OBQA stands for ARC-Challenge, ARC-Easy, and OpenBookQA respectively.
Method	# Params	Model Size (GB)	PIQA	HellaSwag	ARC-C	ARC-E	OBQA	Average
Zero-Shot
LLaMA	7B	13.5GB	77.3	73.0	41.4	52.5	42.4	57.3
13B	26.1GB	79.1	76.2	44.5	59.9	42.2	60.4
30B	65.1GB	80.1	79.2	45.5	58.9	42.0	61.1
+ LoRA	7B	13.5GB	78.6	73.3	43.7	55.8	43.0	58.9(+1.6)
13B	26.1GB	79.6	76.7	46.3	62.0	43.2	61.5(+1.1)
30B	65.1GB	81.8	80.3	48.2	61.6	42.8	62.9(+1.8)
+ PEQA	7B	3.8GB	77.9	71.4	42.4	57.2	42.0	58.2(+0.9)
13B	7.0GB	78.9	74.0	46.4	62.5	42.8	60.9(+0.5)
30B	16.9GB	80.3	78.4	49.8	63.3	42.8	62.9(+1.8)
Five-Shot
LLaMA	7B	13.5GB	79.4	75.3	45.6	65.8	44.0	62.0
13B	26.1GB	80.0	78.4	50.4	70.8	47.2	65.4
30B	65.1GB	82.5	82.2	56.2	74.9	47.0	68.6
+ LoRA	7B	13.5GB	79.9	75.2	46.4	66.5	47.2	63.0(+1.0)
13B	26.1GB	81.1	78.8	53.5	72.4	47.0	66.6(+1.1)
30B	65.1GB	84.1	83.3	59.5	79.2	50.6	71.4(+2.8)
+ PEQA	7B	3.8GB	78.9	73.2	45.1	65.4	44.0	61.3(-0.7)
13B	7.0GB	80.7	76.0	50.9	71.6	48.0	65.5(+0.1)
30B	16.9GB	82.7	80.2	56.8	75.5	47.6	68.6(+0.0)
4.3 Instruction-tuning with the Alpaca Dataset
Although inference cost and training efficiency are crucial, the methods of PEFT and quantization have not been extensively explored. While LoRA and PEQA have been evaluated on adaptation performance for task-specific purposes as discussed in Section 4.2, it has not been widely corroborated that fine-tuning via PEFT can retain the performance for unseen tasks. Thus, given that RTN quantization results in a non-negligible performance degradation in LLMs, it is important to assess how much the performance of low-bit quantized LLMs is degraded on comprehensive tasks. To address these concerns, we conduct comprehensive experiments, benchmarking our techniques on prevalent instruction-following datasets and assessing the response quality of PEQA-tuned LLMs. Furthermore, to determine if PEQA can regain the performance of full-precision LLMs, we employ RTN quantization in conjunction with PEQA instruction-tuning across LLaMAs.
Experimental Settings. We train LLaMAs [6, 7] in various sizes on the Alpaca dataset [59], which is one of the popular instruction-following datasets generated from outputs of InstructGPT [11]. Then, we test the models on other downstream tasks such as common-sense reasoning tasks [52-55] and massive multitask language understanding (MMLU) [56]. Due to limited time and resources, we could not conduct an exhaustive search over hyper-parameters such as the learning rate or epoch. Instead, we followed the training recipe from Taori et al. [59]. The LoRA configuration is set to QKVO16. Detailed settings can be found in Appendix H.
Common-Sense Reasoning. We conducted an experiment on five tasks [52-55] to assess whether the performance of common-sense reasoning and in-context learning can be sustained even after instruction-tuning LLMs on the Alpaca dataset via LoRA or PEQA. As depicted in Table 6, the results show that LLMs fine-tuned with LoRA or PEQA maintain a consistent trend in common-sense reasoning tasks. Furthermore, since PEQA's performance aligns closely with that of full-precision
Table 7: Massive Multitask Language Understanding (MMLU) benchmark performance of PEQATuned LLaMAs using Alpaca datasets. Five-shot accuracy is reported for the MMLU. Quantization precision of PEQA is set to 4-bit. When we quantize LLaMA [6] into 4-bit precision using the RTN method, no group size is applied. For LLaMA2 [7], a group size of 256 is used with the RTN method. Note that RTN stands for round-to-nearest in the table.
# Params	Model Size	Humanities	STEM	Social Sciences	Other	Average
LLaMA [6]	7B	13.5GB	32.6	29.6	38.0	37.9	34.4
13B	26.1GB	42.8	36.1	53.3	53.2	46.1
30B	65.1GB	54.6	46.5	66.1	63.4	57.4
+ RTN (w/o group size)	7B	3.8GB	28.4	25.6	26.9	31.8	28.3
13B	7.0GB	30.5	27.2	35.5	38.8	32.8
30B	16.9GB	39.6	34.0	46.1	49.7	42.1
+ PEQA	7B	3.8GB	35.7	30.9	38.2	40.0	35.8
13B	7.0GB	42.8	37.7	53.6	49.0	45.0
30B	16.9GB	51.1	44.1	62.4	60.7	54.3
LLaMA2 [7]	7B	13.5GB	43.3	37.0	51.8	52.4	45.9
13B	26.0GB	54.4	44.2	63.4	60.8	55.7
70B	138.0GB	65.2	57.9	80.3	74.7	69.1
+ RTN (g256)	7B	3.8GB	39.5	35.5	49.3	49.9	43.2
13B	7.0GB	50.2	42.6	61.3	59.7	53.2
70B	35.3GB	63.7	55.9	78.4	71.6	67.0
+ PEQA	7B	3.8GB	52.0	38.4	54.1	52.0	48.1
13B	7.0GB	60.5	45.0	63.3	57.0	55.3
70B	35.3GB	73.9	55.3	77.8	68.2	67.5
adaptation, this consistency is observed even when the model size has been reduced through low-bit weight quantization. We utilized the evaluation code from Eleuther AI's lm-evaluation-harness [63].
Massive Multitask Language Understanding. To assess whether the performance of PEQATuned models can be restored to the levels of full-precision model's performance, starting from RTN performance, we test our models on the MMLU benchmark containing 57 multiple-choice problems across various domains and levels of knowledge [56]. In this experiment, we utilize the RTN results as a baseline to determine the extent of degradation on quantized LLM. As shown in Table 7, instruction-tuning with PEQA boosts the performance of RTN quantized models. This observation supports our claim that our approach enables LLMs to regain their few-shot in-context learning and understanding capabilities, even though they are significantly smaller than their original model size through quantization. Unfortunately, it seems that the PEQA-tuning does not achieve the best performance in fine-tuning larger models. This might be because PEQA-tuning did not been sufficiently explored different epochs or learning rates. Nonetheless, the observation that the performance of the quantized LLaMAs is restored through PEQA-tuning using an instruction-following dataset highlights the potential to further enhance the accuracy of PTQ methods.
Fine-tuning aligns large language models (LLMs) with specific purposes. To maintain the comprehensive capabilities of LLMs while effectively aligning them, we introduce PEQA, a method that seamlessly combines the advantages of parameter-efficient fine-tuning (PEFT) and quantization in LLMs. PEQA not only reduces DRAM consumption during fine-tuning but also accelerates inference latency for deployment by retaining weights in a low-bit quantized format. Through rigorous testing across various datasets and LLMs, we have found that PEQA can match the performance of full-precision baselines in task-specific adaptations, even with a significant reduction in model size. When combined with instruction-tuning, PEQA's performance demonstrates its ability to both preserve and enhance comprehensive knowledge after the inherent compromises of quantization, recovering the performance of original model by simply updating the quantization scales of the quantized LLM.
[66] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6.
A Common Experimental Settings
For the common experimental settings, AdamW [64] optimizer and linear-decaying learning rate scheduler were used. We use Deepspeed repository [65]  $^2$  for FP16 and BF16 training. Additionally, we utilize Huggingface repository[66]  $^3$  for training, evaluation code and dataset.
B Experimental Settings of Section 4.1
We compare the perplexity when weights are quantized and adapted by quantization-aware training (QAT), LoRA with post-training quantization (PTQ), and PEQA, using the Wikitext2 dataset in Section 4.1. The LoRA configuration is set to QV4. For PTQ method, we utilize OPTQ [28] which is state-of-the-art low-bit weight-only PTQ method. We set the model's maximum sequence length to 1024. Batch size and epoch for all experiments are set to 128 and 15 respectively. The learning rates for the experiments of Table 2 are displayed in Table 8. Learning rates for LoRA and PEQA are shown in Appendix C.
Table 8: Learning rates of QAT in Table 2.
Method	W Bits	GPT-Neo 2.7B	GPT-J 6B	LLaMA 7B	LLaMA 13B
QAT	4	4e-5	5e-6	1e-5	3e-5
QAT	3	6e-5	1e-5	2e-5	1e-5
C Experimental settings of Table 3
In Section 4.2, Table 3 show the scalability and task-specific adaptation performance of PEQA by comparing with LoRA and LoRA+OPTQ on Wikitext2 [51] and PennTreeBank (PTB) [58] datasets. Detailed experimental settings are as follows. LoRA configuration is set to QV4. For PTQ method, we utilize OPTQ [28] which is state-of-the-art low-bit weight-only PTQ method. We set input sequence length after tokenization (block size) to 1024 for under 65B models. For LLaMA 65B, input sequence length after tokenization is set to 768 due to memory issue. Batch size and epoch for all experiments are set to 128 and 15 respectively. Learning rates for Table 3 experiments are shown in Table 9.
Table 9: Learning rate of LoRA and PEQA in Table 3 on Wikitext2 and PTB datasets.
Method	W Bits	GPT-Neo 2.7B	GPT-J 6B	LLaMA 7B	LLaMA 13B	LLaMA 30B	LLaMA 65B
Wikitext2
LoRA	16	5e-4	6e-4	1e-4	1e-4	2e-4	4e-5
PEQA (Ours)	4	5e-5	6e-6	6e-6	1e-5	1e-5	1e-5
PEQA (Ours)	3	6e-5	5e-5	2e-5	6e-5	3e-5	3e-5
PTB
LoRA	16	2e-3	1e-3	8e-4	5e-4	4e-4	6e-4
PEQA (Ours)	4	3e-4	5e-5	5e-5	5e-5	3e-5	6e-5
$^{2}$ https://github.com/microsoft/DeepSpeed
3https://github.com/huggingface/transformers/tree/main/examples/pytorch/
language-modeling
4https://github.com/IST-DASLab/gptq
D The Perplexity of 3-bit and 4-bit PEQA on Wikitext2 Dataset
Figure 3 illustrates the results of 3-bit and 4-bit PEQA's next token prediction performance on the Wikitext2 dataset. As shown in Figure 3, 3-bit performance of PEQA shows lower perplexity than 3-bit post-training quantized LoRA. The results from the 3-bit PEQA show that PEQA allows for continuity in model size options under DRAM usage constraints.
Figure 3: The perplexity over model size of 3/4-bit performance of PEQA and LoRA+OPTQ
E OPT Models Adapted with PEQA and LoRA on Wikitext2 Dataset
Table 10 shows the perplexity of OPT[4] models adapted with PEQA and LoRA on the Wikitext2 dataset. The perplexity gap between LoRA and PEQA becomes smaller as the model size increases.
Table 10: The perplexity (PPL) on Wikitext2 for OPT 1.3B to 66B. In this comparison, only the weights were quantized into 4-bit. A lower PPL value indicates better performance.
Method	W Bits	OPT 1.3B	OPT 2.7B	OPT 6.7B	OPT 13B	OPT 30B	OPT 66B
LoRA(QV4)	16	11.58	10.25	8.96	8.44	7.93	7.64
PEQA(Ours)	4	12.40	10.78	9.34	8.74	8.11	7.86
F LoRA Configuration Comparison on Wikitext2 Dataset
As shown in Table 11, the LoRA target module configuration of QV4 and QKVO16 has not much effect on perplexity on Wikitext2 experimental results. Table 11 shows equal tendency as mentioned in [21]. We utilize QV4 configuration for Section 4.2 and QKVO16 configuration for Section 4.3 respectively.
Table 11: The perplexity (PPL) on Wikitext2 was compared with LoRA QV4 and QKVO16. A lower PPL value indicates better performance.
Method	# Bits	GPT-Neo 2.7B	GPT-J 6B	LLaMA 7B	LLaMA 13B	LLaMA 30B	LLaMA 65B
LoRA(QV4)	16	10.63	8.50	5.53	5.06	4.06	3.82
LoRA(QKVO16)	16	10.67	8.50	5.50	5.06	4.06	3.81
G Experimental Settings of Multi-scale Performance
In Section 4.2, Table 5 shows the perplexity of PEQA with grouping learnable parameters. We set model maximum sequence length to 1024. Batch size and epoch for all experiments are set to 128 and 15 respectively. Learning rates for experiments are shown in Table 12.
Table 12: Learning rate for Table 5.
Model	W Bits	g-1	g256	g128	g64
LLaMA 13B	4	1e-5	4e-5	4e-5	3e-5
3	6e-5	9e-5	9e-5	5e-5
LLaMA 7B	4	6e-6	2e-5	2e-5	1e-5
3	2e-5	6e-5	4e-5	7e-5
H Experimental Settings of Section 4.3
In Section 4.3, we use the Alpaca dataset [59] for instruction-tuning. We set learning rate, epoch, and quantization group size as in Table 13. The batch size is set to 128 for all experiments in this subsection. As mentioned in Section 4.3, due to limited time and resources, we couldn't conduct an exhaustive search over hyper-parameters such as learning rate or epoch. We believe that there are hyper-parameters that can perform better. For LLaMA 1 series (LLaMA 7, 13, and 30B), we truncate the prompt to the length of 2024 since their maximum sequence length is 2024 when evaluating the massive multitask lanugage understanding (MMLU) benchmark. Thus, for LLaMA2-70B, we set the tokenizer max length to 1024 on fine-tuning due to the resource limit. Otherwise, we use default max length of tokenizer<sup>5</sup> on training. For the evaluation, we use default tokenizer setting. For every experiment in this section, the configuration of PEQA is set to 4-bit RTN quantization.
Table 13: The learning rate, epoch, quantization group size [49] for experiments on Section 4.3. the weights were quantized into 4-bit.
Hyper-parameter	LLaMA 7B	LLaMA 13B	LLaMA 30B	LLaMA2 7B	LLaMA2 13B	LLaMA2 70B
Epoch	3	3	5	3	3	5
Learning rate	2e-5	2e-5	5e-6	5e-6	5e-6	5e-6
Group size	Per-channel	Per-channel	Per-channel	256	256	256
5e.g. 'hf-internal-testing/llama-tokenizer' for LLaMA-1 series, 'meta-llama/Llama-2-70b-hf' for LLaMA-2 series.
I LLaMA 7B and 13B on Natural Instruction
To evaluate the instruction-following ability of instruct-tuned models, we test them on another instruction-following dataset, Natural Instruction (NI) [57]. Different from the Alpaca dataset, instructions of NI were collected by humans for existing 61 NLP tasks. For simplicity, we utilize evaluation splits consisting of 12 subtasks and restrict the maximum number of instances for each task to 200. At test time, the model should generate proper output for the given input with instruction for the target unseen task. As shown in Table 14, we find LLaMAs trained with PEQA show consistently better zero-shot task generalization performance (ROUGE-L) in NI for all parameter sizes compared to those from LoRA.
Table 14: Natural Instruction benchmark performance of parameter-efficient instruction-tuned LLa-MAs using Alpaca datasets. Zero-shot performance (ROUGE-L) is reported for the NI. LoRA configuration is set to QKVO16. Quantization precisions of LoRA w/ OPTQ and PEQA are set to 4-bit.
# Params	LLaMA	+LoRA	+LoRA w/OPTQ	+PEQA
7B	9.4	24.4	25.0	27.1
13B	8.9	31.3	29.2	34.1
J Comparison with AlphaTuning
When diving deeper into quantization scales, learnable parameters for both PEQA and AlphaTuning, it's worth noting that PEQA's adherence to uniform quantization means there's only one shared quantization scale for integer weight. Conversely, AlphaTuning's non-uniform approach means that for a  $b$ -bit quantization, there are  $b$  individual quantization scales for each weight matrix. Despite having multiple scales, AlphaTuning only fine-tunes one, leaving the rest static. As such, the number of trainable parameters are identical and AlphaTuning seems to offer a larger potential for a well-fitted model, but it can be easily seen that  $b - 1$  rest static scales introduced in AlphaTuning have limited usability, and thus the method may be prone to overfitting as evident through empirical results.
In Table 15, we conducted training on GPT-Neo and OPT 1.3B using the Wikitext2 dataset. Interestingly, PEQA, drawing from its methodological advantages, consistently demonstrates superior performance to AlphaTuning by at least 0.7 ppl on the Wikitext2 dataset. Both AlphaTuning and PEQA used channel-wise trainable parameters. Batch size of AlphaTuning is set to 32.
Table 15: The perplexity (PPL) of AlphaTuning and PEQA on Wikitext2 with OPT and GPT-Neo 1.3B. The lower PPL, the better.
Method	# Bits	OPT 1.3B	GPT-Neo 1.3B
AlphaTuning	4	13.15	15.03
PEQA (Ours)	4	12.40	14.22
AlphaTuning	3	14.00	17.25
PEQA (Ours)	3	13.40	15.16
Table 16: Learning rate of AlphaTuning in Table 15.
Method	W Bits	OPT 1.3B	GPT-Neo 1.3B
AlphaTuning	4	1e-4	5e-4
AlphaTuning	3	1e-4	1e-3
K Choice of Updating Quantization Scales or Zero-Points
Uniform quantization can represent both asymmetric and symmetric quantizations, hence it's not always necessary to mandate the use of zero-points. This is why adopting a strategy of only learning the scale factor serves as a fundamental and scalable baseline. We opted for this approach to clearly establish its advantages. To determine the efficacy of learning only the scaling factors, we have incorporated additional experiments. By referring to the table below, it's evident that merely optimizing zero-points does not yield effective learning outcomes. Moreover, simultaneously optimizing both zero-points and quantization scales does not present any significant improvement in accuracy either.
Table 17: Perplexity (PPL) of PEQA on the Wikitext2 dataset for LLaMA 7B and LLaMA 13B with weights quantized into 4-bit.
Method	Zero-points only	Quantization scales only (PEQA)	Both zero-points and quantization scales
LLaMA 7B	11.56	5.84	5.86
LLaMA 13B	9.83	5.30	5.34
L Memory Peak on Training