SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $U$  is set as  $U = (H \odot M) \mathrm{diag}(H)^{-1}$  as above. This shows how this single-pass version of greedy updates fits into our adaptive rounding with linear feedback framework.
Analyzing greedy local search as a post-processing pass is a bit more difficult, but we will see that it can also be written as something like adaptive rounding with linear feedback. Suppose that we do a pass of greedy updates, but our quantized weights start at an initial value  $\hat{w} = \tilde{w}$  already quantized from some previous method (e.g. LDLQ). Returning to (8), since we haven't updated  $\hat{w}_j$  yet, we'll have
$$
z = \tilde {w} e _ {j} - \frac {(\hat {w} - w) H e _ {j}}{e _ {j} ^ {T} H e _ {j}}.
$$
Now, all the entries of  $\tilde{w}$  which come after  $j$  are still the ones from  $\tilde{w}$ . This means that we can split this up as
$$
z = w e _ {j} - \frac {(\hat {w} - w) _ {: , 1 : (j - 1)} H _ {1 : (j - 1) , j} + (\tilde {w} - w) _ {: , (j + 1) : n} H _ {(j + 1) : n , j}}{e _ {j} ^ {T} H e _ {j}}
$$
where the first part of this sum comes from the entries which we may have already updated during this pass, the second comes from the entries which are still equal to their initial values in  $\tilde{w}$ , and the case of  $w_{j}$  is handled specially, cancelling it with the  $\tilde{w}e_{j}$  term. We can write this more compactly in matrix form as
$$
z = w e _ {j} - \frac {(\hat {w} - w) (H \odot M) e _ {j} + (\tilde {w} - w) (H \odot M ^ {T}) e _ {j}}{e _ {j} ^ {T} H e _ {j}},
$$
Model	Processing	Absolute Fractional Rank	Approximate Fractional Rank	tr (D) / tr (H)
OPT-125m	Baseline	0.926 (±0.172)	0.112 (±0.127)	0.540 (±0.093)
Incoherent	0.910 (±0.196)	0.124 (±0.141)	0.534 (±0.094)
OPT-350m	Baseline	0.916 (±0.180)	0.047 (±0.032)	0.445 (±0.100)
Incoherent	0.908 (±0.183)	0.059 (±0.062)	0.440 (±0.106)
OPT-1.3b	Baseline	0.541 (±0.404)	0.020 (±0.023)	0.399 (±0.187)
Incoherent	0.543 (±0.405)	0.028 (±0.023)	0.393 (±0.189)
OPT-2.7b	Baseline	0.426 (±0.413)	0.019 (±0.015)	0.384 (±0.206)
Incoherent	0.427 (±0.415)	0.018 (±0.025)	0.375 (±0.205)
Table 6: We compute  $H$  in each layer of a given model, and compute the following summary statistics.  $\operatorname{tr}(D) / \operatorname{tr}(H)$  decreases as the mode size increases, though the variance also increases. We compute the fraction of nonzero eigenvalues (i.e. absolute), and the fraction of eigenvalues  $> 0.01 \cdot \max(\mathrm{eig}(H))$  (i.e. approximate). The fractional rank is  $k/n$  for a rank- $k$  matrix  $H$  with dimension  $n$ . Mean and standard deviations are computed across layers in a model.
where  $M$  is the strictly upper triangular mask and  $\odot$  is elementwise multiplication. This yields a final quantization step of
$$
\hat {w} e _ {j} \leftarrow \mathcal {Q} \left(w e _ {j} - (\tilde {w} - w) \frac {\left(H \odot M ^ {T}\right) e _ {j}}{e _ {j} ^ {T} H e _ {j}} - (\hat {w} - w) \frac {H e _ {j}}{e _ {j} ^ {T} H e _ {j}}\right).
$$
So, more generally, if we define  $U$  as above, and set
$$
V = W - (\tilde {W} - W) (H \odot M ^ {T}) \operatorname {d i a g} (H) ^ {- 1},
$$
we can write a single pass of greedy updates in matrix form as
$$
\tilde {W} \leftarrow \mathcal {Q} (V + (W - \hat {W}) U),
$$
which is very close to our rounding with linear feedback form, albeit with the difference that here  $V$  is in place of  $W$ . This is made explicit in the included Greedy Updates Algorithm.
We can use this algorithm both as a whole quantization method (by setting  $\tilde{W} = W$ ) or as a post-processing step (by setting  $\tilde{W}$  to the output of some other initial quantization algorithm, such as LDLQ). When we do use it as a post-processing step, we typically run multiple passes of greedy updates (e.g. 10 passes): this involves passing the output of the greedy updates algorithm back in as the input guess  $\tilde{W}$  to another run of the greedy updates algorithm, and repeating this multiple times.
C Additional Experimental Descriptions and Results
C.1 Subsections 3.2 and 3.3 (Empirical Properties of  $H$  Across OPT-125m to 2.7b)
Interpreting the exact proxy loss of LDLQ and nearest rounding by empirically comparing  $\operatorname{tr}(D)$  vs  $\operatorname{tr}(H)$ . Theorem 1 gives the average-case proxy loss for LDLQ in terms of  $\operatorname{tr}(D)$ , where  $D$  is from the LDL decomposition of  $H$ . Lemma 3 gives the average-case proxy loss for standard nearest rounding in terms of  $\operatorname{tr}(H)$ . We know that LDLQ is better in practice, but comparing these equations is difficult because we need to reason about  $\operatorname{tr}(D)$  vs  $\operatorname{tr}(H)$ . Our paper resolves this difficulty by deriving bounds on the proxy loss for LDLQ in terms of the spectrum of  $H$  (with and without incoherence). However, we also perform a quick empirical check: if  $\operatorname{tr}(D) \ll \operatorname{tr}(H)$ , then our theory explains the empirical superiority of LDLQ over nearest rounding (at least on these models). Table 6 gives the ratio  $\operatorname{tr}(D) / \operatorname{tr}(H)$  across all layers for OPTQ models 125m to 2.7b; the mean value is always less than 0.55, and it falls as the model gets larger.
$H$  is approximately low-rank. Subsection 3.3 plotted the normalized eigenvalues of  $H$  from 3 randomly chosen layers in OPT-2.7b. Table 6 gives much more evidence that  $H$  is consistently approximately low-rank. Across each model, we calculate the absolute and approximate fractional rank
of  $H$  across all layers in OPT models 125m to 2.7b (explanations in the caption). The approximate fractional rank decreases as model size increases; for OPT-2.7b the fractional rank is  $\approx 0.02(\pm 0.02)$ .
C.2 Subsection 5.1 (Empirical Verification of OPTQ Equivalence)
We share a python script in the supplementary code which empirically verifies that our implementation of LDLQ produces quantized values exactly matching OPTQ's [1] implementation. While we prove the equivalence between LDLQ and OPTQ's respective algorithm statements, empirically comparing ours and Frantar et al. [1]'s code ensures that the respective implementations are sufficiently close to their algorithmic statements. Therefore we can be sure that LDLQ and OPTQ are equivalent in their implementation.
C.3 Subsection 5.2 (Empirical Verification of LDLQ/OPTQ Finite Grid Counterexample)
The following code constructs a weight matrix  $W$  and Hessian matrix  $H$  where OPTQ performs worse than nearest when rounding to a finite grid.
1 import torch
2 def make counterexample(n, d, c=0.01):
3 H = torch.ones(n, n) + torch.eye(n)
4 H[n-1, n-1] = 1.0
5 H[0, 1:(n-1)] += 2 * c
6 H[1:(n-1), 0] += 2 * c
7 H[0, n-1] += c
8 H[n-1, 0] += c
9 H[0, 0] += 4 * c + n * (c**2)
10 W = 0.499 * torch.ones(d, n) + 0.002 * (torch.arange(n) % 2)
11 return W, H
The intuition behind this counterexample is as follows: we want to quantize many coordinates in  $W$  in such a way that OPTQ excepts there to be a very large error correction to quantize the last entry. However, the finite grid restricts this large error correction. Note that we can achieve this poor OPTQ behavior with  $c = 0$ , but here nearest rounding also does poorly. We make a small perturbation  $(c = 0.01)$  to make OPTQ round in the wrong direction, but not nearest.
C.4 Additional Details on the Experimental Setup and Computational Resources
We run experiments on a university cluster managed by a Slurm workload manager which has GPUs with up to 48GB of memory, though larger GPUs are only required for some methods on larger model sizes. Note we use the LAMBADA OpenAI version. When Greedy updates are used, we perform 10 passes over the weights in the same order as LDLQ and OPTQ, except for 5 passes on OPT-30b and OPT-66b. For the incoherence-based quantization range, we tune the parameter  $\rho$  and find that a value of 2.4 works well across all model sizes and quantization methods. We use this value for all our experiments.
C.5 Section 6 (Main Results on Additional Evaluations)
Figure 6 shows additional results for QuIP and OPTQ on WikiText2, PiQA, and StoryCloze when quantizing to 2 and 3 bits per weight. The insights about our method QuIP remain the same after viewing these additional results: QuIP is the first PTQ procedure to achieve good quantization at two bits per weight, across a variety of LLM sizes and evaluation tasks. We evaluate on OPT models (up to 30B); 4-bit quantization works equally well for both methods. QuIP is superior to OPTQ across model sizes and evaluation tasks here.
On WikiText2 2-bit quantization, note that the trend in perplexity for QuIP mirrors the trend in perplexity for OPTQ. We run OPTQ's [1] implementation, though they did not report 2-bit results at this model size. Because OPTQ is equivalent to QuIP's quantization sub-procedure, it thus makes sense that worse performance in the quantization sub-procedure could result in worse overall performance. OPTQ increases perplexity when going from OPT-1.3b to OPT-2.7b. QuIP's perplexity also increases from OPT-1.3b to OPT-2.7b, and is unusually higher than the adjacent OPT-1.3b and
Figure 6: Quantizing OPT models up to 66B parameters. Additional evaluation tasks shown here in the Supplement. Our method QuIP is the first PTQ procedure to achieve good quantization at 2 bits per weight, across a variety of model sizes and evaluation tasks. Note the drop in performance for OPTQ on OPT-66B is documented in their paper.
OPT-6.7b models. However QuIP still beats OPTQ in this setting. Our observations about OPTQ and QuIP on WikiText2 and OPT-2.7b were consistent across multiple independent runs.
C.6 Section 6 (All Methods, All Model Sizes, All Bit Weights, All Evaluation Tasks)
Tables 7-13 provide results on all combinations of the following: methods, model sizes (OPT 125m-30b), bit weights(4,3,2), and evaluation tasks. Across our extensive array of experiments, we see that incoherence processing always enables a step function change in quantization at 2 bits.
Full	QuIP	QuIP-RG	Greedy+IncP	Near+IncP
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	9.56	9.60	9.79	11.48	9.66	9.75	11.68	9.72	9.92	11.59	9.77	9.89	12.04
PTB↓	14.04	14.18	14.37	17.40	14.11	14.44	16.94	14.23	14.45	17.39	14.16	14.49	18.12
C4↓	11.45	11.50	11.66	13.55	11.51	11.68	13.44	11.52	11.71	13.30	11.53	11.74	14.11
ArcE↑	65.40	65.32	65.28	57.87	64.86	63.51	59.51	65.99	63.80	58.80	64.06	64.06	56.36
LAMB↑	72.40	73.20	72.68	65.24	71.86	71.53	62.31	71.71	71.38	64.47	71.41	71.41	60.64
PiQA↑	78.13	78.45	78.73	75.24	78.51	78.73	76.17	77.86	77.58	75.95	78.24	77.53	75.46
SC↑	77.28	76.96	76.51	73.39	77.02	77.08	73.01	76.70	76.64	73.33	76.77	75.94	71.93
Baseline Processing — OPT-30b
Full	OPTQ	LDLQ-RG	Greedy	Near
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	9.56	9.59	10.32	71.70	9.64	10.31	49.40	9.69	13.63	4,817	10.77	1,565	41,548
PTB↓	14.04	14.22	15.36	88.19	14.20	15.15	73.45	14.33	23.05	3,474	15.41	1,526	34,349
C4↓	11.45	11.56	12.23	29.59	11.56	12.15	29.12	11.59	16.30	3,183	13.52	1,808	24,816
ArcE↑	65.40	64.77	60.19	42.47	63.76	63.43	41.20	63.09	50.51	26.30	61.28	34.47	25.80
LAMB↑	72.40	72.39	68.89	25.77	71.94	69.78	26.35	72.37	56.76	00.00	70.42	01.73	00.00
PiQA↑	78.13	78.56	78.02	66.05	78.56	77.80	64.58	78.35	70.46	49.89	77.02	56.37	49.56
SC↑	77.28	77.53	75.62	63.59	76.89	75.56	63.53	76.45	68.43	48.31	75.24	49.59	48.57
Table 7: Quantizing OPT-30b with all combinations of quantization and pre-post processing methods, evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.
Incoherence Processing — OPT-13b
Full	QuIP	QuIP-RG	Greedy+IncP	Near+IncP
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	10.13	10.21	10.5	16.02	10.35	10.69	13.81	10.25	10.61	13.91	10.34	10.59	16.12
PTB↓	14.52	14.69	15.05	21.64	14.73	15.20	22.23	14.85	15.11	20.20	14.93	15.27	23.18
C4↓	12.06	12.16	12.39	16.60	12.18	12.43	15.62	12.21	12.42	15.19	12.26	12.56	17.37
ArcE↑	61.78	61.41	59.47	53.91	60.35	61.78	52.86	60.10	59.43	53.79	60.56	59.30	50.00
LAMB↑	70.25	72.09	71.10	56.24	69.47	69.07	55.70	70.83	68.43	56.98	68.37	67.86	46.48
PiQA↑	76.82	76.61	76.17	72.52	76.55	76.22	72.74	76.33	76.17	71.87	75.08	76.66	70.73
SC↑	76.58	75.62	74.92	70.21	75.88	75.75	70.53	75.43	75.62	72.50	74.47	75.43	68.43
Baseline Processing — OPT-13b
Full	OPTQ	LDLQ-RG	Greedy	Near
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	10.13	10.31	11.60	372.68	10.28	11.54	213.75	10.73	13.67	8,370	11.33	3,333	186,069
PTB↓	14.52	14.91	16.59	344.44	14.85	16.43	220.38	15.25	18.62	7,053	16.40	2,708	121,291
C4↓	12.06	12.26	13.34	135.48	12.24	13.17	67.48	12.55	14.30	4,316	13.32	2,711	93,834
ArcE↑	61.78	64.77	60.19	42.47	60.77	58.54	32.07	56.61	51.22	25.38	61.32	31.10	25.42
LAMB↑	70.25	72.39	68.89	25.77	68.72	65.30	6.58	68.12	59.36	00.02	67.22	00.06	00.00
PiQA↑	76.82	78.56	78.02	66.05	76.28	75.08	59.09	76.50	73.45	50.98	76.06	53.10	49.62
SC↑	76.58	77.53	75.62	63.59	76.32	73.52	56.33	75.68	72.44	49.40	74.41	49.71	48.70
Table 8: Quantizing OPT-13b with all combinations of quantization and pre-post processing methods, evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.
Incoherence Processing — OPT-6.7b
Full	QuIP	QuIP-RG	Greedy+IncP	Near+IncP
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	10.86	10.98	11.51	22.33	11.20	11.61	23.75	11.13	11.62	19.06	11.18	11.73	18.57
PTB↓	15.77	15.93	16.52	31.73	15.99	16.43	45.53	15.88	16.50	35.94	16.06	16.47	27.04
C4↓	12.71	12.86	13.30	21.62	12.88	13.39	24.98	12.89	13.27	19.62	12.96	13.37	19.15
ArcE↑	60.06	59.89	59.60	52.61	59.30	58.21	53.32	59.18	58.25	51.43	59.85	57.62	50.59
LAMB↑	68.72	70.00	68.74	53.97	67.38	65.77	49.91	67.65	67.18	54.80	67.26	65.86	49.49
PiQA↑	76.55	76.77	76.33	72.47	76.71	76.33	72.91	76.39	75.46	72.20	76.55	76.71	71.22
SC↑	74.47	75.18	73.65	68.43	75.05	73.33	69.51	74.35	73.77	68.94	74.22	74.09	68.75
Baseline Processing — OPT-6.7b
Full	OPTQ	LDLQ-RG	Greedy	Near
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	10.86	11.49	14.87	2,958	11.23	12.56	739.9	11.75	39.09	16,298	12.15	6,011	20,780
PTB↓	15.77	16.54	22.05	2,521	16.28	18.58	1,109	16.93	66.57	10,708	18.92	5,440	14,217
C4↓	12.71	13.16	17.13	500.7	12.98	14.34	154.0	13.27	37.13	9,968	14.40	5,225	12,419
ArcE↑	60.06	58.84	53.41	31.86	59.18	55.26	33.00	54.63	32.49	26.09	58.75	25.42	25.80
LAMB↑	68.72	66.18	52.36	01.07	67.46	61.89	01.79	66.19	02.56	00.00	64.53	00.00	00.00
PiQA↑	76.55	76.01	73.23	55.11	76.77	74.48	54.46	74.48	53.59	51.90	76.28	50.71	49.78
SC↑	74.47	73.71	71.42	52.07	74.09	72.37	52.45	72.82	50.99	49.40	73.58	47.87	47.80
Table 9: Quantizing OPT-6.7b with all combinations of quantization and pre-post processing methods, evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.
Incoherence Processing — OPT-2.7b
Full	QuIP	QuIP-RG	Greedy+IncP	Near+IncP
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	12.47	12.39	17.44	2,998	12.58	15.07	1,676	12.68	12.96	155.6	12.79	13.79	28.98
PTB↓	17.97	18.42	20.79	63.59	18.43	20.49	42.05	18.34	20.03	46.28	18.43	19.51	39.23
C4↓	14.34	14.55	15.63	38.07	14.65	15.97	27.89	14.64	15.22	26.84	14.67	15.52	27.34
ArcE↑	54.34	53.28	52.99	46.93	52.02	52.36	46.93	52.90	51.73	43.14	52.61	50.93	44.11
LAMB↑	64.82	66.04	64.99	36.06	64.64	63.46	43.39	64.68	62.95	45.53	65.40	61.05	35.65
PiQA↑	74.76	74.54	73.94	68.06	73.88	73.45	68.28	74.54	73.83	68.28	73.61	73.56	67.85
SC↑	71.74	71.80	70.21	66.14	71.55	70.15	64.67	70.85	71.10	65.82	71.16	70.02	63.27
Baseline Processing — OPT-2.7b
Full	OPTQ	LDLQ-RG	Greedy	Near
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	12.47	12.93	17.09	8,949	12.77	16.47	7,718	12.95	18.92	9,665	16.69	15,685	10,641
PTB↓	17.97	19.10	25.36	8,281	19.05	23.94	7,389	19.06	28.75	8,254	32.22	14,532	10,516
C4↓	14.34	14.99	18.14	4,388	14.85	17.37	2,113	15.01	20.87	5,139	18.75	11,257	9,356
ArcE↑	54.34	52.57	50.04	26.94	52.02	48.95	25.76	52.02	43.39	25.46	52.74	26.56	27.19
LAMB↑	64.82	62.00	51.43	00.00	64.04	53.25	00.00	63.50	40.75	00.00	59.15	00.00	00.00
PiQA↑	74.76	73.88	70.73	48.42	74.54	69.91	49.95	73.61	66.05	50.65	73.83	51.41	50.22
SC↑	71.74	70.91	68.56	48.50	71.42	67.79	47.17	70.66	60.53	48.44	70.59	47.42	47.55
Table 10: Quantizing OPT-2.7b with all combinations of quantization and pre-post processing methods, evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.
Incoherence Processing — OPT-1.3b
Full	QuIP	QuIP-RG	Greedy+IncP	Near+IncP
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	14.62	14.88	16.21	41.64	16.49	17.76	42.37	16.75	17.11	48.69	16.43	17.83	56.56
PTB↓	20.29	20.87	22.76	47.72	21.93	23.25	50.17	22.11	23.76	54.46	22.19	24.82	80.40
C4↓	16.07	16.38	17.12	29.78	17.53	18.44	31.49	17.60	18.54	34.10	17.74	19.03	45.56
ArcE↑	50.84	50.72	49.12	41.88	49.54	48.82	41.20	49.66	48.74	41.08	48.61	46.59	38.64
LAMB↑	58.92	56.36	52.47	27.81	51.62	48.36	27.27	49.95	48.38	19.21	49.76	51.12	20.20
PiQA↑	72.31	71.22	71.11	64.85	71.06	70.24	63.33	71.00	70.35	63.66	71.16	69.80	62.51
SC↑	70.78	70.08	68.81	63.02	69.00	68.05	63.14	68.49	67.92	62.64	69.13	67.79	58.43
Baseline Processing — OPT-1.3b
Full	OPTQ	LDLQ-RG	Greedy	Near
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	14.62	15.59	21.35	7,856	15.36	20.22	7,739	15.58	22.68	9,786	47.62	12,658	11,690
PTB↓	20.29	22.03	30.74	6,858	21.85	30.10	5,368	22.00	35.18	8,441	73.51	14,705	11,690
C4↓	16.07	16.96	21.59	4,028	16.70	20.21	2,123	16.96	22.11	5,129	27.20	6,415	8,360
ArcE↑	50.84	49.33	45.58	25.46	48.95	45.41	26.68	48.19	42.42	26.01	42.80	27.82	25.13
LAMB↑	58.92	57.03	37.32	00.00	58.45	41.08	00.02	59.15	40.97	00.00	36.91	00.00	00.00
PiQA↑	72.31	70.73	68.66	49.73	70.40	67.95	52.18	70.67	66.43	50.87	67.74	51.41	49.78
SC↑	70.78	70.15	65.18	48.38	70.34	66.45	49.27	70.40	64.48	48.76	59.13	47.87	48.25
Table 11: Quantizing OPT-1.3b with all combinations of quantization and pre-post processing methods, evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.
Incoherence Processing — OPT-350m
Full	QuIP	QuIP-RG	Greedy+IncP	Near+IncP
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	22.00	22.5	25.19	672.3	23.57	25.54	418.0	23.14	25.38	239.9	23.41	27.86	1,444
PTB↓	31.07	32.57	35.65	744.2	32.46	37.00	587.4	33.10	37.07	301.0	33.32	39.49	1,354
C4↓	22.59	23.23	25.48	320.0	23.45	25.50	215.4	23.43	25.48	124.1	23.81	27.41	880.2
ArcE↑	40.36	39.44	38.13	27.44	39.31	38.47	29.67	39.77	40.24	30.64	38.89	38.76	28.41
LAMB↑	46.67	46.89	42.03	01.03	43.04	39.80	04.99	42.44	40.62	06.38	41.47	34.45	00.08
PiQA↑	64.80	64.47	63.28	50.87	64.25	63.17	54.79	64.42	64.25	55.01	64.15	63.00	52.23
SC↑	63.14	62.13	61.55	53.15	61.74	61.23	51.43	62.83	61.62	53.28	62.38	61.49	50.22
Baseline Processing — OPT-350m
Full	OPTQ	LDLQ-RG	Greedy	Near
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	22.00	24.16	33.51	18,687	23.77	31.87	10,446	27.01	137.3	23,952	25.94	64.56	23,668
PTB↓	31.07	34.17	47.69	18,161	33.35	44.38	8,508	40.39	153.5	15,176	36.78	87.22	28,881
C4↓	22.59	24.71	31.26	8,418	24.10	29.86	3,064	27.84	73.59	9,099	26.21	55.15	17,094
ArcE↑	40.36	38.43	38.38	26.30	39.06	37.42	25.46	38.34	31.06	24.33	38.68	36.11	25.88
LAMB↑	46.67	45.60	39.20	00.00	45.26	32.54	00.02	51.45	16.63	00.00	40.66	27.46	00.00
PiQA↑	64.80	64.04	63.44	51.25	65.13	61.97	49.67	63.49	55.44	50.60	63.38	60.55	51.58
SC↑	63.14	63.78	61.04	47.55	62.57	60.53	48.95	61.36	54.87	48.44	63.02	56.84	48.95
Table 12: Quantizing OPT-350m with all combinations of quantization and pre-post processing methods, evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.
Incoherence Processing — OPT-125m
Full	QuIP	QuIP-RG	Greedy+IncP	Near+IncP
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	27.66	33.35	34.22	347.4	31.51	42.94	361.8	30.65	55.54	230.8	31.93	37.57	397.5
PTB↓	38.99	40.80	47.34	430.3	43.28	51.69	414.1	41.96	48.79	250.6	43.08	52.20	441.9
C4↓	26.56	27.63	30.92	177.4	28.74	33.54	159.0	28.82	31.41	99.01	29.28	33.88	224.0
ArcE↑	40.03	38.89	37.92	31.99	39.27	38.26	31.36	38.80	37.67	33.21	38.55	37.42	32.91
LAMB↑	39.16	33.03	26.37	01.05	33.75	16.96	02.17	37.78	25.34	04.66	35.65	25.21	01.82
PiQA↑	61.92	61.64	61.64	54.24	61.64	61.92	55.44	61.10	60.83	56.47	61.43	61.10	53.48
SC↑	59.96	60.03	59.20	52.13	59.07	59.26	51.94	60.15	59.52	54.04	59.13	58.88	53.41
Baseline Processing — OPT-125m
Full	OPTQ	LDLQ-RG	Greedy	Near
W16	W4	W3	W2	W4	W3	W2	W4	W3	W2	W4	W3	W2
Wiki↓	27.66	31.44	53.26	4,563	32.29	53.25	3,704	77.80	1,791	3,707	37.14	1,293	5,375
PTB↓	38.99	45.31	74.79	4,410	45.56	75.85	3,596	101.1	1,403	4,622	53.93	1,418	4,267
C4↓	26.56	29.13	42.55	2,260	29.40	41.77	1,820	65.54	809.5	1,897	33.90	836.5	3,665
ArcE↑	40.03	38.51	35.73	28.62	39.02	36.36	27.19	34.05	26.43	27.15	36.66	30.39	26.01
LAMB↑	39.16	33.69	12.36	00.00	33.26	15.00	00.00	12.25	00.00	00.00	18.22	00.08	00.00
PiQA↑	61.92	60.83	59.47	52.23	61.70	59.58	50.05	57.62	49.29	50.49	61.43	55.88	51.20
SC↑	59.96	58.88	56.97	49.78	59.20	57.03	48.95	50.99	47.55	48.82	59.96	50.03	47.93
Table 13: Quantizing OPT-125m with all combinations of quantization and pre-post processing methods, evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.
C.7 Section 6 (Evaluating the Effectiveness of the Proxy Objective)
In Table 14 we show the proxy loss of the four quantization methods we evaluate, evaluated over OPT models  $125\mathrm{m}$  to  $2.7\mathrm{b}$ . The proxy is averaged over models proxy losses normalized by their model dimension; we use  $H$  matrices computed as a result of OPTQ and nearest rounding. We do not conduct any processing in the proxy evaluation; this is an evaluation of the rounding methods only. Trends in the proxy reflect end-to-end results. OPTQ/LDLQ, LDLQ-RG, and Greedy are roughly equivalent at 2 bits, and do better than Nearest.
C.8 Section 6 (Evaluating Unbiased Rounding in LDLQ/OPTQ)
Note in our formulation for Adaptive Rounding with Linear feedback, the  $\mathcal{Q}$  subroutine could be biased, or unbiased. It is typical to perform biased rounding in practice; here we investigate if there is
WBits	LDLQ/OPTQ	LDLQ-RG	Greedy	Near
4	104.09	105.23	120.74	301.18
3	529.53	475.25	537.98	1,308.05
2	2,554.89	2,291.02	2,587.17	5,971.69
Table 14: Weighted average of proxy Loss  $\operatorname{tr}\left((\hat{W} - W)H(\hat{W} - W)^T\right)$  over OPT models 125m to 2.7b. Proxy is averaged over models normalized by their model dimension (768, 1024, 2048, 2560) respectively, to ensure proxy loss is comparable across models of different size. We do not conduct any processing in the proxy evaluation. Trends in the proxy largely reflect end-to-end results: at 2 bits OPTQ, LDLQ-RG, and Greedy are roughly equivalent, and all do better than nearest.
AVERAGE(Perplexity Unbiased - Perplexity Biased) on Wiki, PTB, C4 (↓)
Incoherence Processing	Baseline Processing
WBits	125m	350m	1.3b	2.7b	125m	350m	1.3b
4	1.23	0.73	0.79	0.19	27.81	5.58	1.62
3	13.26	7.79	2.14	4.66	880.4	499.4	28.63
2	2,501	18,732	544.8	2,251	241.3	17,945	4,831
any benefit to switching to unbiased rounding schemes. Table 15 computes the average perplexity difference (i.e. unbiased - biased) for LDLQ/OPTQ on WikiText2, PTB, and C4. That is, we run LDLQ with the  $\mathcal{Q}$  subroutine as stochastic rounding, instead of nearest. The average difference is positive (and large for 2 and 3 bits), meaning that unbiased rounding performs worse than biased (i.e. nearest) across OPT models  $125\mathrm{m}$  to 2.7b. These results indicate that in practice, we want to stick with biased rounding schemes.
C.9 Section 6 (Evaluating Algorithm 5 Which Accounts for Clamping)