SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remarks. This lemma means that multiplying by a random matrix in this family suffices to make a matrix incoherent with parameter  $\mu$  only poly-logarithmic in the matrix size. In our experiments we use  $k = 2$  factors to construct the orthogonal matrices  $U, V$ .
4.2 Additional Heuristics
We outline QuIP pre-processing and post-processing in Algorithms 1 and 2, respectively. In line 5 of Algorithm 1, we apply the aforementioned fast orthogonal multiplication procedure to ensure  $W$  and  $H$  are incoherent. We also randomly permute entries at the fast matrix multiplication step to prevent any correlation between attention heads from worsening performance. We introduce a number of additional heuristic improvements that further improve performance.
Incoherence-Based Heuristics. Line 4 diagonally rescales  $W$  and  $H$  to minimize  $\ell(\hat{W}) \approx \operatorname{tr}(H) \| W \|_F^2$ , effectively trading off the spectrum of these matrices to find a minimum. Motivated by the incoherence of  $W$ , Line 6 computes the quantization range depending on the spectrum  $\| W \|_F$ , instead of the typical  $\max_{i,j} |W_{ij}|$ . Our full QuIP procedure is described in Algorithm 3, which contains calls to the pre- and post-processing sub-steps in Algorithms 1 and 2.
Greedy local search. Our basic procedure yields a good initial guess with error guarantees. We can further lower the proxy loss by running coordinate descent after LDLQ (but before post-processing), updating the weights in the same order as in the initial pass. See Supplement B for full details.
Algorithm 3 QuIP: Quantization with Incoherence Processing
Require:  $b\in \mathbb{N},H\in \mathbb{R}^{n\times n}$  SPD,  $W\in \mathbb{R}^{m\times n}$ $\mathcal{Q}\in \{\mathrm{Near},\mathrm{Stoch}\}$ $\rho \in \mathbb{R}_+$ $\alpha \in [0,1]$  1:  $\hat{W},H,s,\tilde{D}\gets \mathrm{Alg}1(b,H,W,\rho ,\alpha)$  ▷QuIP Incoherence Pre-Processing 2:  $H = (\dot{U} +I)D(\dot{U} +I)^{-1}$  LDL decomposition 3: for  $k\in \{1,\dots ,n\}$  do  $\hat{W}_k\gets \mathrm{clamp}(\mathcal{Q}(W_k + (W - \hat{W})\dot{U}_k),0,2^b -1)$  LDLQ 4: return  $\hat{W}\gets \mathrm{Alg}2(b,H,\hat{W},s,\tilde{D})$  ▷QuIP Incoherence Post-Processing
5 Extensions and Further Analyses
5.1 OPTQ is a Special Case of LDLQ
We prove a novel theoretical insight: QuIP without incoherence processing (i.e., LDLQ) is equivalent to a more efficient version of the OPTQ algorithm. That is, OPTQ falls under our class of adaptive rounding procedures with linear feedback, and is within-class optimal.
Theorem 6. OTPQ [8] falls within the class of adaptive rounding procedures with linear feedback as described by Eq. (2), and is equivalent to LDLQ in Section 3.
Remarks. To the best of our knowledge, this equivalence yields the first theoretical analysis of OPTQ. Even though the two methods are equivalent, LDLQ is more efficient. OPTQ's implementation requires a matrix inversion of  $H$ , and two Cholesky decompositions. Our implementation of LDLQ performs no matrix inversion, and only one Cholesky decomposition.
Empirical Verification. The quantized outputs of the OPTQ implementation [8] are shown to be exactly identical to the outputs of our LDLQ implementation. Synthetic random data was used, with  $W \sim \mathrm{Unif}[0,1]^{1000 \times 1000}$ . Full details can be found in Supplement C.
5.2 A Bound for Rounding to a Finite Grid
In Section 3, we saw that LDLQ (equivalently, OPTQ) is optimal for minimizing the adaptive rounding objective. However, this analysis assumed rounding to the integers. In practice, we do not want to round  $W$  just to the integers, but instead to scale it, shift it, and round it a finite subset corresponding to a  $b$ -bit integer. To do this, the "real" LDLQ algorithm uses a clamp operation to restrict the range of quantized values. Is LDLQ still optimal when this small change is made? It turns out that the answer is no, as the following concrete example illustrates.
Figure 4: LDLQ underperforms.
Finite Grid Counterexample. Figure 4 illustrates the behavior of LDLQ and other rounding methods—when restricted via clamping to a finite 4-bit grid  $[0,15]$ —on a particular example where  $H$  is a (cleverly chosen) small perturbation of  $(I_{n} + \mathbf{1}_{n\times n} - e_{n}e_{n}^{T}) / n$ , and  $W$  has  $m = 16$  and is a small perturbation of  $\mathbf{1}_{m\times n} / 2$ . Details of the setup appear in Supplement C. The figure shows that clamped LDLQ with nearest rounding is asymptotically worse, and the clamping to the finite grid is what causes it to be worse in this case.
Note that in our experiments in practice, OPTQ has been shown to soundly beat nearest rounding. This clamping issue does not seem to arise in practice; however, since it is possible we do need to take it into account to prove useful end-to-end bounds.
A Procedure With a Bound. In order to address the above issues in theory, here we describe a method that acts to restrict the value of  $|\hat{W}_{ij} - W_{ij}|$ , so that the rounded weights will remain inside the grid if  $W$  is sufficiently far inside. We do this via the optimization problem with hyperparameter  $c$
$$
\begin{array}{l} \text {m i n i m i z e :} \operatorname {t r} \left(H R ^ {T} R\right) \\ \text {o v e r :} R \text {u n i t u p p e r t r i a n g u l a r} \tag {7} \\ \end{array}
$$
subject to:  $e_i^T R^T Re_i \leq 1 + c$ ,  $\forall i \in \{1, \dots, n\}$ .
Our "fixed" algorithm solves this convex problem (e.g. with ADMM), then runs QuIP using stochastic rounding and  $\bar{U} = R^{-1} - I$  in place of the LDL decomposition. Observe that for sufficiently large  $c$ , this is exactly equivalent to base QuIP, since the solution of that optimization problem is given by the LDL decomposition when the constraint is dropped. Doing this (the full algorithm is given in the supplemental) yields the following theorem.
Theorem 7. Suppose that we run Algorithm 5 (Supplement) to quantize a matrix  $W \in \mathbb{R}^{m \times n}$  by solving the objective (7). Then there exists an assignment of the algorithm's hyperparameters  $c$  and  $\rho$  such that with probability at least  $1 - \delta$ , all the quantized weights will be in range (no overflow or need for clipping) and
$$
\operatorname {t r} \left((\hat {W} - W) H (\hat {W} - W) ^ {T}\right) = \tilde {\mathcal {O}} \left(\frac {1}{n ^ {2} 4 ^ {b}} \operatorname {t r} \left(H ^ {1 / 2}\right) ^ {2} \| W \| _ {F} ^ {2}\right).
$$
In practice, because clamping rarely causes issues, and because of the significant additional compute needed to solve this program, we always just use QuIP as described in the previous sections, which is equivalent to setting  $c$  large and using nearest rounding.
Overview. We quantize the OPT [35] family of models (up to 66B parameters) and Llama 2 70B [27] using various quantization and processing methods. QuIP is superior to OPTQ and other baselines across all model sizes and evaluation tasks. Most interestingly, incoherence processing yields excellent performance using as little as two bits per weight when paired with any of the quantization methods we consider (including nearest rounding). Two-bit quantization with QuIP is viable at even moderate model sizes (1B parameters), a regime where other two-bit quantization methods fail. At the largest model sizes, the difference between 2-bit and 16-bit weight performance becomes small. We compare the throughput of QuIP with OPTQ's efficient implementation on language generation and show that it is not much slower. Additional results on the effectiveness of the proxy loss, unbiased rounding, and Algorithm 5 are presented in the Supplement C.
Setup. The experimental infrastructure is built on top of OPTQ's [8] repository which is implemented in PyTorch [23]. We quantize the HuggingFace implementations of the OPT and Llama 2 model families. All models are quantized on a single GPU, with up to 48GB of memory. Our calibration set is the same as OPTQ; 128 random 2048 token segments from the C4 dataset [25] consisting of generic text data from crawled websites. Therefore, no task-specific data is viewed when quantizing. Following OPTQ, quantization is performed one Transformer block at a time: loaded into GPU memory, the Hessian computed, and then the weights quantized. The current block's inputs are then passed through the quantized block to produce inputs for the following block. The Hessian is computed from the quantized Transformer up to that point rather than from the full precision model; like OPTQ, we find this improves quantization. Further details on the setup can be found in Supplement C, including a description of the computational resources used to perform the experiments.
Methods. We evaluate compositions of several quantization and pre/post processing methods. For quantization methods, we evaluate nearest rounding, LDLQ (or OPTQ), and two variations. LDLQ-RG re-orders the weights based on  $\mathrm{diag}(H)$  to modify the quantization order and adds further greedy updates to the proxy. "Greedy" performs the greedy updates only. We evaluate the baseline preprocessing from OPTQ which adds  $H\gets H + \alpha *\mathrm{mean}(\mathrm{diag}(H))I$  for numerical stability. We also evaluate our incoherence processing in Algorithms 1 and 2, denoted as "IncP". With this notation  $\mathrm{QuIP} = \mathrm{LDLQ} + \mathrm{IncP}$ , and  $\mathrm{QuIP - RG} = \mathrm{LDLQ - RG} + \mathrm{IncP}$ .
Datasets. We evaluate on the following language generation tasks: WikiText2 [17], Penn Treebank (PTB) [16], and C4. We also evaluate on zero-shot tasks, including LAMBADA (LAMB) [21], ARC Easy (ArcE) [1], PiQA [26], and StoryCloze (SC) [18]. See Supplement C for the full set of results.
Main Results. QuIP is the first PTQ procedure to achieve good quantization at two bits per weight, across a variety of LLM sizes and evaluation tasks. In Figure 5 we compare QuIP and OPTQ when quantizing to 2 and 3 bits per weight (4-bit quantization works equally well for both methods); we evaluate OPT models (up to 66B) on PTB, C4, ARC Easy, and LAMBADA. QuIP is superior to OPTQ across the model sizes and evaluation tasks. At three bits, QuIP matches the full precision model reasonably well. At two bits and for larger LLMs ( $>2\mathrm{B}$  parameters), QuIP begins to approach the performance of the full precision model. As model size increases, so does the quality of QuIP's
Figure 5: Quantizing OPT models up to 66B parameters. Our method QuIP is the first PTQ procedure to achieve good quantization at 2 bits per weight, across a variety of model sizes and evaluation tasks.
OPTQ	QuIP (Ours)
WBits	Wiki↓	C4↓	ArcE↑	PiQA↑	SC↑	Wiki↓	C4↓	ArcE↑	PiQA↑	SC↑
16	3.319	5.709	59.72	80.90	79.95	3.319	5.709	59.72	80.90	79.95
4	3.596	5.905	58.96	80.52	79.12	3.531	5.869	59.81	80.47	79.63
3	4.907	7.099	54.38	78.56	77.72	3.853	6.135	59.81	80.25	79.31
2	123.908	70.541	25.34	50.54	51.75	6.326	8.937	54.38	75.08	75.37
Table 1: Quantizing Llama 2 70B with QuIP and OPTQ, and evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits.
Baseline Processing	Incoherence Processing (Ours)
WBits	Wiki↓	PTB↓	C4↓	ArcE↑	LAMB↑	Wiki↓	PTB↓	C4↓	ArcE↑	LAMB↑
16	9.56	14.04	11.45	65.40	72.40	9.56	14.04	11.45	65.40	72.40
OPTQ	QuIP
4	9.59	14.22	11.56	64.77	72.39	9.60	14.18	11.50	65.32	73.20
3	10.32	15.36	12.23	60.19	68.89	9.79	14.37	11.66	65.28	72.68
2	71.70	88.19	29.59	42.47	25.77	11.48	17.40	13.55	57.87	65.24
LDLQ-RG	QuIP-RG
4	9.64	14.20	11.56	63.76	71.94	9.66	14.11	11.51	64.86	71.86
3	10.31	15.15	12.15	63.43	69.78	9.75	14.44	11.68	63.51	71.53
2	49.40	73.45	29.12	41.20	26.35	11.68	16.94	13.44	59.51	62.31
Greedy	Greedy + IncP
4	9.69	14.33	11.59	63.09	72.37	9.72	14.23	11.52	65.99	71.71
3	13.63	23.05	16.30	50.51	56.76	9.92	14.45	11.71	63.80	71.38
2	4816.6	3473.81	3183.2	26.30	0.00	11.59	17.39	13.30	58.80	64.47
Near	Near + IncP
4	10.77	15.41	13.52	61.28	70.42	9.77	14.16	11.53	64.06	71.41
3	1564.9	1526.2	1808.2	34.47	1.73	9.89	14.49	11.74	64.06	71.41
2	41547.8	34348.6	24815.7	25.80	0.00	12.04	18.12	14.11	56.36	60.64
Table 2: Quantizing OPT-30b with various quantization and processing methods, and evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.
Wbits	Rescale	Incoherence	Rescale+Incoherence	Rescale+Incoherence+Quant Range
4	24.30	24.32	24.05	23.89
3	32.62	42.28	31.32	26.36
2-bit quantization. We provide plots on the remaining datasets in Supplement C. Note that the dip in OPTQ on OPT-66B is documented in their paper.
Table 1 shows the results of quantizing Llama 2 70B using QuIP and OPTQ. Again, QuIP achieves good quantization at two bits while OPTQ does not.
Incoherence Processing Ablation. Table 2 shows all combinations of quantization and processing methods evaluated on OPT-30B. At lower weight bits, QuIP's incoherence processing dramatically improves the performance of all quantization methods, across all evaluation tasks. Remarkably, all quantization methods—even nearest—are viable at two bits with our incoherence processing. Our modifications in QuIP-RG sometimes give an improvement over QuIP, but further study is required to evaluate these modifications. Figures for OPT-125M to 13B are in Supplement C.
Table 3: Ablating sub-steps of QuIP's incoherence processing, see Algorithm 1. Perplexities are averaged over WikiText2, PTB, and C4 for OPT-350m.
Method	Throughput
QuIP	81ms
OPTQ	53ms
Throughput Comparison. We evaluate the additional overhead of our incoherence processing during model inference by modifying OPTQ's efficient forward pass. OPTQ's implementation contains a quantized-matrix full-precision-vector product kernel and was shown to offer speedups over a FP16 baseline. Our incoherence processing additions are performed in PyTorch. Table 4 shows that our QuIP implementation is about  $1.5 \times$  slower than OPTQ.
Further Ablation. QuIP's incoherence processing contains several sub-steps. Table 3 shows their relative contributions; all are necessary for the full improvement. Table 5 shows that the random permutation step within the fast orthogonal multiplication also significantly reduces perplexity.
This paper introduced quantization with incoherence processing (QuIP), an algorithm consisting of (1) an optimal adaptive rounding procedure which minimizes a quadratic proxy of the weight error, and (2) efficient pre- and post-processing to ensure the incoherence of the weight and Hessian matrices by multiplying them by a Kronecker product of random orthogonal matrices. We showed that QuIP quantization is optimal in a general class of adaptive rounding methods with linear feedback; this theoretical analysis is the first for any quantization algorithm that scales to LLM-sized models.
Empirically, QuIP achieves the first viable two-bit quantization results for LLMs, especially at large model sizes, hinting at the feasibility of accurate 2-bit inference in LLMs.
Table 4: Average per-token throughput (batch size 1) when generating sequences of length 128 with OPT-66B on an A6000 GPU.
Wbits	ΔPerplexity from random permute↓
4	-0.22
3	-9.96
2	-74.2
Table 5: Ablating random permutation within fast orthogonal multiplication. Differences in perplexity are averaged over WikiText2, PTB, and C4 for OPT-125m.
This work was partially funded by the National Science Foundation under awards DGE-1922551, CAREER awards 2046760 and 2145577, by the National Institute of Health under award MIRA R35GM151243, and a gift from CISCO.
Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[28] Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In International Conference on Machine Learning. PMLR, 2020.
[29] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. In Conference on Neural Information Processing Systems, 2022.
[30] BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, and François Yvon et. al. Bloom: A 176b-parameter open-access multilingual language model, 2023.
[31] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438, 2023.
[32] Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael W. Mahoney, and Kurt Keutzer. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning. PMLR, 2021.
[33] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Conference on Neural Information Processing Systems, 2022.
[34] Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Luzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089, 2023.
[35] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
A Checklist
Our work pushes the quantization of large language models into the 2 bits per weight regime. Our aim is to drive foundational research on theoretical and empirical aspects of quantization. The ultimate goal is to enable more powerful LLMs to run more efficiently. However our work is unaware to what ends those LLMs are used.
The adaptive rounding [3] proxy objective considers each layer in isolation; it remains to be seen what other computationally tractable proxies could improve quantization. For example quantization methods do exist which consider interactions between layers, but so far have been too computationally expensive to be applied to the largest open LLMS.
A.3 Experiments, Reproducibility
Our code is included in the Supplement. See the included README for instructions on how to reproduce the various experiments, including random seeds. The code also downloads all datasets used to quantize or evaluate the models.
B Additional Method Clarifications
B.1 Subsection 4.2 (Incoherence-Based Heuristics)
Line 4 diagonally rescales  $W$  and  $H$  to minimize  $\ell(\hat{W}) \approx \operatorname{tr}(H) \| W \|_F^2$ , effectively trading off the spectrum of these matrices to find a minimum. Note to minimize  $\operatorname{tr}\left(D^{-1} HD^{-1}\right) \| WD \|_F^2 = \left(\sum_{i=1}^{n} H_{ii} / D_i^2\right)\left(\sum_{i=1}^{n} D_i^2 \| W_i \|^2\right)$  implies that  $D_i = \sqrt{H_{ii} / \| W_i \|}$ . Motivated by the incoherence of  $W$ , Line 6 computes the quantization range depending on the spectrum  $\| W \|_F$ , instead of the typical  $\max_{i,j} |W_{ij}|$ . The parameter  $\rho$  controls the quantization range; we tune it and find that a value of 2.4 works well across all our experiments. We use  $\rho = 2.4$  consistently across all experiments. Our full QuIP procedure is described in Algorithm 3, which contains calls to the pre- and post-processing sub-steps in Algorithms 1 and 2.
B.2 Subsection 4.2 (Greedy Updates)
In this subsection, we describe the "greedy local search" method mentioned in the main body of the paper in more detail. The basic idea is to iterate over coordinates of the weights in the same order as the initial quantization method, modifying each weight in turn—but still restricting it to be a representable quantized value—so as to minimize the proxy loss while keeping the other weights fixed. These greedy updates amount to coordinate descent on the proxy loss, but restricted to the quantization grid. Greedy updates can be performed after any initial quantization method, or as a standalone method. When performed after an initial quantization method, greedy local search is a descent method because the individual weight updates cannot increase the loss, but when performed alone, these greedy updates are not a descent method because the initial point  $(\hat{W} = W)$  is not feasible because it contains unquantized values that are off the representable quantization grid. Concretely, a greedy update of weight  $(i,j)$  to the grid  $\{0,1,\dots,2^b -1\}$  does the following, where  $\ell$  is the proxy loss:
$$
\hat {W} _ {i j} \gets \arg \min _ {z \in \{0, 1, \dots , 2 ^ {b} - 1 \}} \ell (\hat {W} - e _ {i} e _ {j} ^ {T} \hat {W} _ {i j} + e _ {i} e _ {j} ^ {T} z).
$$
(Note that  $\hat{W} - e_i e_j^T \hat{W}_{ij} + e_i e_j^T z$  is the result of setting the  $(i, j)$ th entry of  $\hat{W}$  to  $z$ .) A full pass of greedy updates constitutes  $mn$  of these updates performed in the same order as LDLQ. This algorithm is very simple, since it is just greedy coordinate descent. In the rest of this subsection, we will give a bit more intuition about this method by showing how this greedy algorithm falls within our framework of adaptive rounding with linear feedback.
Algorithm 4 Greedy Updates: A Single Pass
Require:  $b \in \mathbb{N}$ ,  $H \in \mathbb{R}^{n \times n}$  SPD, weights  $W \in \mathbb{R}^{m \times n}$ , initial guess  $\tilde{W}$
1:  $\hat{W}\gets \tilde{W}$
2:  $U\gets (H\odot M)\mathrm{diag}(H)^{-1}$ $\triangleright M$  is the strictly upper triangular mask
3:  $V\gets W - (\tilde{W} -W)(H\odot M^T)\mathrm{diag}(H)^{-1}$ $\triangleright$  can skip if  $\tilde{W} = W$  by setting  $V\leftarrow W$
4: for  $k \in \{1, \dots, n\}$  do  $\hat{W}_k \gets \mathrm{clamp}(\mathcal{Q}_{\mathrm{near}}(V_k + (W - \hat{W})U_k), 0, 2^b - 1)$
5: return  $\hat{W}$
An application of greedy local search as a single-pass stand-alone method falls under our Adaptive Rounding with Linear Feedback framework, with the linear feedback set to  $U = (H \odot M) \mathrm{diag}(H)^{-1}$ , where  $M$  is the strictly upper triangular mask and  $\odot$  denotes the Hadamard (entrywise) product, as we will derive below. For ease of explanation consider a single (row) weight vector  $w \in \mathbb{R}^{1 \times n}$ . When looking only at column  $j$ , the proxy loss from setting  $\hat{w}_j$  to  $z$  is
$$
\begin{array}{l} \ell (\hat {w} - \hat {w} e _ {j} e _ {j} ^ {T} + z e _ {j} ^ {T}) = (\hat {w} - w) H (\hat {w} - w) ^ {T} + 2 (z e _ {j} ^ {T} - \hat {w} e _ {j} e _ {j} ^ {T}) H (\hat {w} - w) ^ {T} \\ + \left(z e _ {j} ^ {T} - \hat {w} e _ {j} e _ {j} ^ {T}\right) H \left(z e _ {j} ^ {T} - \hat {w} e _ {j} e _ {j} ^ {T}\right) ^ {T}. \\ \end{array}
$$
This is just a quadratic function in  $z$ , and so its minimum value on the grid  $\{0, 1, \dots, 2^b - 1\}$  will just be its minimum value on  $\mathbb{R}$  rounded to that grid. To find this minimum over  $\mathbb{R}$ , we differentiate to minimize, yielding
$$
0 = 2 e _ {j} ^ {T} H (\hat {w} - w) ^ {T} + 2 e _ {j} ^ {T} H (z e _ {j} ^ {T} - \hat {w} e _ {j} e _ {j} ^ {T}) ^ {T},
$$
and solving for  $z$
$$
z = - \frac {(\hat {w} - \hat {w} e _ {j} e _ {j} ^ {T} - w) H e _ {j}}{e _ {j} ^ {T} H e _ {j}} = \hat {w} e _ {j} - \frac {(\hat {w} - w) H e _ {j}}{e _ {j} ^ {T} H e _ {j}}. \tag {8}
$$
Since when we use greedy local search as a stand-alone method, we have not updated  $\hat{w}_j$  yet, at this point  $\hat{w}e_j = w e_j$ , and so this means that a single step of greedy updates looks like
$$
\hat {w} e _ {j} \gets \mathcal {Q} \left(w e _ {j} - (\hat {w} - w) \frac {H e _ {j}}{e _ {j} ^ {T} H e _ {j}}\right)
$$
for  $\mathcal{Q}$  referring to nearest rounding with the necessary clamping. Since  $\hat{w} - w$  is zero for all entries following the  $j$ th one, this is equivalent to
$$
\hat {w} e _ {j} \leftarrow \mathcal {Q} (w e _ {j} - (\hat {w} - w) U e _ {j})
$$