Improving the Straight-Through Estimator with Zeroth-Order Information
Authors: Ningfeng Yang
Tor M. Aamodt
Venue: NeurIPS 2025 poster ()
Keywords: quantization
straight-through estimator
quantization-aware training
Primary area: deep_learning

Abstract: We study the problem of training neural networks with quantized parameters.
   Learning low-precision quantized parameters by enabling computation of gradients via the Straight-Through Estimator (STE) can be challenging. 
 While the STE enables back-propagation, which is a first-order method, recent works have explored the use of zeroth-order (ZO) gradient descent for fine-tuning.
  We note that the STE provides high-quality biased gradients, and ZO gradients are unbiased but can be expensive.
  We thus propose First-Order-Guided Zeroth-Order Gradient Descent (FOGZO) that reduces STE bias while reducing computations relative to ZO methods. 
  Empirically, we show FOGZO improves the tradeoff between quality and training time in Quantization-Aware Pre-Training. 
  Specifically, versus STE at the same number of iterations, we show a 1-8% accuracy improvement for DeiT Tiny/Small, 1-2% accuracy improvement on ResNet 18/50, and 1-22 perplexity point improvement for LLaMA models with up to 0.3 billion parameters.  For the same loss, FOGZO yields a 796$\times$ reduction in computation versus n-SPSA for a 2-layer MLP on MNIST. Code is available at [https://github.com/1733116199/fogzo](https://github.com/1733116199/fogzo).

Improving the Straight-Through Estimator with Zeroth-Order Information
Ningfeng Yang University of British Columbia nxyangfece ubc.
Tor M. Aamodt University of British Columbia aamodt ubc.
Abstract
We study the problem of training neural networks with quantized parameters. Learn- ing low-precision quantized parameters by enabling computation of gradients via the Frraight-Througe Estimator (STE) can be challenging. While the STE enables back propaarop,,,tion, which is a first-order method, recent works have explored the use of zeroth-order (ZO) gradient descent for fine-tuning. We note that the STE provides high-quality biased gradients, and 70 gradients are unbiased but can be expensive. We thus propose First-Order Guideed Zeroth Eeroth-Order Gradient Descent (FOGZO) that reduces STE bias while reducing computations relative to ZO meth- ods. Empirically, we show FOGZO improves the tradeoff between quality and train- ing time in Quantization-ANee number of iterations, we show a 1-8% accuracy improvement for DeiT Tiny/Small, 1-2% accuracy improvement on ResNet 18/50. and 1-22 perplexity point improve- ment for LLaMA models with up to 0.3 billion parameters. For the same loss, FOGZO yields 796x reduction in computation versus n-SPSA for 2-layer ML. on MNIST. Code is available at http://hhhb..coc//333111111/////
Pre-Training Specifically, versus STE at the same
Introduction
Quantization is effective method reduce the compute and memory complexity of Deep Neural Networks (DNNs), enabling DNNs to be deployed to sscouree-constraine platforms. However, naively applying quantization often leads detrimental accuracy degradation. While Post-Training Quantization (PTQ) (23 43
20) methods can mitigate such accuracy degradation and produce high-quality 4tt models with little computation and memory overhead, Quantization-
29
24
12) methods can produce even more accurate models at lower precisions by integrating quantization into the training loop. Lower preer-precision models can be important in edge applications where large hardware deployments are expensive to change. As such, this work focuses on improving QAT.
Aware Training (QAT) B35
B3
28
21
QAT methods can be further taxonomized into Duantization-Awar Pre-Training (QAPT) 35 21
and Quantization-Awanl Fine-Tunine (QAFT) 330 12 (54). QAFT methods initialize low-precision model from a full-precision pre-trained checkpoint and subsequently fine-tune the model for short duration (12) small downstream dataset. On the other hand, QAPT methods randomly initialize a low-precision model and pre-train for long durations on large-scale pre-training datasets (35). Compared to first pre- pre-training full precision and subsequently applying PTQ/QAFT, QAPT can have higher pre-training speed and lower pre-training cost, when weights and activations are quantized, the forward pass may be computed in low precision 5T) This work aims improve the accuracy/prediction quality of QAPT.
A core challenge of QAT is the non-differentiabiliy of the objective function. The rounding function used to discretize weights/activations has zero gradient almost everywhere and non-differentiabl rounding boundaries. A gradient that zero almost everywhere suggests parameters remain
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

unchanged during gradient descent. To work around this, the Straight-Through Estimator (19) 3 (STE) is a well-known method to enable BackProp through non differentiable operations such as rounding, as if they were the identity function or other smooth functions (52). At high precision, the STE performs surprisingly well (3) despite having little theoretical justification (0) However, at low precision, the STE does not work well (33 25). On the other hand, zeroth-order (ZO) methods, which estimate gradients using finite differences, while more theoretically sound (3) are computationally intensive and may yield inferior-quality results for fixed training budget.
We discuss the strengths and weaknesses of the STE and ZO methods and propose FOGZO, which uses the efficiency of the STE to accelerate n-SPSA (45) 46 BT), a ZO method. Our heuristic derivation suggests that FOGZO can suppress the STE gradient when points in an incorrect direction. Experiments across a wide variety of benchmarks suggest that FOGZO can outperform the STE, while requiring less computation than n-SPSA.
2 Background and Motivation
Consider neural network with continuous parameters 0 and objective function I(O).. While the loss depends on them as well, for notational convenience, we omit the network input, output, and dataset. To quantizo/diseretize 0, we augment the objective function to indicate a quantized loss function (()). We focus on parameter (as opposed to activation) quantization; thus, L can be expressed as:
(0) L(θ) where @ is a vector of quantized parameters
Ø = [ (sign(()~(())
for binarization
(I)
((cip((oumd(((-(()))
for multi-bit quantization
where f(b) anb is affine dequantization function that converts discretized weights y back to their original domair with a quantization scale a, and f-1 is the inverse of f. The functions sign, clip, and round handle the discretization. During QAT, É. is optimized by varying 0. However, when BackProp is naively applied, the gradient differentiable operations like sign and round, which are flat almost everywhere and non-differentiable rounding boundaries.
aL 36
is always zero because of the existence of non-
In this section, we discuss existing methods that address the non-differentiabiliyy of i, including first-order methods like the Straight-Throup Estimator and zeroth-order methods like n-SPSA We analyze the strengths and weaknesses of both methods, and finally discuss biased gradient estimators which can potentially address the weaknesses of zeroth-order methods.
2.1 The STE: Empirically Good but Theoretically Questionable
The STE circumvents the non non-differentiabiliyy of the round and sign functions by replacing their Jaco- bian with the Jacobian of smooth approximation. We denote these approximations as smooth amondh and smooth_sign, respectively. For example, to perform BackProp on Equation [ we use the
approximation shown Equation
ai ÖL ab 08 ab ag
00
0f Usign 0f
@smooth
sign
01-1
for binarization
where
0f
08
aclip
Damooth round 8 for multi-bit quantization
delip
Bround
8f
The most famous example of smooth round is the identity function but there are alternatives like Soft Rounding (2), HTGE (36), and Confidence-Guiie Guided Masking 25 For smooth sign, popular choices include Hardtanh (38), tanh (38), or ApproxSign (26). The smoothness of these approxima-
tions ensures the gradient is not zero everywhere. and therefore training can make progress. The STE easy to implement, as it only requires changing the backward pass of round/sign operators. The STE has the same computational complexity as regular training, as it only requires O(1) operations for the backward pass of every weight/activation. Despite the simplicity of the STE, works surprisingly and remains the most popular method for QAT.
well across variety of benchmarks (3 13 4. 28)
Note that other formulations of f possible, such ʃ(v) a( 2) where is point, (wb & where M is an outlier smoothing matrix like the Hadamard matrix. The value of & that minimizes L may differ from that which minimizes L.

---

Despite its empirical success and popularity, the theoretical optimality of the STE has remained a topic of debate. While some believe the STE is suboptimal and propose workarounds like Coordinate Descent (30), Stochastic Rounding (17), and Breudo-Quantizan
Noise (44), others provide theo- retical justification for using the STE on shallow networks and suggest potential for extending their theoretical insights to deeper networks (52). While some observe that specific choices of smooth approximations work better for specific benchmarks (50) 38 (2)), others suggest all smooth approx- imations are equally optimal, and that any observed differences are simply due to implicit weight initialization strategies and learning rate schedules (42). While the STE works well for high-precision networks, it has been shown (33) 25 to introduce parameter oscillations at rounding boundaries, which slows down convergence and reduces the efficiency of training.
Based on the above observations, we conjecture that the STE is a "sufficiently good" gradient estimator that generalizes to a wide variety of techniques, networks, and datasets. However, the STE occasionally produces gradients with "wrong" directions (see Section A). This can contribute
to undesirable behaviour (such as the above-noted parameter oscillations (3 25)). As elaborated
below, if we can correct the occasional "flaws" of the STE, we can achieve better-than-STE training accuracy without sacrificing ease of implementation and compute efficiency.
2.2 Theoretically Sound but Expensive Alternatives to the STE: Zeroth-Order Methods
Why has the STE stood the test of time and remained the most popular gradient estimator for 13 years, despite being theoretically questionable? We believe this is because the theoretically sound alternative, known zeroth-order (ZO) methods, is impractical for deep learning. ZO methods operate by perturbing weights and observing changes in loss. For example, Ghadimi et al. (15) and Gasnikov et al. (14) showcase that, given an arbitrary non- con-differentiable function L, it is possible to construct a randomly smoothed approximation Limoooh using Equation 3 where c is a positive hyperparameter that represents the smoothness, and p(u) is perturbation probability distributior 3 symmetric, zero-mean, and vatterrrianceissoooppicc. Once we have the smooth approximation Lmmothh
that is typically
its gradient can then be numerically measured using Equation 4 also known as n-SPSA 33 47)
Randomized Gradient Estimator 34), or the zeroth-order gradient estimator (27 1.
Lsmooth (0) = Eu-p(u)) [E + eu)]
(3)
1 L(0 + eu:) - (()-)u))
V Lamooth (0)
Us where u, ~ p(u)
(4)
T1
2e
In n-SPSA, the variance of the measured gradient correlates with the sample size In an ideal world
where afford infinite computation, or - oo, we can very accurately measure V Ligmoot (0)
with zero variance. In such a case, n-SPSA can outperform the STE. Intuitively, if training with Lmmmouth can reduce Tsmoouth it will also reduce f. by extension, since I.smooahh (0) R i.(0) for all o. However, in a practical system, using large can be prohibitively expensive, as it requires evaluating the loss 2n times. Therefore, reducing n when computation is a limited resource introduces large variance into the measured gradient, which hinders fast convergence. As such, ZO methods for deep learning (54) 31 6) either underperform compared to first-order methods or only work for specific training scenarios (e.g., fine-tuning).
This presents a dilemma. While n-SPSA has stronger theoretical guarantees than the STE, without sufficient computation, n-SPSA converges more slowly due to its large gradient variance. Is there way we can reduce the computation while still mitigating gradient variance?
2.3 Biased Gradient Estimators: A Potential to Reduce Computation
A method to reduce computation or increase efficiency without increasing gradient variance is to intentionally introduce bias into the gradient sampling process
(5). Unlike n-SPSA, which is unbiased and relies solely on a stochastic oracle, biased gradient estimators rely source of bias (9) Bias can be harmful but also potentially helpful if the to-be-estimated gradient is likely pointing in a similar direction and has similar magnitude that bias. A good source of bias typically has lower variance than unbiased estimators and is more efficient to compute. With a carefully selected bias
In some literature, p(u) is known stochastic oracle. A typical choice of p(u) is the standard normal.
3

---

source, biased gradient estimators can potentially match the performance of unbiased ones while using less computation (TT).
Given this insight, we ask the research question: Is it possible to combine the STE, 70 methods, and biased gradient estimators create method that outperforms the STE, retains some theoretical guarantees of ZO methods, while being computationally cheap like biased gradient estimators? In particular, is possible for the proposed method outperform the STE across a variety of benchmarks while still being practical for deep learning?
3 Proposed Technique
In this section, we present our proposed algorithm, FOGZO, first-order-uiided zeroth-order gradient estimator. FOGZO leverages the insight that since the STE achieves good empirical results, its estimated gradient must be mostly accurate except for a few edge cases (Section A where it may point in wrong directions. This makes the STE good source of bias to reduce both the computation and variance of n-SPSA. In addition, FOGZO utilizes a key property of n-SPSA- that it suppresses bad gradient samples- achieving a method that both relies on and is better than the STE. We present the full FOGZO algorithm, provide heuristic derivation, and finally discuss how properly select hyperparameters for FOGZO.
suppress the errors introduced by STE (Section 3.2) in these edge cases,
3.1 Algorithm
FOGZO can be summarized in Equations 5 and 6 The pseudocode equivalent is presented in Algorithm [ Instead of relying solely on the unbiased component u, in Equation FOGZO (Equation 5) additionally introduces a biased component using the gradient estimated by the STE, denoted g- While is possible use g directly, we note that n-SPSA can benefit from using oracles with properties like unit magnitude, zero mean, and symmetry. As such, we first normalize to unit vector 9 and then randomly choose between ğ and - with 50% chance each (denoted as s,ğ) to ensure the biased component is symmetric and zero-mean. In Equation 3 the biased component &: and unbiased component v, are mixed into single sample denoted as V/. We additionally introduce a mixing ratio 3. As we will discuss in Section 3.2 / determines the degree of trust in the STE. Similar to n-SPSA Equation 4 finite difference then used to generate the final FOGZO gradient estimation G' as shown in Equation 6 The mixing ratio 3 is linearly decayed from I to some hyperparameter Bmin < 1, as shown Algorithm line which we discuss more the next section.
02 BBsg VI Bru, where 8; ~ 2 Ber(0.5) 1,
and us p(u)
(5)
G 1(0 + tv,)) i(O -EV;)
2/
(6)
71
Notably, as shown in Algorithm FOGZO estimates the gradient twice, using both first-order (one
forward pass and one backward pass, Line 5) and zeroth-order (2n forward passes, Lines 8 16p. While this makes FOGZO more expensive than the STE, having a source of bias allows FOGZO to
be orders of magnitude faster (n E (()) than n-SPSA, while still retaining some of its benefits. In practice, we always use n = 1 for any deep models, which is equivalent to an additional computation cost of forward passes compared to training with the STE. Similar MeZO (31), we perturb and
restore the parameters in place for memory efficiency (Lines 10 12 and 14)
3.2 Heuristic Derivation
In this section, we perform heuristic derivation to understand the expected behaviour of the FOGZO gradient G. Utilizing the fact that Lamoath from Equation is approximation to L and first-order

---

Algorithm FOGZO Gradient Descent
1: function FOGZO(n, Smins p(())
2:
for ( 0 do
3:
=
Bmin) + 9min # linear beta decay from to Amin
4:
woode..weight.grad-
5:
model(data) lloss. baskwaadd # regular backward pass using some pre-configured STE g model. grad
6:
7:
model.weight.ger graphtttrr--00 for i < to n do
8:
9:
Generate v; from Equation
10:
wodel.weight += eu2 # perturb model weights in-place loss1 model(data).loss positive direction loss model.weight- -= 2en; # perturb model weights in-place loss2 model(data).loss negative direction loss wedel += ct; restore model weights in-place
11:
12:
13:
14:
model.weight grad (lossl loss2) / (2ne) * v; # Equation
16:
end for optimizer.stepO) gradient descent end for end function
18:
Taylor expansion, we derive Equation
7
i(0 + Eui) - L(O -Ev;)
Lumooth (0 + Evs)
- LLmoooth (0 -evs)
G
2
2c
26
1=1
Tammoath (0) + (ENs) I smooth (0)
s.mmooth (0) + (Ev:)² V T-mmooth (0)
1/
11
2c
i=1
1 72
o V Lamooth (f))e,
Σ 71
1 n
(o Lamoott (0))
1
Ug Lmmotth
n2
See Section for a discussion on the approximation error of the two = symbols in Equation 7
Substituting Equation 5 into Equation 7 and using the fact that is zero-mean and unit- variance/isotropic,, we then derive an approximation of E[G] Equation 8 which suggests with large n, the FOGZO gradient approximates a linear interpolation of the unbiased gradient V Lamoooh and its biased projection onto the rank-1 subspace described by the normalized STE gradient ğ. This biased projection term can interpreted as rank-1 gradient compression similar to MeZO (11). When small, the unbiased term has much larger variance than the biased term.
G[G] = Flvv?| Smmoothh (0)
El(V Bsg + /1 Bu)( 3s9 /1 /u)²18 Lssmoooth (0)
El + (1
B(1 () + suğ" ) Lamoooth (8)
(8)
(8gg" + +(1 V smooth (0)
= 3 "g""T"""" V i Issmooath (0) +(1 B)
Tssmooth (0)
biased
unbiased
Equation suggests FOGZO can suppress the flaws of the STE. When the STE is accurate, when points direction similar to L- 8 V I-amooth evaluates to a large scalar, and the biased term is allowed to contribute to G. However, when the STE is inaccurate, i.e. when it points in an orthogonal direction to V Lamooth the scalar g"T V Lamooohh evaluates to 0, and the biased term no longer contributes to G. In essence, the magnitude the biased term depends on the correctness of the STE gradient, or whether it points a similar direction to V Limmooth-
In addition, Equation provides insight on how to select the mixing ratio 3. When aligns with V Lemooth the STE gradient is more "trustworthy". As such,
large 3 to increase the
5

---

contributions of the biased term while reducing the unbiased term which has larger variance. However, when the STE misaligns with Lamooth the model is in an edge case where the STE gradient is inaccurate. In such cases, the biased term has relatively small magnitude, and training can stop making progress. To allow training to proceed, we can decrease / to increase the unbiased yet noisy term to prevent gradient signal from vanishing. Since the flawed behaviours of the STE tend to happen at the later stages of training (33), we decided to set 3 1 at the beginning of training, and gradually decay it to Smin as shown in Algorithm [ This coincides with the fact that at the later stages of training, the learning rate is much smaller, which makes the network more resistant to large gradient variance. Therefore, a small / which increases the unbiased term, along with the gradient variance it introduces, is more acceptable the later stages of training.
3.3 Selecting the Right Smoothness c and Perturbation p(u)
Choosing appropriate smoothness c and perturbation p(u) is critical for FOGZO's performance. yet tuning them by grid search is prohibitively expensive. We therefore seek principled way to configure these hyperparameters given a particular STE to be used with FOGZO. Our key insight is that every STE implicitly defines a form of smoothing. An STE estimates the Jacobian of non-differentiable operator (round sign) with the Jacobian a surrogate function (smooth_rouue round or smooth_sight and the surrogate can be viewed as the expected value of the original operator under some random perturbation. In other words, each STE corresponds to pair (E, p(u)) describing its implicit smoothness and perturbation distribution. If we can determine this implicit pair, we can use to guide the choice of FOGZO's (e, p()). Doing so aligns FOGZO's randomized smoothing behavior with that of the STE, while leaving the granularity of smoothing (operation-wiss versus full-model) as the only remaining difference. Formally, for an operator (z)) (such as sign) and its STE surrogate smooth (z), we assume:
smoothh (x) = Eu--p(a) [h(x + Eu)
and solve for (E, p()). As an example, consider the hardtanh STE (8), which approximates the clip(ær -1, 1). as the
sign function. We express the hardtanh function, defined as hardtanh(r) expected sign function under a perturbation distribution p(u) and solve for that p(u). For notational convenience, we denoted P as the CDF of p. which suggests P(-co) = 0 and P(co) 1.
hardtanh (x)
Eu [iign(x + ēu]]
sign(æ (u)p(u)du
(substitute Eu and du dz/E)
-x2z + 1 pl2/0)/Edz
(10)
sign(x +
-00
= P(-x//E) + P(-x)) + P(co) P(-x/6)
= 1 - 2P-2p(-2// hardtanh(z)
Differentiating both sides with respect to gives the corresponding PDF p(u).
2p(-x/E)f hardtanh (z) (substitute y -z/E and x = -Ey)
2p(u)/C hardtanh' Ty) = hardtanh' (čy)
(11)
p(y) [2/2, if 1/2 S y S 1/2
0, otherwise
Hence p(u) corresponds to a uniform distribution 1(11//,1/0). To ensure p(u) is unit- variancefisotropic, we can set ē = 1/v3 and p(n) = U(-V3. v3). Similarly, we solve for the and p(u) for the identity-TTR(), the tanh-STE (38) the ApproxSign-STI (26) and present our results in Equation 12 See Section D for proofs and further discussion.
T(x) = Eurp(n) [round(x + Eu)], where ē = 1/(2v3) and p(u) = U(-V3, V3)
tanh(x) Eu-p(a)) (sign(= i E.), where ē π/ / .7/122 and p(u) E(1 tanh? 2 (εu))
ApproxSign(z) = Eu-p(n) [sign(x r + Eu)]. where غ = 1/ and p(u) = tri(u/V6)v6
(12)

---

n-SPSA
8.0.006
0.9.90
8 0.999
B=1.000
STE
Gradien: Norm
2.2
Train Loss
2.1
100
Train
2.0
2.0
10-1
10"
10
10
10
10"
10
10
10
10-1
10
10a
FLOPs Per Iteration
FLOPs
Per iteration
Epsilon Scale
Figure Comparing the STE, n-SPSA and FOGZO on a 2-layer MLP with 2-bit weights.
If the round sign function directly applied the raw parameter 0, we e and p(u) = p(u). For a more general quantization setup like Equation [ where 0 is first processed by f i.. divided by a quantization scale and then discretized, set ae and p(u) - p(u). If there are multiple quantization scales in model, we use the averaged quantization scale as
4
Evaluation
We first compare the STE, n-SPSA, and FOGZO with experiments on shallow networks Section 4.1 as rumning n-SPSA with large n is infeasible on deeper networks. Section 4.22 we compare the STE and FOGZO on deeper networks with n = 1. Lastly, in Section 4.3 we compare the evaluation quality of FOGZO against the STE under the same training time.
4.1 Shallow Networks: Comparing the STE, n-SPSA, and FOGZO
In this section, we seek confirm the following 3 statements: the ZO method n-SPSA can outperform the STE when there is sufficient computation (large sample size n)); when there is insufficient computation, n-SPSA can underperform by large margin due its large gradient variance; FOGZO can significantly reduce this reliance computation, and can outperform the STE with only a small increase in computation budget.
We train MNIST on a 2-layer MI.P (784-10-10) with the identity STE, the 70 method n-SPSA, and FOGZO. We quantize weights 2-bit with constant quantization scale a that shared by both linear layers. Followed from the insight in Section 3.3 we use
ae a (2(3)) and p(u) U(-v3, v3). For FOGZO, we use / {0.666, 0.9, 0.999, 1}. For both FOGZO and n-SPSA, we use n E {1, 4, 10, 100, 1000, 7960}. We do not decay 3 for simplicity and simply keep it constant hyperparameter, We present our results in Figure For more hyperparameters, see Section
As shown in Figure left, while n-SPSA outperform the identity STE, it comes the cost of requiring orders of magnitude more computation. When n-SPSA has a low n. it underperforms due to the large gradient variance (shown in Figure middle). FOGZO addresses this limitation. As 3 increases, FOGZO consistently reduces gradient norm/variance and training loss for low computation. At B 0.999 and 1, FOGZO can outperform the STE, while requiring only additional forward passes per iteration. Crucially, when 3 = 1, FOGZO completely relies on its source of bias, and therefore cannot outperform the STE. This suggests the optimal / is close to, but not exactly, one.
In Figure right, additionally test the insights in Section 3.3 by setting caε and sweeping a range of epsilon scale c, while keeping n = and B = 0.999. Results suggest that c = is one optimal configuration, although larger values of equally good.
The experiments on shallow networks suggest FOGZO may be a promising alternative for the STE. At the cost of 2 additional forward passes per iteration, FOGZO can outperform the STE. This is especially useful for cases where a mild increase in training cost is acceptable as long as inference quality can be maximized. However, can FOGZO outperform the STE in deeper networks?
7

---

STE
const B-1
linear decay B
DeîT-T IM100 2-bit identity
LLaMA-9m 328M-token 2-bit identity
Resnet-18 IM100 2-bit identity
Train Loss
3.3
4.75
3.2
470
DeiT-T IM100 1-bit tanh
LLaMA-9m 328t4-token C4 1-bit tanh
Resnet-18 IM100 1-bit tanh
3.8
483
Train
482
0.80
10:12
10-11
10
10
1n
1071
10
to 10 10
10115
10-12
1e-ll
10-14
10:13
10
1073
DeiT- IM100 1-bit approxsign
LLaMA-9m 328M-token C4 1-bit approxsign
Resnet-18 IM100 1-bit approxsign
Train Loss
1.25
3.0-
5.0
10
10
10
10
10
1 min
B_min
1- B_mmin
Figure 2: Training Loss vs Amin- For 2-bit weights, we compare Cientity-STE. against Identity- FFGGOO For 1-bit weights, we compare tanh against tanh-FOGZO and ApproxSign against Approx FOGZO.
Model
Dataset
STe
IdeatityFOGGZZ
tanh
lanb-FOGZO
ApproxSign
ApproxSign FOGZO
Deiy-Tiny
Imagenett TO0
62,72
70,06
41.98
46.8
30.16
39.61
LLaMA-9m
328M C4 tokens
109.95
105.64
123.97
121.51
139.17
137
Resnet
Imagenet-I00
75.02
Table 1: Accuracy or perplexity of models with 1-bit or 2-bit weights and fixed a.
4.2 Deeper Networks: Comparing the STE and FOGZO
In this section, we confirm that in deeper networks, FOGZO can still outperform variety of STEs with a slight increase in computation budget. We select list of representative Deep Learning model architectures: Resnet (8)) DeiT [o 48), and LLaMA (49). Due to resource constraints, we train on the smaller variants of these architectures: Resnet-18, Resnet-50, ViT-Tiny, ViT-Small, and LLaMA- ( (9M, 20M, 30M, 50M, 100M, 200M]. We evaluate the most challenging QAT precisions: 2-bit and 1-bit. For datasets,
Imagenet- IK (41), Imagenet- 100(7), and C4 (39). For FOGZO, since it is impossible to use a large n, we simply fix n = 1. Unlike Section 4.1 here we use linearly scheduled Bs with the hyperparameter 8min. Similarly, set and p(u) based the insight in Section
4.2.1 FOGZO can Improve Variety of STES
In this section, we confirm that regardless of the STE used as source of bias, FOGZO can improve the gradient quality. We compare identity-STE against -oenz for 2-bit quan- tization, tanh-STE against tanh-FOGZO and ApproxSign-STE against ApproxSign-FOGZ0 for binary quantization. Similar to Section 4.1 for every linear and convolution layer, we inject a weight quantizer, where the quantization step size is fixed and shared by all layers in a model. For more hyperparameters, see Section E.2.
We perform Smin-Sweee and present the results in Figure 2 As shown, regardless of model architecture, dataset, and the chosen source of bias, FOGZO (blue) is capable of outperforming the STE (brown) within range of 8min ~ 1 (or 1 - 3min R 0). As Smin increases, training loss rapidly reduces. However, at Amin = the benefit of FOGZO saturates as training becomes completely dominated by STE-biased signal. At 8min
(labeled as "const 3 = 1"" FOGZO cannot outperform STE consistently. For each graph Figure 2 we select the 3min with the lowest training loss and report its evaluation quality in Table [ For vision models, we report top-1 accuracy. For language models, we report perplexity. As shown, with properly tuned Amin: FOGZO can outperform the STE by large margin.
8

---

STE
const B-1
linear decay B
DeiT-T IM100 LSQ+2-bur 2-bit identity
1LaMM-9m 328M-C4 2-tt identity
Resnet-18 IM100 L-Q+2-bit identity
3.20
0,46
Train
2.38
4.69
0.44
10
10
10 m0
10
19:12
10
10
10
10
30"
10
11
10
DeiT-S IM100 LSQ+2-bit identity
LLaMA-20m 1.38-C4 LSQ+2-biE identity
Resnet-50 IM100 LSQ+2-bit identity
Train .05S
2.82
0.45
2.00
4.100
10
10
10
10
10
10
10
10
10
10
13
10"
1 9_min
B_min
1
8_min
Figure 3: Training Loss vs 8min We compare LSQ+ SeentttdtttyySTT STE against L.SQ-Identiy-FOOG700
Model
Dataset
LSQ + Identity-STE
LSQ + Idemtity--yy
DeiT-Tiny
Imagenet- 100
3.E0.017/22.761101
199-:0..012/72..2:0000
DeiT-Smmll
Imagenet- 100
2.2-+0.012/7.5555000X
4.57:0.016/800006--000
DeiT-Tiny
Imagenet-
4.26:0.015763119-0...3
4.25::0.02 /5.5000055/633380..55
LLaMA-9m
328M C4 tokens
4.66:10.0600110431116..6 4.62:-0..00 /99.744449999
LLaMA-9m
3.28B C4 tokens
141+:0.020/2..54:1111 4.38+00.000/7990 79.
LLaMA-20m
1.3B C4 tokens
4084100.020/59.21111.. 4.0791000200588..
LLaMA-20m
13B C4 tokens
3.9410.008/50.855000
3.9310.008/50.661
Resnet-18 18
Imagenet- 100
0.45 0.004//88..33.....
0.44 /0,0098/00041..11
Resnet-50
Imagenet- 100
D.45:00.0182/8821110000
01050.0006//33.7--0000
Resnet-18
Imagenet-1K
1.493+t0.0032 66.51
1.487=10.0008
13:500
Table 2: Cross-entropy loss accuracy or perplexity of models with 2-bit weights and learnable a.
4.2.2 Integration with SOTA Methods and Scaling up
Weight-Only Quantization: We test FOGZO's compatibility with a SOTA QAT method LSQ (13). In Section 4.2.1| the quantization setup is rather naive the quantization scale is predetermined and shared by all layers. Here, we make a learnable and unique per layer and use the identity STE for 2-bit weight quantization. We follow the insight from Section 3.3 and use the average quantization scale a to set the smoothness. In addition, we further train models 2 to 4x larger than the ones in Section 4.2.1 to if FOGZO scales with parameter count. Lastly, increase a factor of 10x and test FOGZO's extensibility to larger datasets. This is done by switching from Imagenet-100 to Imagenet-IK: IK and increasing the number of C4 tokens by 10x. We report in Figure 3aadd Table the training loss and evaluation accuracy or perplexity of the STE and FOGZO with the optimal Amin- To ensure consistency, in each cell in Table [
dataset size by
we report the mean and two standard deviation of five different runs. Despite the change in quantizer design and model/dataset size, FOGZO is still eapable of outperforming the STE.
Weight-Activatie Quantization: We further evaluate the effectiveness of FOGZO SOTA weight- activation quantization methods QuEST (35) and LSQ. We use the publicly available source code of QuEST (35) and added the implementation of Algorithm and pre-train LLaMA models with n non-embedding parameters plus e embedding parameters, where E {30MM 50M, 100M, 200M/ and e E [65MM, 75M, 100M, 100M). ). For all experiments, we use a 8.in of le- le-10, quantize activation and weights of all linear layers to 2-bit, and pre-train with 100n C4 tokens. We present our results in Table Results suggest FOGZO can improve both QuEST and LSQ when activations and weights are quantized, and can even shrink the perplexity difference between these methods.
4.3 Reducing Training Time Overhead
In this section, we show the training time overhead of FOGZO can be drastically reduced by augmenting Algorithm|] [ to perform the first r% of training in STE mode, allowing FOGZO to achieve lower perplexity than the STE under the same training time. Concretely, we only execute lines 6 16 in Algorithm|] when > 7/100 * T. Note that 0 corresponds to the unmodified

---

Total Parameters IL +e
95M
95M
95M
95M
125M
125M
30M
30M
30M
30M
50M
50M
Tokens
3B
3B
3B
3B
5B
SB
Method
QuEST
QuEST-FOGZO
LSQ
LSQ-FOGZO
QUEST
QUEST-FOGZO
Perplexity
37.75
37.37
39.06
37.38
32.28
32.06
Total Parameters n
125M
125M
200M
200M
300M
300M
50M
50M
100M
100M
200M
200M
Tokens
5B
10B
10B
20B
Method
LSQ
LSQ-FOGZO
QuEST
QUEST-FOGZO
QuEST
QUEST-FOGZO
Perplexity
33.28
32.41
26.63
26.45
22.90
22.72
Table 3: Perplexity of LLaMA models trained using LSQ, LSQ-FOGZO, QuEST, and QUEST-
FOGZO with 2-bit weights and 2-bit activations.
Method
C4 Tokens (billion)
Evaluation Perplexity
Training Time (hours)
STE
3.174
38.69
3.3
90% STE+ + 10% FOGZO
38.67
3.3
STE
3.348
38.43
3.5
80% STE + 20% FOGZO
38.38
3.5
STE
3.522
38.25
3.7
70% STE + 30% FOGZO
37.93
3.7
Table 4: Training Time Perplexity of LLaMA-300 Nvidia RTX 5090
We LSQ-STE and LSQ-FOGZO train LLaMA model with 30M non-embedding parameters on one Nvidia RTX 5090 GPU and report the training time and C4 evaluation perplexity in Table For a fair comparison, adding more pre-training tokens, while decreasing the training time of the LSQ-FOGZO runs by choosing r {70, S0, 90). Results suggest FOGZO can outperform the STE when training time is identical. In addition, FOGZO can have higher data efficiency than STE, as FOGZO runs use less training data achieve lower perplexity.
Algorithm 1
artificially increase the training time of the LSQ-STE runs by
Related Works and Limitations
Most existing QAT methods use Backpropagation (40) to calculate gradient estimates, including but are not limited the STE/operation-wiss Jacobian estimators 3 26 50), Coordinate Descent (30), Stochastic Rounding (17), Bseudo-Quantizen
Noise (44), and EWGS (22). FOGZO is orthogonal
these works and may benefit from using them as a source bias. Existing zeroth-order (5th
31 () use only forward passes to estimate gradients, and are either specialized for fine-tuning, or underperforms compared first-order methods. FOGZO is specialized for pre-training, and the first perttalll--eeoth--rro orddr method to consistently outperform first-order method STE for DNNs.
Although FOGZO works well with standard mixed- preccssoo training methods (32 that keep weights in 32t FOGZO suffers from loss of precision when weights are stored in 16-bit due to the challenge of modifying weights with small updates. Because of this same challenge, SOTA LLM pre-training pipelines like ReepSeek-V3 (8) still use 32-bit weights. We expect that the field evolves, newer techniques will developed to reduce rounding error, enabling large-scale pre-training with BF16 weights. These newer techniques can then be used to relieve the numerical issues of FOGZO.
6 Conclusion
In this work, we compare two QAT methods, the STE and n-SPSA. We introduce FOGZO which utilizes the STE source of bias accelerate n-SPSA, creating the method FOGZO. We show that FOGZO can suppress the flaws of its bias source, allowing FOGZO to improve upon the STE. Experiments shows FOGZO improve the gradient quality of variety of STEs and for different model architectures and datasets. We hope FOGZO can influence future research in STEs as FOGZO provides a quantitative measure of STE gradient quality.
10

---

Acknowledgements
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) through a Discovery Grant and an NSERC Strategic Network Grant.
References
[1] Alekh Agarwal, Ofer Dekel, and Lin Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In Annual Conference Computational Learning Theory, 2010.
semanticscholarr httpsscholarrrrrooooooooooo orgo /tt.ppcc..//ppp..eeeeccccccoooor
Dtg//orrpusDD::18114553 118314530
[2] Eirikur Agustsson and Lucas Theis. Universally quantized neural compression. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20 Red Hook, NY, USA, 2020. Curran Associates Inc.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013. URL: httt::htt/:///rtt org/
abs/1308. 3432
arXiv: 3432
[4] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In 2020 IEFE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2978-2985, 2020. doi: 10. 1ww11110////// 1t..0099//WW555495388 2020 00356
[5] Jia Bi and Steve R. Gunn. A stochastic gradient method with biased estimation for faster nonconvex optimization. In Abhaya C. Nayak and Alok Sharma, editors, PRICAI 2019: Trends in Artificial Intelligence, pages 337-349, Cham, 2019. Springer International Publishing.
[6] Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Konstantinos Parasyris, Jiancheng Liu, Yihua Zhang. Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. In The Twelfth International Conference on Learning Representations, 2024. URL: htt.::tt/////eereeeet wwwpeeewwwwiiiweffrreeeeeee net/forum? id=qBWhjsNPEY
[7] Yubei Chen Chun-Hsiao Yeh. IN100pytorch: Pytorch implementation: Training resnets on imagenet-100 100.
http://ggtttuute.e com ccw.://anieelccyhhh/mageeeeeeeee http://aagneeee----o-o--o-oorroe 100-Pytorch 2022.
[8] DeepSeck-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. I.. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong. Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun. Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li. Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He. Yukun Zha. Yumfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou,
11

---

Z. F. Wu. Z. Z. Ren, Zehui Ren. Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report,
2025. URL: htttt:://saaii..ottttt /aw.rrg////a///ae hwtp://bs/241..... 2412. 19437 arXiv: 2412. 19437
[9] Yury Demidovich, Grigory Malinovsky, Igor Sokolov, and Peter Richtárik. A guide through the zoo of biased SGD. In seieth- Conference on Neural Information Processing Systems,
2023. URL: https::/opperreiieet wttp:/oorro/ddddddddddaa..
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogmition at scale. In International Conference on Learning Representations, 2021. URL: https://openrevies: wtt.://ww...e.ooooooooccoooooooa
[11] Derek Driggs, Jingwei Liang, and Carola-Bibiane SchAljnlieb On biased stochastic gradient estimation. Jourmal of Machine Learning Research, 23(24):1-43, 2022. URL: http://jmmr
org/papers/ 20133/20-31666
[12] DaYou Du Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao. Xiaowen Chu, and Ningyi Xu. BitDistiller: Unleashing the potential of sub-4-bit LLMs via self difl-istiilattin. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 102-116. Bangkok, Thailand, August 2024. Association for Computational Linguistics. URI:: https //aclanthology-oo org/ 2024. acl moll-ong... 7/ doi: 10: (18633/v1/2000 ac1-224..1loooo...
[13] Steven K. Esser. Jeffrey L. McKinstry. Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. In International Conference on Learning Representations, 2020. URL: http::////pppprrreiiseeeeefrrrrr
[14] Alexander Gasnikov, Darina Dvinskikh, Pavel Dvurechensky, Eduard Gorbunov, Aleksandr Beznosikov, and Alexander Lobanov. Randomized Gradient Methods in Convex Optimiza- tion, page 1-15. Springer International Publishing, September 2023. URL: http:
doi org.org//ie 000000000000700---000---------10 259--1
doi: 10. / - 3-0------0000--6-222
859-1
[15] Saeed Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. 2013. URI: httppp:://aard...oogaaaaaaa////a 5549 arxiv: 1309 5549
[16] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training ima- genet in hour, 2018. /tttps:///aattt
org/ abs/1706. 1706. 02677 arxiv: 1706. 02677
[17] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1737-1746, Lille, France, 07-09 Jul 2015. PMLR. URL:
mlr wwwpppeeeeeeep..p v37/ goptta1. html
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016 doi: 10::1001109///0
2016.90 690
[19] Geoffrey Hinton. Neural networks for machine learning. coursera, video lectures., 2012.
[20] Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li. Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: dense-and-sparse quantization. In Proceedings of the 4 International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
12

---

[21] Tanishq Kumar, Zachary Ankner, Benjamin Frederick Spector, Blake Bordelon, Niklas Muen- nighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Re, and Aditi Raghunathan. Scaling laws for precision. In The Thirleenth International Conference on Learning Representations, 2025. URL: htttt::::pppereeeeeeeee
[22] Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element- wise gradient scaling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[23] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized LLMs. In The eiihth- Annual Conference on Neural Information Processing Systems, 2024. URL: htL::https:// httpps://opeereviiiiiite wwt.:eee//oooueeee wtt/mrormmmiddii/// mp8u2P
net/
[24] Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. QLLM: Accurate and efficient ow-bitwidth quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL: /httts:///perrre net/forum?id PatforrmmmmdaPPpppuuWWma
[25] Shih- Shih-Yang Liu, Zechun Liu, and Kwang-Ting Cheng. Oscillation-free quantization for low-bit vision transformers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 21813 mlr wwwps/eeoe/liuueeuuu
21824. PMLR, 23-29 Jul 2023. URL: ptttps://proceediee html
[26] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang. Wei Liu, and Kwang- -ing Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Computer Vision - ECCV 2018: I 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV. page 747-763, Berlin, Heidelberg. doi: 10. 1000000077000 030--3-30--0126-0-0 01267-0-47
2018. Springer-Verlag.
[27] Zhuanghua Liu, Cheng Chen, Luo Luo, and Bryan Kian Hsiang Low. Zeroth-order methods for constrained nonconvex nonsmooth stochastic optimization. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Reseurch, pages 30842-30872. PMLR, 21-27 Jul 2024.
URL: URL::htps:: http::ttt//////t /ttp:eee///rooceiingg. mlr. wwwps///3///iei....e 13u2/112//1/1 .ttml
[28] Shuming Ma, Hongyu Wang. Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang. Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in 1.58 bits, 2024, URL: htttts:ttt:ariiiii http//axww.ooggab////a/aaaa/ 2402. 17764
arXiv: 2402 17764
[29] Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL: https://opperreveex wewtee/ormmidd-oooniaaaaa
[30] Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedeley, Konstantin Pavlovich Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtárik. PV-tuning: Beyond straight-through estimation for extreme LLM compression. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL: https://opeaneviev.ntttt YvA8UF0137)
id=
[31] Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc.
[32] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Gareia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018. URL: wtt.:e//ooorm..trrrgsssee.eeeee
13

---

[33] Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming oscillations in quantization-awe aware training. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, URL: https: /ttph:t//tpsssseeee pwwpee/i//oceeeeegs..oooo alr.press. pages 1631 18 8-16330. PMLR, 17-23 Jul 2022.
v162/nage122a, html
[34] Yurii Nesterov and Vladimir Spokoiny, Random gradient- free minimization convex functions. Found. Comput. Math., 17(2):527-566, April 2017. doi :0: 100.000000////000000000 9296-2
[35] Andrei Panferov, Jiale Chen, Soroush Tabesh, Roberto L. Castro, Mahdi Nikdan, and Dan Alistarh. QuEST: Training accurate LLMs over highly-compressed weights and activation. In Sparsity in I.LMs (SLLM)): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference, 2025. URL: https://openreva
net/forun?id-Elggzzye
[36] Zehua Pei. Xufeng Yao, Wenqian Zhao, and Bei Yu. Quantization via distillation and contrastive learning. IEEE Transactions on Neural Networks and Leurning Systems, 35(12):11144-1777 2024. doi : 10. 1119//19TNNS.S. 3300039
[37] PyTorch. Pytorch example training recipe for resnets. URL: https: //www http://seeeeettttttcc.oooooorr/e
hhwads///////////eaaa/aaaaaaaaas /acaa///maii/////e wal.mmaiiaammeee www.aaagggggaaiiia py
[38] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2247-2256, 2020.
doi 10. 1109/CVPR42600. 2020 00232
[39] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-60 67, 2020. URL:
http://jmlr org/ htts/papers/v// 20220020222 074.hmll
[40] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by October 1986. URL: http:://x.dd/ doi
pack-propggating errors. Nature, 323(6085):533-536
10t//00.133/// 103/3333333///23330 323533a0
doi :0. /033//3333333
[41] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision, 211-3)2 December 2015. doi: 10. 1007 1t1.110//1223-00--001111111
[42] Matt Schoenbauer, Daniele Moro, Lukasz Lew, and Andrew Howard, Custom gradient estima- tors are straight- estimators in disguise, 2024. URL: 05171
2rX15:2405... 05171
[43] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quan- tization for large language models. In The Twelfth International Conference Learning Representations, 2024. URL: https: http::://pppevvveeeeeee net/ fdoorrrmtd-Suuuthhoh.
[44] Juncheol Shin, Junhyuk So, Sein Park, Seungyeop Kang, Sungjoo Yoo, and Funhyeok Park. Nipq: Noise proxy-based integrated pseudo-quantization In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3852-3861 2023. doi: 001:10.1109
CVPR52729. 2023. 00375
[45] James C. Spall. one-measurement form simultancous perturbatioon Automatica, 33(1):109-112 1997. URL: http::::/wwwww.cccncccecc
pertuuaatiouusptttuuurttttononno stochassic atppaoooocccccccccccpppppcocoooo
com/ www//cciecee// doi 10w./10011000050 (1001//80000-1000880)00000110010 1098
[46] J.C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 3733):33,-341, 1992. doi :oi:10 :00 1109/9. 119632
14

---

[47] J.C. Spall. Implementation of the simultaneous perturbation algorithm for stochastic opti- mization. IEEE Transactions on Aerospace and Electronic Systems, 34(3):817-823, 1998.
: 1109/7 1109/7770589
[48] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Marina Mcila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR, 18-24 Jul 2021. URL: https://proceeddng: mlr pwrss/vee3/
touvron2la.html
[49] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Ro- driguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and effi- cient foundation language models, 2023. URL:
13971
230.1:22002.33771
[50] Zhijun Tu, Xinghao Chen, Pengju Ren, and Yunhe Wang. Adabin: Improving binary neural net- works with adaptive binary sets. Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XI, page 379-395 Berlin, Heidelberg,
2022. Beringer-Verlag.
doi: 10. 00.0007777--0 0-1 20083-0033-00 0_23
[51] Haocheng Xi, ChangHao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit integers. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL:
https://openreviers httpp/////pareeeeeeeeeeoorrrrrae wttp://ooooooddddd/ddiii...
[52] Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley J. Osher, Yingyong Qi, and Jack Xin. Understanding through estimator in training activation quantized neural nets. In International Conference on Learning Representations, 2019. URL: https::/ net//forun'.d-SkhhhRRRkk
[53] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: memory-efficient 11m training by gradient low-rank projection. In Proceedings of the 41st International Conference on Machine Learning, ICML' JMI.R.org. 2024.
[54] Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong. and Zheng Zhang. Quzo: Quantized zrder fine fine-tuning for large language models, 2025. URL: /https:// htpps://arxxs ///arivv...v/////a org aba 12346 12346
arXiv 2502
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made the abstract and introduction are
FOGZO converges better solution than the first-order methods like the Straight- Through Estimator, which is experimentally confirmed on a 2-layer MI.,, 2 Resnets, 2 DeiTs, and a series of LLaMA models with parameter counts up to 300 million.
FOGZO requires orders of magnitude less computation than zeroth-order methods (to outperform first-order methods), which is experimentally verified 2-layer MLP. Due to resource constraint, we are unable to verify this in larger networks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses the limitations of FOGZO.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
The claim that FOGZO can suppress the flaws the STE is justified with a heuristic derivation in Section 3.2 whose approximation errors quantified in Section The proofs of the implicit smoothness and perturbation of existing STEs (identity, hardtanh, tanh, ApproxSign) is provided in Section
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The FOGZO algorithm is presented in Algorithm 1 How to properly select the
hyperparameters 3, c, p(u), is discussed in Sections 3.2 and 3.3 The detailed configurations of experiments are discussed in Section44
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided link to our source code and instructions reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ mnips.cc/publiic/
stipoimiiiiioooooooo.ooc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Yes, we provide details of every experiments the main text and the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the mean and two standard deviation training loss and evaluation perplexity/accuracy of 5 experiment runs in Table 2
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the paper provides details on the computer resources needed to reproduce the experiments in the main text and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttts/s//herrrps..c/p/pll/////
Answer: [Yes]
Justification: The research conducted in the paper does not violate any aspect of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
19

---

Justification: There is no societal impact of the work performed. It simply aims to improve the quality quantized models. The models we aim to improve already exist, and we simply hope to reduce their compute resources.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release pretrained models datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators of assets are properly mentioned and cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
20

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provide link to code with proper documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
22

---

A The STE is Occasionally Flawed
In this section we demonstrate, with examples, that the STE can occasionally produce gradients with wrong directions.
Suppose we wish to optimize the objective function h(0) ground where @ E IR and g(v) v/ 4/4. Note (o(0) is non-decreasing function (visualized in Figure 4 as it satisfies the property h(0 e) < h()) for any 0 and any > O. In other words, h can always be made smaller by reducing 0, regardless of what 0 is.
This suggests valid gradient estimators must encourage 0 to reduce regardless of its value, any high-quality gradient estimator must produce a positive gradient.
Suppose we use the identity STE and its estimated gradient:
Ka (0) = g (round(0)) = 3(round( -1/4
(13)
The STE gradient (0) is not always positive. When -0.5 < 0.5, we have (0) < 0. which suggests we should increase B. This means any 9 initialized between -0.5 and 0.5, will gradually increase until reaches or becomes larger than 0.5. When 2 0.5. Ma (0) becomes positive again, and the gradient descent algorithm will decrease 0 until it becomes less than 0.5. This cycle repeats itself, causing 0 to always oscillate around 0.5. In deep neural networks, the problem of parameter oscillation has been shown to slow down convergence when training with the STE 225 33).
However, we emphasize this is an occasional flaw, or an edge case of the STE gradient. Indeed, (0) is almost always positive, which suggest we should almost always reduce 0. Even when the STE gradient makes mistakes, complementary techniques like SGD with momentum can help escape the local minima at -0.5 < 0 < 0.5. However, this is not guarantee, and the effects of momentum depends on hyperparameters like the learning rate. Using the STE gradient requires further tuning of these hyperparameters, which can be undesired.
We next show that this not a flaw of just the identity STE. Suppose we use soft rounding 2 we have the estimated gradient
Moot rounding (0) = g' (round ()) soft_rounding'(0 (0)
(14)
(3 (round 00)² 1/4) soft poff_roouningg(00 (0)
Since
is an increasing function, soft rounding' (0) is always positive. Therefore, rounding (0) is still negative at -0.5 < 0 < 0.5, similar to h (0). Therefore, the same oscillation
behaviour will still happen.
The reason parameter oscillations happen, is because h is fundamentally different from g (visualized in Figure 4. The former is a non-decreasing function, whereas the latter is not. In essence, the additional round function hides the local minima of g, making h non-decreasing function.
gixt 0.25x
hixt giroundix))
10-
h_steldl ait. h hustei g(round(xi)
0.5
-1.0
-1.00
-0.75
0..00
0.25
0.00
0.25
0.50
0.75
100
Figure 4: Example of the occasional flaw of the STE. g(x) x², h((x) g(round (z)), and
We (I) = 3z)) 1002-1144 1/4.
23

---

Is our example objective function h just special counterexample, or does the same behaviour happen in deeper networks, where both g and h can be more complex? The answer is likely yes. Quantization tends to hide local details, whereas the STE uses local details as proxy to global behaviour.
B Approximation Errors of the Heuristic Derivation
In Equation we replace the non-differentiable É with a differentiable approximation Lsmothh, which enables second-order Taylor Expansion that provides some insight into the expected behaviour of the FOGZO gradient G. In this section, quantify the approximation error of replacing i. with smooooth and the approximation error of using the first-order Taylor Expansion (i.e. dropping the second-order term, third-order term, etc.)
We note Lxmoooth has its own smoothness and perturbation distribution p(u), not to be confused with the FOGZO-specifee smoothness and perturbation distribution p(u) in Equation 7 For clarification, will refer to the smoothness and perturbation of Lmmoooh and p(u). We will choose p(u) to be uniformly distributed on the d- d-dimensional unit sphere. We assume L be (C1, C2) coarse Lipschitz, or that (xx) - L(y)l s Gillæ - yll C2.
The first approximation error [i.(0) smoothh (0) can be bounded as follows.
(0) - i smoothh (0) = [() L(e + éx)l
< (0) I( + Eu)l]
(15)
S C|éEi| + C2
= C. + C2
For the second symbol, the approximation error can be quantified with a big-O bound. Since
T (0 + ev:) = Tssmmoahh (0) + (ewx) ) smmeoth (0) + (ev:) ( smmooth (0) O(| (16)
2
and
Lamooti (f - -00) smmoth (0) - (ev;) TV Lmmoooth (f) + (ev;) (0n=n (17)
Subtracting the two, we have
Lemooohh (0 eu:) Lumooth (0 (eee ((ee:)"VVLann (0) ((lee;:1)))
(18)
Note that all even-order terms cancel out due to the symmetric nature of the central finite difference, leaving only the third-order term which is dominating for sufficiently small and Hessian-Lipschitz L'smooth
Next, we divide by 2e and multiply by vs and get
Lmmooth (0 + ev;) Lssmosah ev;) = "uea V L (0) + 0(22|ou:1))
(19)
2
where the term ((E21lu:)))) e; ; is the approximation error of the second ss symbol.
C Random Sign Flip
As discussed in Section 3.1 FOGZO introduces random variable s; that ensures v; /Bs, - -Bu; is symmetric. Since the central finite difference method already introduces symmetry, the second-order term cancels out discussed Section EB] Therefore, s; not necessary in theory. However, in practice, we find using s; can improve the performance slightly. For example, when we train Resnet-50 on Imagenet-100, we notice that FOGZO (with s;) achieves a training loss of 0.3872 (averaged over 5 runs), whereas variant of FOGZO without achieves training loss of 0.3883 (averaged over 5 runs). We attribute this to floating point error breaking the symmetry of the central finite difference method. Therefore, we include in FOGZO for its empirical performance gain.
24

---

D Operation- Wise Jacobian Estimators
In this section, we prove several rounding/sign gradient estimators, also known as STES, are express- ible as randomized smoothing variants of round/sign. We denote round as R and sign as S. A smooth approximation of some function h with smoothness and perturbation p(u) is denoted as h(,) In the main text, p(p) is simply referred to as hmmothh
D.1 The Identity Function
Function
Gradient
p(u), E=1
1.0
R(x)
VR(x)
2
((x)
Vi()
1.0
0.6
U(-0.5, 0.5)
0.5
0.4
-?
0.2
0.0
0.0
-2
-2
2
2
Figure 5: The identity STE uses the identity function to approximate rounding.
The identity STE (3) uses the identity function (xx z, visualized in Figure
Statement to prove: 1(x) = (R.p(u)())) where p(u) = U ((-0.5/e,,55//e
-ustit-05//.05/// [R(x eu)]
0.5//
R(x + cu)p(u)du
(20)
0.3/e
0.5/e
R(x + cu)edu
0.3/e
We substitute 2 eu, == (z - x)/c, and du dz/c
+0.5
R(z)dz
[a]+0.5
+055
R(z)dz +
R(z)dz
22550055 0.5
[a]+0.5
0.55
[ [r)dz +
[x] dz
Jz-a-02
+x05
(21)
[[ + 0.5 (a 0.5)) [
(or
0.5 ([x) 0.5))
[x][[=] - I + 1)) + [x:
[
[ x))
[x]] + [
(- [=]) [x])) + [x]] (x:
[x]]
[c]
[[x=] [x]] [x]] + [
(xr [x]] [x]
To ensure p(u) has unit variance, we can use 0.5/53 and p(n) = U(-V3.V3
25

---

D.2
Hardtanh
Function
Gradient
p(u). E=1
1.0
VS(x)
Vhardtanh(x)
0.4
0.3
0.0
U(-1, 1]
hardtanh(x)
-0.5
0.1
-1.0
0.0
-2
-2
2
Figure 6: The Hardtanh function approximates the sign function.
The Hardtanh STE 38 uses the hardtanh function defined as follows:
1,
<
hardtanh(x))
1, i 1,
(22)
otherwise
Statement prove: hardtanh(z) (x)), where p(u) U/-1/,,
E/a00(-1/c101996 eu)]
f S(x cu)p(u)du
(23)
-
pl/e
S(x cu)du
2 J
We substitute 2 = cu, u = 2/c, and du dz/c.
S=s00((11c1/00000-- 1
S(x + z)dz
(24)
==-1
We first consider the case 1
(=11((1/611/(1))
1
/ S(r + uddu
2
1
{ S(x u)du + 2/ S(x + u)du
1
_
2
1 (')
Idu +
1du
2 2 -1
-1
(25)
((-x) - (-1)) (1+
(11))+((1+)))
1
1
(a
1) + 2 (1+
2
(r 1+1+x)
2
2 (2x)
26

---

Then we consider the case < -1.
-==100(-1/////(000
1 1
2 _ S(x + u)du
/ 1du
(26)
2 (1-(-1))
2
Finally, consider the case >> 1.
1 /
S(x + u)du
2 _
1du
2 -1
(27)
(1 (-1))
2
1
2
2
10
To ensure p(u) has variance, we can use E 1/v and p(u) = (3(-3,,v v3).
D.3
Tanh-Based Functions
Function
Gradient
p(u),
1.5
1.0
0.5
5(x)
V5(x)
0.5
tanh(x)
1.0
Vtanh(x)
0.4
0.3
0.0
0.5 tanh (
0.5
-0.5
0.1
-1.0
0.0
0.0
-2
-2
Figure 7: The tanh function approximates the sign function.
The tanh STE (38) uses the tanh function, which corresponds smoothing the sign function via perturbation distribution with PDF p(u) = =1( tanh²(eu)) and CDF P(u) 2 + tann((eu). This distribution is equivalent to logistic distribution with mean and scale
27

---

Statement to prove: tanh(æ) (e)(())()),, where p(u) 2(1 tanh (eu)).
S (z.))(
Buwp(u) +
S(x eu)p(udu
z/c
S(x + eu)p(u)du Sr, S(x + eu)p(u)du
z/e
-p(u)du + L a
plu)du
-//E
- c1 - (cu))
c11 - (anh"((c))) + 2 wx/e
(28)
2
[tanh(eu)] [tanh(eu]] -x/c
2
[tanh( -x) ttnhh- (-oc)] + [tanh(co) tanh(- x)
tanh (xx) tanh( (anh(-oo) taan((c) 2
tanh( (-a)
tanh( (xx)
tanh -x)
2
2
tanh( x)
tanh xz) D
To ensure p(u) has unit variance, we use e VT2 and p(u) = {(1 - tenn)
Soft Rounding 2) and HTGE (36) utilize the fact that:
R(x)
x
0.5 +
S(x - (Lx + 0.5))
(29)
Rather than smoothing R, Soft Rounding and HTGE use tanh to smooth S in Equation 29 As we have proved that tanh randomized smoothing variant of S, omit further discussion of Soft Rounding and HTGE.
D.4
Polynomial- Functions
Function
Gradient
p(u), E=1
1.0
1.0
V5(x)
Uritu)
VApproxSign(xI
0.5
Stxt
0.6
ApproxSignix)
0.4
-0.5
-1.0
0.0
-2
-2
-2
2
Figure 8: The ApproxSign function approximates the sign function.
The ApproxSign STE (26) uses the ApproxSign function, defined as:
-, if -1
2r + x2 2r - x2 1
if =
ApproxSign(z))
(30)
if
otherwise
28

---

Its derivative is defined as:
22 2x.
if -
AppprxxSign(z) = 2 2x,
if S x otherwise
(31)
0,
Statement to prove: ApproxSign(z) Se,((u) (2r), where p(u) = e tri(eu). The tri function is defined as:
tri(u) = [1 - 1|//,,
if 4 S 1
(32)
10,
otherwise
We know that
(e.)))(())) (z)
Euvee + ew]]
(33)
Jum-1/c S(x + euje tri(eu)du
We substitute eu,, z/c, and du dz/e.
S,, = { ==-1
S(x 2)
(34)
We now consider four cases:
x: 2 1:
The integration bound implies that x 2 -1; therefore, r + 2 0 and S(z) = 1.
Se.e (x) = [" S(x z)tri(z)dz / tri(z)
(35)
T < 11
The integration bound implies that = S 1; therefore,
< 0 and S(z) = -1
S ((ee)((e)) / S(x + =)rr(=)d2 /" tri(()dd
(36)
-1 0: S ((
S(x + =)tri(z)dz
S(x + 2)tri())dz+ + f0² S(x+ 2)tri()dd [ S(x + 2)tri(2)dz
-tri2)dz + -tri(z)dz+ + /' tri(z)dz
-0.5 +
-(1
2)d2 +
(1 2)
Jo
-2
1
-
1
-0.5
+
2
2
2
-m
-0.5
(x +
+
0.5
(-x
2r + 2
(37)
29

---

. 05z<l: S (())))()((2)))
S(r + 2)tri
S(x 2)tri(z)dd [ S(x + 2)rr(2)dz [ S(x 2)tr(z))dz
tri(2)d2 + { tri(z)dz + / tri(z)dz
-(1 +2222 +
(1+ +122)d2 + 0.5
-1
1 -
1
2
2
2
0.5
2
2
-
(r
0.5 + ---x +
2" =²² + 0.5
2x
(38)
To ensure puu) has unit variance, we can use following:
Jo and p(u) = 76
tri( tri()a). or equivalently the
if |u| S v6 otherwise
p(u)
v8 0,
(39)
D.5
Gonfidence-Guiieed Masking (CGM)
Function
Gradient
p(u), E=T
R(x)
vS(x)
T=0.5
T=0.5
0.4
T=0.25
T=0.25
T=0.125
T-0.125
0.3
U(-1,
0.2
-2
0.0
-2
2
Figure 9: Confidence-Guider Masking approximates rounding.
Confidence Guided Masking 25 uses a hyperparameter confidence threshold 0 < T 0.5, and is defined as follows:
VCGM(x) {
if R(x)) 0.5
(40)
otherwise
This is equivalent to using the following gradient, but with the learning rate scaled by a factor of 27.
(0
if [ R(x)) 0.5 - T,
VCGMccaled (x)
(41)
otherwise
We prove that CGM is equivalent to a smoothed variant of R.
30

---

Statement to prove: VCGM (2) = VR, V.(.,().))))))..
(=IT(((-1)((1)
Eu R((x Tu)
S R(æ + Tu)ppuudu
(42)
[
R(x Tujdu
R(z)dz
2T T=n-T
We first consider the case x +T < |x]] + 0.5.
This means R(x 2) R(x) R(x T) [
and
[ø]
R(x) < 0.5 - T.
which the positive half of z R(x)) < <.5 T.
(==T,U(-1,))())))
1
R(z)dz
2T
I=--T
N+T
[r] dz
2T
==x-T
(43)
1
([x])((x + +T) (a- (x-T))
2T
1
(2x))
2T
[x]
Thus, V (Be)), 0 the floor function is flat when + T < [x] 0.5.
Next, we consider the a T 0.5.
This means R(x - T) R(x) RT)+T)= = [x] and x [æ]] 0.5 T, or equivalently
x R(x) > T 0.5, which is the negative half of (a R(x)] < 0.5 - T.
R (=-TV(-L,))))))
1
T+T
R(z)dz
2T
T=z-T
1
aT+
[z]dz
27
I=z-T
(44)
([x])((x + +T) - (x - T))
2T
1
([x])(27)
27
[x]
Thus (R=-T.U(LLI)(()) = 0, as the ceiling function flat when x T>> [x] + 0.5.
Finally we consider the case x T < x 0.5 T.
This is exactly when - - R(x) < 0.5 - T is false.
31

---

Re=T.U(-11)(())
r+T
R(z)dz
2T
==--T
[2,5
1 2T
[x]dz +
|x|d=
2T
2=0-T
2T +0,5
([=])[[[=] 0.5) (x ( + 27 (1=)((<++ - ([æ] + 0.5))
2T
- + 27
0.5
27
1
([x))(2T)
+
(x [x| - 0.5 + 7))
27
27
1
|x| +
(x
||| 0.5 TT)
2T
(45)
Thus RR=T.U(LLLI)()
(x) = 27
To ensure unit-variance perturbation, we can use
and p(u) = U(- v3).
v3
E Experiment Configurations
In this section, we discuss the configurations of our experiments.
E.1 Shallow Networks: Comparing the STE, n-SPSA, and FOGZO
We use batch size of 512 and a learning rate of 2e-3 batch_size 32. We train for 10 epochs on MNIST. We use the AdamW W optimizer and the Cosine Annealing scheduler. We quantize only the weights and use precision of 2 bits. For the quantization scale shared among the 2 linear layers, we follow the initialization technique LSQ (13 and set ar = 2- absmean(0), (0;)/ //Q////Qp where a; is the quantization scale of layer i, and 0, is the weights for layer We then perform a weighted average = Σ a; nu..n)))l)) where allows us to obtain a quantization scale a that is acceptable to both linear layers. Before training, we compute a once and never modify it during training.
Σ 0,.numll(), and / is the number of linear layers. This
In terms of weight quantizer design, we mostly follow LSQ, except that a is fixed to reduce potential factors that prevent us from clearly identifying the strengths and weaknesses of the STE, SPSPSA and FOGZO.
① a round(clip((//a, QN QP)) where QN -22-1 and Qp 2p-1
(46)
However, n-SPSA and FOGZO with large sample size n are too expensive to calculate efficiently, even for a simple 2-layer MLP. As such, we use a parallelized implementation that performs forward passes in parallel. We implement this by duplicating the model weights n times and perturbing each copy with different sampled perturbation u, v;. During each forward pass, all n copies calculated parallel to reduce computation time.
For the FL.OPs estimation in Figure I we count the FLOOPs of linear layers using the formula where, given an activation matrix with shape (b, c), and weight matrix with shape (c, 0), the forward pass FLOPs is 2boc and the backward pass is 4boc.
Without quantization, the 2-layer MLP can reach training loss of 0.23. With 2-bit weight quantization and using the Straight-Through Estimator, the training loss increases to 2.02. Using FOGZO with 0.999 and n 1, the training loss improves to 1.97.
E.2 Deeper Networks: Comparing the STE and FOGZO
In this section, we discuss the configurations of experiments on deeper networks. All models with fewer than 30 million parameters are trained on Nvidia RTX 2080 Ti GPUs with 11 GB of memory.
32

---

All models with parameter counts between 30 million and 200 million are trained on Nvidia RTX 5090 with 32 GB of RAM. All models with more than 200 million parameters are trained on Nvidia A 80GB. We follow the training recipes and hyperparameters of (18), (48), (53) (35) We make no changes to the training recipes with few exceptions for simplicity: we do not use EMA evaluation; we do not use gradient accumulation and simply follow the linear scaling rule (16) to scale batch
size and learning rate; we keep BatchNorm in training mode but freeze the running stats when we measure the loss of the perturbed model.
E.2.1 FOGZO can Improve a Variety of STES
We adapt the training recipe from (8) for Resnets using the source code from (37), the training recipe from (88) for DeiTs, and the training recipe from (53) for LLaMAs. We train Resnet models with single RTX 2080 Ti, DeiT models with four RTX 2080 Tis, and LLaMA models with one RTX 2080 Ti. If RTX 2080 Ti cannot fit the batch size specified in the original training recipe, iteratively reduce the batch size by 2x and learning rate by 2x until we no longer run out of memory. Similar non-learnable quantization scale shared by all linear layers in model,
to Section E.1 calculated with the weighted average of all LSQ quantization scale initialization. Since LSQ does support 1-bit quantization, instead adapt the quantization scale calculation from (28). which uses a; = absmean(0:). if the target STE (tanh and ApproxSign) is designed for 1-bit quantization. For Resnets, we additionally implement zeroth-order-safe BatchNorm, which simply disables the update of running statistics during zeroth-order operations in Algorithm
33


Reviews:
[{"review_id": "oqSSn2R5U5", "reviewer": "Reviewer_tcg3", "summary": "The straight through estimator (STE) is a commonly used work-around to circumvent the non-differentiability of rounding operators, and is frequently used in quantization aware training of neural networks. This paper proposes blending the STE with zero-order gradient estimation, essentially by taking their weighted sum as the gradient estimate. In particular, the method relies on perturbing the current parameter estimate in a random direction in order to obtain the zero-th order estimate. The proposed method is given a heuristic derivation,  as are the probability distributions associated with different straight through estimators,  and the method is numerically tested by training neural networks of various sizes and architectures.", "questions": "- Section 2.2: $u$ is a vector, so when you say it is unit variance, do you mean it is isotropic, i.e., having an identity as its covariance matrix? \n- Eq (4) looks a little different from the definitions used in the n-SPSA papers [45, 46]. Is the difference cosmetic or substantial? \n- \"While some believe the STE is completely unjustifiable\" -- who? what are there arguments? \n- Eq (5) the multiplication by $s_i$ seems to suggest that you have a 50/50 chance of biasing your update in the negative direction to that of $\\hat{g}$. Is there an explanation for why this is useful (beyond the need for making $v_i$ 0-mean)?\n- Section 3.2: I am hesitant to call this section \"analysis\" given that it relies on the heuristic approximations (indicated by the $\\approx$ symbol in equation (7). Perhaps, \"heuristic derivation\", or \"heuristic motivation\" are better? \n- Eq (8): third = sign, there is no need for the expectation here as all sources of randomness have been taken care of. \n- Line 187, \"are set\" instead of \"is set\"? (similar subject-verb agreement issues elsewhere, like line 195). \n- Section 3.3: This section is a bit awkwardly written. One needs to read all the way to the equations to understand what is meant by (implicit) smoothness and perturbation of an STE. \n- Eq (9): is it assumed (without loss of generality) that $x>0$. \n- hardtanh should be defined as it's definition is used in the derivation.\n- Figure 1: I found this figure confusing to read. Why use the reciprocals of the number of flops and tranining loss? Also, I would have found an explicit reporting of $n$ helpful. \n- Figure 1: How is the gradient norm computed? for a fixed iteration, over all the iterations? \n- What was the range of n used in section 4.1? \n- I appreciated the discussion of activation quantization, even if it was a stumbling block for the method.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The paper is generally well written and generally well motivated. \n- The arguments for the proposed algorithm were convincing (albeit heuristic). \n\nWeaknesses: \n- The theoretical components of the paper are rather heuristic. In other words there are lots of $\\approx$ symbols hiding approximation errors that are never quantified. \n- In some places, the paper suffers from awkward writing.\n- The plots were a little hard to read due to some unusual presentation choices.\n- The appendix is written a bit too casually.", "was_revised": true, "final_justification": "The authors answered most of my questions, the most important of which pertained to the \"analysis\", and they stated that they made the important edits I asked for in my review.", "rebuttal": "We thank Reviewer tcg3 for reviewing our paper. We have updated a local version of our paper following your advice. Since NeurIPS does not allow us to submit an updated version during the rebuttal, we promise to keep those changes in the camera-ready version and avoid introducing similar mistakes.\n\n- Section 2.2: u is a vector, so when you say it is unit variance, do you mean it is isotropic, i.e., having an identity as its covariance matrix?\n   - Yes, we meant the covariance matrix of u, or $E[uu^T]$, is an identity matrix. We use this property in Equation (8) to show the unbiased-ness of the unbiased component.\n\n- Eq (4) looks a little different from the definitions used in the n-SPSA papers [45, 46]. Is the difference cosmetic or substantial?\n   - SPSA was invented by James C. Spall in the late 80s. Spall is the author of both [45] and [46]. SPSA has many variants such as one-measurement, two-measurement (central/forward/backward difference), and specific choices of the unbiased noise. The formulas shown in [45] and [46] are variants of SPSA. The SPSA formula we used in FOGZO is taken from [34]. Note that paper [34], when referring to SPSA, also cites a paper authored by Spall. We chose to additionally cite [45] and [46] to pay respect to Spall.\n\n- \"While some believe the STE is completely unjustifiable\" -- who? what are there arguments?\n   - By “completely unjustifiable” we mean, more precisely, that prior authors [33], [44] and [35] have argued that STE is suboptimal and unstable. Quoting [33], “STE leads to instability” and “STE … is known to be quite noisy especially for extreme quantization”. Quoting [44], “STE incurs unstable convergence during QAT, resulting in notable quality degradation in low precision”. Quoting [35], “When using the popular straight-through estimator (STE) for QAT, weights seemingly randomly oscillate between adjacent quantization levels leading to detrimental noise during the optimization process.” In essence, these papers suggest the STE is suboptimal due to its noisy behaviour when training in low precision. \n\n- Section 3.2: I am hesitant to call this section \"analysis\" given that it relies on the heuristic approximations (indicated by the ≈ symbol in equation (7). Perhaps, \"heuristic derivation\", or \"heuristic motivation\" are better?\n- Eq (5) the multiplication by si seems to suggest that you have a 50/50 chance of biasing your update in the negative direction to that of g^. Is there an explanation for why this is useful (beyond the need for making vi 0-mean)?\n\n   - We have updated the section header to “Heuristic Derivation” and removed all instances of the word “analysis” in the paper. Below we try to quantify the approximation error of the two ≈ symbols in equation (7) under certain conditions, and also explain why $s_i$ is useful.\n\n   - The first ≈ symbol in Equation (7) would be **exact** equality if we take the expectation of both the LHS and the RHS and $v_i$ is unbiased noise, as per the paper “Randomized Smoothing for Stochastic Optimization, Duchi et. al. 2012”. However, in the case of FOGZO, $v_i$ is not unbiased but depends on $g$, which depends on the chosen STE, the network architecture $L$, and $\\theta$, the model’s current position in the loss landscape. Therefore, we find it challenging to quantify the approximation error in the general case, especially when the STE introduces unpredictable biases that are dependent on the network architecture. As such, we rely on the fact that $\\hat{L}$smooth is an approximation to L for the first ≈ symbol. This is a crucial step in the derivation as $\\hat{L}$ is not differentiable and not even Lipschitz (due to vertical jumps at the rounding boundaries), and Taylor Expansion cannot be used. By changing $\\hat{L}$ to $\\hat{L}_{\\text{smooth}}$, we can unlock Taylor Expansion which provides us some insight on FOGZO. \n     - we note that in a special case, when $\\epsilon$ approaches 0, the approximation error vanishes because $L_{smooth}$ approaches $L$. This is possible, for example, when FOGZO is paired with an STE with decaying implicit smoothness. For instance, in [39], a variant of the tanh-STE is used with its implicit smoothness gradually decayed during the process of training. Since FOGZO uses the implicit smoothness of the chosen STE, $\\epsilon$ would also gradually decay if FOGZO was paired with such STEs. In such cases, the approximation error would vanish at the very end of training.\n\n   - For the second ≈ symbol in Equation (7), the approximation error can be quantified with a big-O bound. \n\n     - Since $\\hat{L}_{smooth}(\\theta + \\epsilon v_i)=$ \n\n       - $\\hat{L}_{smooth}(\\theta)$\n       - $+ (\\epsilon v_i)^T \\nabla \\hat{L}_{smooth}(\\theta)$\n       - $ + (\\epsilon v_i)^T  {\\nabla}^2 \\hat{L}_{smooth}(\\theta) (\\epsilon v_i) / 2$\n       - $ + O(||\\epsilon v_i||^3)$\n\n     - and that $\\hat{L}_{smooth}(\\theta - \\epsilon v_i)=$ \n       - $\\hat{L}_{smooth}(\\theta)$\n       - $ - (\\epsilon v_i)^T \\nabla \\hat{L}_{smooth}(\\theta)$\n       - $ + (\\epsilon v_i)^T  {\\nabla}^2 \\hat{L}_{smooth}(\\theta) (\\epsilon v_i) / 2$\n       - $ + O(||\\epsilon v_i||^3)$\n     - Subtracting the two we have \n       - $(2\\epsilon v_i)^T \\nabla \\hat{L}_{smooth}(\\theta)+O(||\\epsilon v_i||^3)$\n     - Next we divide everything by $2\\epsilon$ and multiply by $v_i$, and get\n         - $v_i v_i^T \\nabla \\hat{L}_{smooth}(\\theta)+O(\\epsilon^2||v_i||^3)v_i$\n         - the big-O term is the approximation error.\n\n     - Additionally, If $v$ is zero-mean and symmetric, the term $O(\\epsilon^2||v_i||^3)v_i$ should vanish on expectation, giving us \n       - $E[v v^T \\nabla \\hat{L}_{smooth}(\\theta)]$ which is the LHS of Equation (8)\n       - This is why it is desired to obtain a zero-mean (and symmetric) $v_i$, and why $s_i$ is useful. It allows us to remove the approximation error (the big-O term).\n       - We use the word \"should\" here because in practice $\\hat{L}_{smooth}(\\theta)$ might not be Hessian-Lipschitz and $\\epsilon v$ might not approach zero, both preventing us from writing the big-O notation in the first place. Therefore, we additionally use empirical observations to confirm that we should use $s_i$ in FOGZO. Specifically, when we train Resnet-50 on Imagenet-100, we notice that FOGZO (with $s_i$) achieves a training loss of 0.3872 (averaged over 5 runs) whereas a variant of FOGZO without $s_i$ achieves a training loss of 0.3883.\n\n- Line 187, \"are set\" instead of \"is set\"? (similar subject-verb agreement issues elsewhere, like line 195).\n- Section 3.3: This section is a bit awkwardly written. One needs to read all the way to the equations to understand what is meant by (implicit) smoothness and perturbation of an STE.\n   - We have updated the paper to correct for grammatical errors, remove redundant sentences/paragraphs, and introduce the equations earlier in the section. \n\n- Eq (9): is it assumed (without loss of generality) that x>0.\n   - Since the integration is with respect to the perturbation noise $u$, $x$ is simply treated as a constant throughout Equation (9). We do not make any assumptions on the sign of $x$.\n\n- hardtanh should be defined as it's definition is used in the derivation.\n   - We have updated the paper. Hardtanh$(x)=clip(x, -1, 1)$\n\n- Figure 1: I found this figure confusing to read. Why use the reciprocals of the number of flops and training loss? Also, I would have found an explicit reporting of n helpful.\n- What was the range of n used in section 4.1?\n   - We sincerely apologize for the inconvenience. We thought reviewers might prefer a graph where the top-right corner is the best position. We now realized this was clearly a bad choice and we will remove the reciprocals in the camera-ready, making the lower-left corner the best position. In Figure 1, we used $n \\in$ {1, 4, 10, 100, 1000, 7690\\} where 7690 is the number of parameters in the 2-layer MLP model we used for the experiment.\n\n- Figure 1: How is the gradient norm computed? for a fixed iteration, over all the iterations?\n   - The gradient norm is the average gradient norm computed over the first iteration. This is because the first iteration gradient norm is often the noisiest, and has the largest impact over training (deciding whether the model will diverge).\n\n- I appreciated the discussion of activation quantization, even if it was a stumbling block for the method.\n\n   - Although we initially planned to leave activation quantization for future work, all reviewers list activation quantization as a weakness/question. Upon further investigation, we present new evidence that FOGZO can be applied to SOTA Quantization-Aware Training methods with **both** weight and activation quantization. In addition to LSQ [14], we additionally test QuEST [36], a new SOTA QAT method published (in June 2025) after we submit FOGZO (in May 2025). QuEST introduces a hadamard transform before quantization, and can significantly outperform LSQ. We add FOGZO to the publicly available source code of QuEST following Algorithm (1). We compare QuEST against QuEST-FOGZO, and LSQ against LSQ-FOGZO, and report the evaluation perplexity. Note that the results below are training results of models with both weights and activations quantized to 2-bit, and we use a $\\beta_{min}$ of 1 - 1e-10.\n\n   - We use the benchmarks in the QuEST source code: LLaMA-30m and LLaMA-50m. Note that LLaMA-30m has 95 million parameters and LLaMA-50m has 125 million parameters, which is 5x to 6x larger than the largest model we have used in the submitted paper. Following QuEST’s advice, we train llama-30m with 3 billion tokens, and llama-50m with 5 billion tokens. \n\n|Model|QuEST|QuEST-FOGZO|LSQ|LSQ-FOGZO|\n|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|\n|LLaMA-30m|37.75|**37.37**|39.06|**37.38**|\n|LLaMA-50m|32.28|**32.06**|33.28|**32.41**|\n\n   - Results show that FOGZO can improve both QuEST and LSQ. In addition, FOGZO can even shrink the perplexity difference between these two SOTA methods."}, {"review_id": "aC7EumeYMV", "reviewer": "Reviewer_j9sv", "summary": "The paper proposes FOGZO, a novel method that combines the Straight-Through Estimator (STE) and Zeroth-Order (ZO) gradient descent for training quantized neural networks. While STE is efficient but biased, and ZO methods are unbiased but computationally expensive, FOGZO leverages STE as a bias source to guide low-cost ZO updates, suppressing STE’s flaws. The method introduces a mixing parameter to balance biased and unbiased gradients. Experiments show FOGZO improves training accuracy and efficiency across various models, including ResNet, DeiT, and LLaMA, outperforming both STE and ZO baselines.", "questions": "1. Figure 1 is hard to understand. Why are reciprocals of loss and flops used? They make readers hard to clearly and easily figure out which line is better. \n2. The chosen $\\beta$ value is very close to 1, implying that the STE part dominates the ZO part. It is better to call it ZO Guided STE? In addition, as $\\beta$ is very close to 1 and $1-\\beta$ is very close to 0. Does it cause any numerical issues during training, especially in FP16/BF16 training?", "limitations": "NA", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Introduces FOGZO, a first-order-guided zeroth-order gradient estimator that leverages the biased yet efficient STE to guide the unbiased zeroth-order optimization.\n2. Provides theoretical analysis showing that FOGZO can suppress incorrect STE gradients while preserving computational efficiency.\n3. Performed numerical experiments to demonstrate the theoretical findings. \n\nWeakness: \n1. Increased the training costs and training time. \n2. More hyperparameters are introduced, which may require more careful tuning. \n3. Experiments are primarily conducted on small-scale models. Gains with SOTA QAT methods, like LSQ, are marginal for most models except DeiT-Tiny.", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer j9sv for reviewing our paper.\n\n## Small Scale Models and Marginal Gains with SOTA QAT Methods\n\nWe provide new evidence that FOGZO can work well with larger models: LLaMA-30m and LLaMA-50m. Note that LLaMA-30m has 95 million parameters and LLaMA-50m has 125 million parameters, which is 5x to 6x larger than the largest model (~20 million) we have used in the submitted paper. \n\nIn addition to LSQ [14], we additionally test QuEST [36], a new SOTA QAT method published (in June 2025) after we submit FOGZO (in May 2025). QuEST introduces a Hadamard transform before quantization, and can significantly outperform LSQ. \nWe add FOGZO to the publicly available source code of QuEST following Algorithm (1).  Following QuEST’s advice, we train llama-30m with 3 billion tokens, and llama-50m with 5 billion tokens. We compare QuEST against QuEST-FOGZO, and LSQ against LSQ-FOGZO, and report the evaluation perplexity. Note that the results below are training results of models with both weights and activations quantized to 2-bit, and we use a $\\beta_{min}$ of 1 - 1e-10.\n\n|Model|QuEST|QuEST-FOGZO|LSQ|LSQ-FOGZO|\n|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|\n|LLaMA-30m|37.75|**37.37**|39.06|**37.38**|\n|LLaMA-50m|32.28|**32.06**|33.28|**32.41**|\n\nResults show that FOGZO can improve both QuEST and LSQ. In addition, FOGZO can even shrink the perplexity difference between these two SOTA methods.\n\n## Training Time\nThree reviewers have listed increased training time/computation overhead as a weakness of the method, and we agree that this is a valid concern. However, we note that there are certain use cases where model quality takes precedence over training time. For example, the operator of a large fleet of already deployed self-driving cars may find it more economical to invest extra computation time in training versus needing to upgrade hardware in existing cars if those are the only two options for improving accuracy/performance/safety.  Intellectually, we are interested in the limits of inference accuracy at low precision without regard to training cost and believe this direction has merits in such scenarios. \n\nWhile FOGZO introduces an additional training overhead of 2 forward passes per iteration, we show that the training time overhead can be reduced by using the STE at the beginning of training, and switch to FOGZO at the end of training. Concretely, this means we only execute lines 6-16 in Algorithm 1 if $t \\ge r * T$, where $r$ is how much (70%, 80%, 90%, etc.) of the training is performed in STE mode. In the following table, we use LSQ to train llama-30m with 3 billion C4 tokens. We report evaluation perplexity and training time. While FOGZO alone incurs an overhead of 58% to tradeoff a perplexity decrease of 1.6, a mixing strategy that uses 90% of STE and 10% of FOGZO can still improve over the STE by a perplexity decrease of 0.4 with only a 6% increase in training time. \n| Method              | Evaluation Perplexity | Training Time (hours) |\n|---------------------|-----------------------|-----------------------|\n| STE                 | 39.06                 | 3.1                   |\n| 90% STE + 10% FOGZO | 38.67                 | 3.3                   |\n| 80% STE + 20% FOGZO | 38.38                 | 3.5                   |\n| 70% STE + 30% FOGZO | 37.93                 | 3.7                   |\n| FOGZO               | 37.38                 | 4.9                   |\n\nBelow we extend the training time of the STE by adding more training data. We compare this pure-STE-with-more-data strategy and the previous mix-STE-FOGZO strategy under the same training time. Each row is a separate training run. Results show mix-STE-FOGZO outperforms. This also suggests FOGZO can have higher data efficiency than STE, as the FOGZO runs use less training data to achieve lower perplexity. \n\n| Method              | C4 Tokens (billion) | Evaluation Perplexity | Training Time (hours) |\n|---------------------|---------------------|-----------------------|-----------------------|\n| STE                 | 3.174               | 38.69                 | 3.3                   |\n| 90% STE + 10% FOGZO | 3                   | **38.67**                 | 3.3                   |\n| STE                 | 3.348               | 38.43                 | 3.5                   |\n| 80% STE + 20% FOGZO | 3                   | **38.38**                 | 3.5                   |\n| STE                 | 3.522               | 38.25                 | 3.7                   |\n| 70% STE + 30% FOGZO | 3                   | **37.93**                 | 3.7                   |\n\n\n## Using Reciprocals in Figure 1\nTwo reviewers noted using the reciprocal of training loss and flops makes the figure hard to read. We sincerely apologize for the inconvenience. We thought reviewers might prefer a graph where the top-right corner is the best position. We now realized this was clearly a bad choice and we will remove the reciprocals in the camera-ready, making the lower-left corner the best position. \n\n## Questions\n\n1. The chosen β value is very close to 1, implying that the STE part dominates the ZO part. It is better to call it ZO Guided STE? In addition, as β is very close to 1 and 1−β is very close to 0. Does it cause any numerical issues during training, especially in FP16/BF16 training?\n\nFor clarification, training with the STE uses gradient g, and training with FOGZO and a β value of exactly 1 uses a gradient that is approximately $\\hat{g} \\hat{g}^T \\nabla L_{smooth}$ (as shown in Equation 8). This means even when β is exactly one, FOGZO can still suppress the STE gradient if it is inaccurate, which may or may not be a good thing. If the STE is only occasionally inaccurate, FOGZO can prevent the STE from making mistakes when it is wrong, resulting in improved model quality. However, if the STE is constantly inaccurate, FOGZO will constantly suppress the STE gradient, and training can stop making progress. As shown in Figure 2, when FOGZO is used with a β value of exactly 1, outperforming the STE is not guaranteed (but it can happen). Therefore, we believe that although we use a β value close to 1, FOGZO is still fundamentally a zeroth-order method (as it has an incorrect gradient suppression effect similar to SPSA). However, we agree that “ZO Guided STE” and “STE/FO guided ZO” are very similar to one another, and both are accurate descriptions to some degree.\n\nMost of our results are trained in FP32 precision. However, in the QuEST/LSQ/training time experiments above, we use torch.amp.autocast to perform bfloat16 training. We did not observe any numerical issues."}, {"review_id": "QAL5sKDUvG", "reviewer": "Reviewer_y8vC", "summary": "This paper introduces ​​FOGZO​​, a novel method for QAT that synergistically combines the computational efficiency of the STE  with the unbiased gradients of ZO methods. FOGZO demonstrates significant empirical improvements over STE and ZO baselines across multiple architectures (ResNet, DeiT, LLaMA) and datasets (ImageNet, C4), with minimal computational overhead. The paper is well-written, technically sound, and contribute a new direction for optimizing STE.", "questions": "1. Could perturbing activations (not just weights) during ZO steps resolve the performance gap? \n2. Or can your maintain the STE for activation scale training to see how FOGZO works with weight-activation quantization?\n3. Could you compared your methods with EWGS? This is a paper using second-order information to suppress the STE, maybe both the efficiency and performance.\n4. Does FOGZO’s gradient variance reduction amplify benefits from optimizers like AdamW or Lion?\n5. Could you share the time cost and memory usage of your methods compared to STE?\n6. I saw you added hardmard transform in your llama models, would this method be helpful with hardmard transform or can this improve the performance after transformed?\n7. Could you share the FLOPs for different step-size of ZOs?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths​:\n- The combination of STE and ZO gradients​​ via a biased-unbiased gradient mixing strategy is novel and new for me.\n- The theoretical analysis is detailed, showing the effective of FOGZO.\n- Comprehensive experiments​​ on vision and language models show consistent gains of FOGZO.\n- The paper is well-written and easy to follow.\n\nWeaknesses:\n- The searching for hyperparameter beta is time-cost.\n- Although FOGZO reduces the time cost compared to ZO based method, but it is still time cost, narrowing the use range for large models.\n- It doesn't work good with weight-activation quantization.", "was_revised": true, "final_justification": "This paper presents a new approach to quantization gradient estimation. It looks novel to me as it uses zero-order optimization, which is the first time taking this approach into the gradient estimation for quantization. The experiments show the performance improvement of this method. However, the increase in training time is too significant to make it feasible for a real-world situation. It's still a toy. \n\nCombining the rebuttal of the authors, I would give a borderline accept.", "rebuttal": "We thank Reviewer y8vc for reviewing our paper.\n\n## Activation Quantization\nAlthough we initially planned to leave activation quantization for future work, all reviewers list activation quantization as a weakness/question. Upon further investigation, we present new evidence that FOGZO can be applied to SOTA Quantization-Aware Training methods with **both** weight and activation quantization. In addition to LSQ [14], we additionally test QuEST [36], a new SOTA QAT method published (in June 2025) after we submit FOGZO (in May 2025). QuEST introduces a hadamard transform before quantization, and can significantly outperform LSQ. We add FOGZO to the publicly available source code of QuEST following Algorithm (1). We compare QuEST against QuEST-FOGZO, and LSQ against LSQ-FOGZO, and report the evaluation perplexity. Note that the results below are training results of models with both weights and activations quantized to 2-bit, and we use a $\\beta_{min}$ of 1 - 1e-10.\n\nWe use the benchmarks in the QuEST source code: LLaMA-30m and LLaMA-50m. Note that LLaMA-30m has 95 million parameters and LLaMA-50m has 125 million parameters, which is 5x to 6x larger than the largest model we have used in the submitted paper. Following QuEST’s advice, we train llama-30m with 3 billion tokens, and llama-50m with 5 billion tokens. \n\n|Model|QuEST|QuEST-FOGZO|LSQ|LSQ-FOGZO|\n|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|\n|LLaMA-30m|37.75|**37.37**|39.06|**37.38**|\n|LLaMA-50m|32.28|**32.06**|33.28|**32.41**|\n\nResults show that FOGZO can improve both QuEST and LSQ. In addition, FOGZO can even shrink the perplexity difference between these two SOTA methods.\n\n## Training Time\nThree reviewers have listed increased training time/computation overhead as a weakness of the method, and we agree that this is a valid concern. However, we note that there are certain use cases where model quality takes precedence over training time. For example, the operator of a large fleet of already deployed self-driving cars may find it more economical to invest extra computation time in training versus needing to upgrade hardware in existing cars if those are the only two options for improving accuracy/performance/safety.  Intellectually, we are interested in the limits of inference accuracy at low precision without regard to training cost and believe this direction has merits in such scenarios. \n\nWhile FOGZO introduces an additional training overhead of 2 forward passes per iteration, we show that the training time overhead can be reduced by using the STE at the beginning of training, and switch to FOGZO at the end of training. Concretely, this means we only execute lines 6-16 in Algorithm 1 if $t \\ge r * T$, where $r$ is how much (70%, 80%, 90%, etc.) of the training is performed in STE mode. In the following table, we use LSQ to train llama-30m with 3 billion C4 tokens. We report evaluation perplexity and training time. While FOGZO alone incurs an overhead of 58% to tradeoff a perplexity decrease of 1.6, a mixing strategy that uses 90% of STE and 10% of FOGZO can still improve over the STE by a perplexity decrease of 0.4 with only a 6% increase in training time. \n| Method              | Evaluation Perplexity | Training Time (hours) |\n|---------------------|-----------------------|-----------------------|\n| STE                 | 39.06                 | 3.1                   |\n| 90% STE + 10% FOGZO | 38.67                 | 3.3                   |\n| 80% STE + 20% FOGZO | 38.38                 | 3.5                   |\n| 70% STE + 30% FOGZO | 37.93                 | 3.7                   |\n| FOGZO               | 37.38                 | 4.9                   |\n\nBelow we extend the training time of the STE by adding more training data. We compare this pure-STE-with-more-data strategy and the previous mix-STE-FOGZO strategy under the same training time. Each row is a separate training run. Results show mix-STE-FOGZO outperforms. This also suggests FOGZO can have higher data efficiency than STE, as the FOGZO runs use less training data to achieve lower perplexity. \n\n| Method              | C4 Tokens (billion) | Evaluation Perplexity | Training Time (hours) |\n|---------------------|---------------------|-----------------------|-----------------------|\n| STE                 | 3.174               | 38.69                 | 3.3                   |\n| 90% STE + 10% FOGZO | 3                   | **38.67**                 | 3.3                   |\n| STE                 | 3.348               | 38.43                 | 3.5                   |\n| 80% STE + 20% FOGZO | 3                   | **38.38**                 | 3.5                   |\n| STE                 | 3.522               | 38.25                 | 3.7                   |\n| 70% STE + 30% FOGZO | 3                   | **37.93**                 | 3.7                   |\n\n\n## Questions\n1. Could perturbing activations (not just weights) during ZO steps resolve the performance gap?\n\nOur initial thought is to use activation perturbation. However, this requires capturing the STE-estimated gradient of activations, which can be prohibitively expensive to store. Upon further investigation, we found that FOGZO can be used as is for activation quantization as noted above.\n\n2. Or can your maintain the STE for activation scale training to see how FOGZO works with weight-activation quantization?\n\nYes, this is exactly how we produce our new results above. We use the STE for both activation and weight quantizers, and simply apply FOGZO as is following Algorithm 1.\n\n3. Could you compared your methods with EWGS? This is a paper using second-order information to suppress the STE, maybe both the efficiency and performance.\n\nWe find EWGS to be an elegant method that addresses the flaws of the STE in an orthogonal direction. We believe EWGS and FOGZO may benefit from each other. This is because FOGZO assumes nothing about its source of bias/the chosen STE, only that it is a “mostly accurate” gradient estimator. In fact, the computation reduction effect of FOGZO relies on this prior belief that the source of bias is sufficiently accurate. Without a proper high-quality source of bias, FOGZO reduces to SPSA, which suffers from large gradient variance when computation is limited. Since EWGS uses second-order information to produce better quality descent directions, EWGS may be used as a better-than-STE source of bias for FOGZO. In addition, we note that as in Figure 1, FOGZO is by no means a perfect gradient estimator compared to SPSA with unlimited computation. Therefore, there is still a large accuracy gap that can be obtained with better sources of bias like EWGS. \n\nUnfortunately, we are unable to complete experiments on EWGS by the rebuttal deadline due to resource limitations. We will be sure to at least cite and discuss EWGS in the camera-ready version.\n\n4. Does FOGZO’s gradient variance reduction amplify benefits from optimizers like AdamW or Lion?\n\nWe note that FOGZO’s gradient variance reduction is compared to SPSA. Compared to the STE, we find that FOGZO still generates slightly larger gradient variance due to $\\beta_{min}$ not being exactly 1, as shown in Figure 1 empirically and discussed theoretically in Section 3.2. Therefore, we believe FOGZO actually benefits from the gradient variance reduction effect of optimizers like Adam. We use AdamW by default for experiments that use models with the Transformer architecture (such as DEIT and LLaMA). We also used AdamW for the experiments in Figure 1. \n\n5. Could you share the time cost and memory usage of your methods compared to STE?\n\nBelow we report the training time on the largest model, one from each model architecture. As noted above, we show that FOGZO’s overhead can be reduced with a mixing strategy between STE and FOGZO. \n\n| Model      | Data                   | GPU         | STE (hour) | FOGZO (hour) |\n|------------|------------------------|-------------|------------|--------------|\n| Resnet-50  | 90 epoch Imagenet-100  | RTX 2080 Ti | 13.6       | 24.6         |\n| DeIT-Small | 300 epoch Imagenet-100 | RTX 2080 Ti | 14.7       | 23.6         |\n| LLaMA-50m  | 5B C4 token            | RTX 5090    | 6.5        | 10.2         |\n\nBelow we report the training memory on the largest model, one from each model architecture. We do not observe any difference in training memory. We believe this is because the bottleneck of training memory is the backward pass. Once the backward pass is completed, most of the intermediate results are freed, and FOGZO’s subsequent two forward passes has the same training memory demand of inference.\n\n|            | STE (GB) | FOGZO (GB) |\n|------------|----------|------------|\n| Resnet-50  | 7.3      | 7.3        |\n| DeIT-Small | 10.2     | 10.2       |\n| LLaMA-50m  | 19.9     | 19.9       |\n\n6. I saw you added hardmard transform in your llama models, would this method be helpful with hardmard transform or can this improve the performance after transformed?\n\nYes, FOGZO can improve the performance when hadamard transform is used as noted in the results above with QuEST. \n\n7. Could you share the FLOPs for different step-size of ZOs?\n\nWe do not quite understand this question. By step size, are you referring to the learning rate? Is this the FLOPs measurement in Figure 1?"}, {"review_id": "wlPLVP7M9i", "reviewer": "Reviewer_WuB8", "summary": "The paper addresses the challenges of training neural networks with quantized parameters by proposing First-Order-Guided Zeroth-Order Gradient Descent (FOGZO), a hybrid gradient estimator. FOGZO combines the Straight-Through Estimator (STE), which is computationally efficient but biased, with zeroth-order (ZO) methods, which are theoretically sound but computationally expensive. The authors theoretically analyze their approach, claiming reduced bias compared to STE and improved efficiency relative to ZO methods, supported by empirical evaluations across diverse neural network architectures.", "questions": "Please address the above-mentioned weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\nIntegrating STE with ZO methods offers a relatively novel methodological perspective. \n\nThe paper provides detailed mathematical analyses demonstrating how FOGZO interpolates between biased STE gradients and unbiased ZO gradients. The paper evaluates multiple neural architectures on relevant datasets, providing some empirical evidence supporting their claims.\n\nImproving quantization-aware training is relevant given the increasing demand for efficient neural networks in resource-constrained environments.\n\n# Weaknesses\n\nThe paper is very hard to follow. The authors overuse fuzzy adverbs of frequency such as mostly, often, some, etc.\n\nWhile experiments do indicate improvements over STE, the practical significance of some results is relatively limited (e.g. only 0.5 to 2% improvement on popular models like ResNet). Given that, despite being more efficient than pure zeroth-order methods, FOGZO still incurs a substantial computational increase (~50%) over STE, it's practical relevance might be limited. Moreover, the approach currently applies strictly to weight quantization, potentially reducing it's relevance in practice even further.\n\nThe experimental results seem to be limited to individual runs. Repreating each experiment multiple times and reporting mean and std would make results more robutst. Right now, the instances where FOGZO does improve over STE could be outliers, particularily in experiments where the difference to the baseline is not that large.\n\nQuantization in Graph Neural Networks is not discussed, despite these architectures having gained importance over the recent years.\n\n# Conclusion\n\nThis is a borderline paper to me. If the authors sufficiently address the weak empirical part, I might consider changing my score from borderline reject (3) to borderline accept (4), but not more given the hard-to-follow write-up and expected limited impact.", "was_revised": true, "final_justification": "The authors alleviated some of my concerns and I increased my score accordingly.", "rebuttal": "We thank Reviewer WuB8 for reviewing our paper. We apologize for the poor writing quality. As authors, it is our job to clearly convey the message to readers, and we have not done a sufficiently good job. Since NeurIPS does not allow us to submit an updated version of the paper during the rebuttal, we promise to update the camera-ready to avoid overusing adverbs of frequency and make the paper easier to follow.\n\n## Activation Quantization\nAlthough we initially planned to leave activation quantization for future work, all reviewers list activation quantization as a weakness/question. Upon further investigation, we present new evidence that FOGZO can be applied to SOTA Quantization-Aware Training methods with **both** weight and activation quantization. In addition to LSQ [14], we additionally test QuEST [36], a new SOTA QAT method published (in June 2025) after we submit FOGZO (in May 2025). QuEST introduces a hadamard transform before quantization, and can significantly outperform LSQ. We add FOGZO to the publicly available source code of QuEST following Algorithm (1). We compare QuEST against QuEST-FOGZO, and LSQ against LSQ-FOGZO, and report the evaluation perplexity. Note that the results below are training results of models with both weights and activations quantized to 2-bit, and we use a $\\beta_{min}$ of 1 - 1e-10.\n\nWe use the benchmarks in the QuEST source code: LLaMA-30m and LLaMA-50m. Note that LLaMA-30m has 95 million parameters and LLaMA-50m has 125 million parameters, which is 5x to 6x larger than the largest model we have used in the submitted paper. Following QuEST’s advice, we train llama-30m with 3 billion tokens, and llama-50m with 5 billion tokens. \n\n|Model|QuEST|QuEST-FOGZO|LSQ|LSQ-FOGZO|\n|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|\n|LLaMA-30m|37.75|**37.37**|39.06|**37.38**|\n|LLaMA-50m|32.28|**32.06**|33.28|**32.41**|\n\nResults show that FOGZO can improve both QuEST and LSQ. In addition, FOGZO can even shrink the perplexity difference between these two SOTA methods.\n\n## Training Time\nThree reviewers have listed increased training time/computation overhead as a weakness of the method, and we agree that this is a valid concern. However, we note that there are certain use cases where model quality takes precedence over training time. For example, the operator of a large fleet of already deployed self-driving cars may find it more economical to invest extra computation time in training versus needing to upgrade hardware in existing cars if those are the only two options for improving accuracy/performance/safety.  Intellectually, we are interested in the limits of inference accuracy at low precision without regard to training cost and believe this direction has merits in such scenarios. \n\nWhile FOGZO introduces an additional training overhead of 2 forward passes per iteration, we show that the training time overhead can be reduced by using the STE at the beginning of training, and switch to FOGZO at the end of training. Concretely, this means we only execute lines 6-16 in Algorithm 1 if $t \\ge r * T$, where $r$ is how much (70%, 80%, 90%, etc.) of the training is performed in STE mode. In the following table, we use LSQ to train llama-30m with 3 billion C4 tokens. We report evaluation perplexity and training time. While FOGZO alone incurs an overhead of 58% to tradeoff a perplexity decrease of 1.6, a mixing strategy that uses 90% of STE and 10% of FOGZO can still improve over the STE by a perplexity decrease of 0.4 with only a 6% increase in training time. \n| Method              | Evaluation Perplexity | Training Time (hours) |\n|---------------------|-----------------------|-----------------------|\n| STE                 | 39.06                 | 3.1                   |\n| 90% STE + 10% FOGZO | 38.67                 | 3.3                   |\n| 80% STE + 20% FOGZO | 38.38                 | 3.5                   |\n| 70% STE + 30% FOGZO | 37.93                 | 3.7                   |\n| FOGZO               | 37.38                 | 4.9                   |\n\nBelow we extend the training time of the STE by adding more training data. We compare this pure-STE-with-more-data strategy and the previous mix-STE-FOGZO strategy under the same training time. Each row is a separate training run. Results show mix-STE-FOGZO outperforms. This also suggests FOGZO can have higher data efficiency than STE, as the FOGZO runs use less training data to achieve lower perplexity. \n\n| Method              | C4 Tokens (billion) | Evaluation Perplexity | Training Time (hours) |\n|---------------------|---------------------|-----------------------|-----------------------|\n| STE                 | 3.174               | 38.69                 | 3.3                   |\n| 90% STE + 10% FOGZO | 3                   | **38.67**                 | 3.3                   |\n| STE                 | 3.348               | 38.43                 | 3.5                   |\n| 80% STE + 20% FOGZO | 3                   | **38.38**                 | 3.5                   |\n| STE                 | 3.522               | 38.25                 | 3.7                   |\n| 70% STE + 30% FOGZO | 3                   | **37.93**                 | 3.7                   |\n\n\n## Individual Runs of Experiment\n\nBelow we report 4 rows of Table 2 with repeated runs and different random seeds. We are currently running experiments for the other rows in Table 2. Due to resource constraints, some of the larger experiments (such as the Imagenet-1k ones) require ~25 days. However, we will include all of Table 2 in the camera-ready.\n\nFor each data point, we repeat the experiment 5 times and report mean and 2 standard deviations (covering about 96% probability mass). We report both training loss and evaluation accuracy. However, we note that training loss is the most direct indicator of gradient quality. Evaluation accuracy, on the other hand, is an indicator of both gradient quality and effectiveness of regularization techniques, which is not a focus of this work.\n\n| Experiment                                                | LSQ + Identity STE           | LSQ + Identity FOGZO         |\n|-----------------------------------------------------------|------------------------------|------------------------------|\n| DeiT-Tiny + Imagenet-100 (Training Loss / Eval Accuracy)  | 3.00±0.0166 / 72.76±0.31     | **2.98±0.012 / 72.92±0.60**  |\n| DeiT-Small + Imagenet-100 (Training Loss / Eval Accuracy) | 2.62±0.012 / 79.55±0.86      | **2.57±0.016 / 80.06±0.48**  |\n| Resnet-18 + Imagenet-100 (Training Loss / Eval Accuracy)  | 0.45±0.0084 / **80.23±0.84** | **0.44±0.0098** / 80.04±1.14 |\n| Resnet-50 + Imagenet-100 (Training Loss / Eval Accuracy)  | 0.43±0.0182 / 82.81±0.82     | **0.39±0.0086 / 83.67±0.26** |\nLastly, we note that in the language model quantization community, reporting the standard deviation is not a convention, unlike in the vision model community. For example, papers [26,43,32,27,36] do not report standard deviation in their tables.  We hypothesize this is due to the prohibitive cost of running large models multiple times, and we are also experiencing the same issue. \n\n## Questions\n\n1. Quantization in Graph Neural Networks is not discussed, despite these architectures having gained importance over the recent years.\n\nQuantization methods for Graph Neural Networks typically utilize the fact that real-world graph dataset are sparse and nodes have different numbers of neighbours, and propose specialized quantization methods that reduce the quantization error based on the degree of nodes such as [a] and [b]. These methods can benefit from general-purpose quantization methods like LSQ and QuEST, which accelerates linear and convolutional layers regardless of the architecture of the model. FOGZO is designed to improve general-purpose quantization methods, by using their estimated gradient as a bias source to accelerate SPSA, while using SPSA to reduce the gradient bias of the STE.\n\n[a] Shyam A. Tailor, Javier Fernandez-Marques, & Nicholas D. Lane. DEGREE-QUANT: QUANTIZATION-AWARE TRAINING FOR GRAPH NEURAL NETWORKS\n\n[b] Zeyu Zhu, Fanrong Li, Zitao Mo, Qinghao Hu, Gang Li, Zejian Liu1,\nXiaoyao Liang, &  Jian Cheng. A2Q: AGGREGATION-AWARE QUANTIZATION FOR GRAPH NEURAL NETWORKS"}]

Author rebuttal:

Meta-review: **Overall Summary:**

This paper proposes an enhancement to the straight-through estimator (STE) for training quantized neural networks by incorporating zeroth-order information. The method introduces random perturbations to more faithfully approximate gradients, thereby reducing bias inherent in conventional STE. Experiments on CIFAR-10 and ImageNet show consistent improvements in top-1 accuracy for low-bit quantization, outperforming baseline STE and related estimators. Strengths include the simplicity and plug-in nature of the method, clear motivation, and empirical validation across multiple benchmarks, demonstrating that the proposed modification leads to tangible gains in challenging quantization regimes.

**Justification:**

The proposed estimator strikes a strong balance between accuracy and computational cost, retaining the efficiency of STE while improving its reliability. Reviewers agreed that the empirical improvements, though modest, are consistent and meaningful for advancing practical deployment of quantized models. The method is general, easy to implement, and directly applicable across architectures, which increases its utility for practitioners. The reviewers are in consensus on the acceptance. 


**Rebuttal Summary:**

During rebuttal, the authors clarified efficiency concerns by showing that the added computational overhead is minor relative to the accuracy gains, and provided ablations supporting robustness to hyperparameter choices. These clarifications alleviated reviewers’ doubts, and several acknowledged that the simplicity and practicality of the method enhance its impact. Although the paper does not introduce fundamentally new theoretical machinery, its systematic evaluation, practical relevance, and ease of adoption make it a valuable contribution. Therefore, I recommend acceptance as a poster.
Decision: Accept (poster)