Ternary Momentum For Quantized Training
Authors: Noga Bar
Amit Attia
Michal Moshkovitz
Dotan Di Castro
Venue: Accepted by TMLR ()

Abstract: Quantization enables efficient inference on resource-limited devices, yet training still depends on high-precision gradients and optimizer states.
We address this gap by introducing stochastic ternary momentum, a fully quantized optimizer that operates with quantized parameters, ternary gradient information, and enables ternary momentum states for stable and memory efficient quantized optimization.
Our method replaces deterministic and full-precision updates with integer-valued updates driven by stochastic sampling, ensuring that expected updates match standard momentum while maintaining strict memory constraints.
It eliminates re-quantization overhead and preserves quantization consistency throughout training.
We establish theoretical convergence guarantees of our ternary momentum method for convex objectives over bounded integer domains and for non-convex objectives over unbounded integer domains. Experiments on vision and language tasks demonstrate that our approach retains strong performance while reducing optimizer memory by 95\% compared to full-precision, advancing the feasibility of fully quantized training.

Published in Transactions on Machine Learning Research (03/2026)
Ternary Momentum For Quantized Training
Noga Bar" & Amit Attia Tel Aviv University
Michal Moshkovitz & Dotan Di-Castro
Bosch Research
Reviewed on OpenReview: https: // openreview. net/ forum? id= A3mVmPlah)
Abstract
Quantization enables efficient inference on resource-limited devices, yet training still de- pends on high-precision gradients and optimizer states. We address this gap by introducing stochastic ternary momentum, a fully quantized optimizer that operates with quantized parameters, ternary gradient information, and enables ternary momentum states for stable and memory efficient quantized optimization. Our method replaces deterministic and full- precision updates with integer-valued updates driven by stochastic sampling, ensuring that expected updates match standard momentum while maintaining strict memory constraints. It eliminates re-quantization overhead and preserves quantization consistency throughout training. We establish theoretical convergence guarantees of our ternary momentum method for convex objectives over bounded integer domains and for non-convex objectives over un- bounded integer domains. Experiments on vision and language tasks demonstrate that our approach relains strong performance while reducing optimizer memory by 95% compared to full-precision, advancing the feasibility of fully quantized training.
1 Introduction
Quantization has emerged as a critical technique for deploying deep learning models on resource-constrained devices, as it reduces both memory footprint and computational cost during inference [Courbariaux et al.
2016 Han et al. 2016 Wang et al. 2019b Jacob et al. 2018 Frantar et al. 2023 Li et al. 2016). However, quantization typically applied only deployment, while training still requires substantial memory and high-precision computations, particularly for large models where gradients and optimizer states must be stored (Rajbhandari et al. 2020). The gap between training and deployment requirements becomes significant when using extremely low-bit bit-width quantized models, where the difference in resource demands is most pronounced. Successfully addressing the challenge of efficient training holds immense promise: it would unlock the ability to train larger models on devices with limited memory.
When training quantized models, there are four primary sources of memory usage: the parameters and activations, which are used during both training and inference, and the gradients and optimizer states, which are required only during training. In this work, our focus is on memory efficiency during training, specifically low bit-width optimizer states with only ternary gradient information to allow low-resources quantized optimization.
The use of quantized gradients is natural line of research toward memory-efficiet training, In this scenario, rather than relying on full precision availability of the gradients, only partial information about them is available. We focus on the case of ternary or binary values that provide extremely coarse approximations of the true gradient. Such estimators have been studied mainly in the context of commmncation efficiency for distributed learning hat than memory reduction (Wen al. 2017
Sun el. al. 2023 Bernstein et. al. 2018
* Noga conducted this work during an internship Basch.

---

Published in Transactions on Machine Learning Research (03/2026)
Jiang et al. 2024 Karimireddy et al. 2019a]. Those methods are designed for full-precision parameters and optimizer states which suffer from high memory consumption.
Another aspect of memory consumption during training is the use of high-dimensionall full-precision opti- mizer states which are required for standard optimization algorithms such as Adam (Kingma & Ba and momentum-basee methods (e.g., heavy-ball momentum (Polyak) 1964] and Nesterov momentum (Nes-
2014]
terov
1983 2013)), leading to substantial memory overhead during training. While several approaches aim
to reduce this cost by exploiting the tensorial structure of gradients to lower the dimensionality of optimizer
states
ZZho et al. 2024 Zmushko et al. 2025 Lialin et al.
2024 Luo et al. 2024 Zhang et al., 2025 He
ct al. 2025; Zhu et al.] 2025), more direct and general strategy is to quantize these states into low-bit
represenlations, independent of their structure. Although previous works have explored 4-bit quantization
of optimizer states (Li et al.) 2023 Modoranu et al.) 2024 Wang et al.) 2024 Alistarh et al., 2017], achiev- ing extremely low bit-width representations, such as ternary quantization, remains largely unresolved. The difficulty lies in the reliance of momentum-based methods on averaging tochniques that inherently assume continuous-valued states, making them dependent on floating-point storage and computation. A naïve so- lution is to re-quantize the momentum states after each update, but this often causes severe precision loss under strict bit budgets. The problem becomes even more acute when optimizing quantized parameters: since each update is already coarse, degraded momentum estimates can lead to poor update decisions. In such cases, the accumulation of inaccuracies from both quantized parameters and quantized optimizer states can produce unstable trajectories. Consequently, repeated quantization (whether stochastic or determin- istic) may yield updates that are either excessively large or negligibly small, destabilizing or harming the convergence of the optimization process. This fundamental mismatch between the continuous nature of con- ventional optimizers and the discrete structure of quantized training represents key bottleneck to achieving truly memorr-efficient optimization.
Additionally, when aiming to apply standard optimization methods which are designed for piehiprecision gradients, with only extremely limited gradient information, it is unclear how they should be adapted to the coarse gradients, how their convergence guaranlees extend, and what information their auxiliary states can meaningfully capture. While sign-based optimization methods have recently attracted attention, their integration with conventional optimizers under strict memory constraints the parameters and optimizer states remains unexplored.
In this work, we address the several memory sources of the optimization. The first is the parameter domain, where we restrict the parameters to a small predefined set of discrete values, 0 E {Tminn ... maxx 1d where d is the model dimension, with particular emphasis on ternary parameters, rmin = -1, 7mxx = The second the gradient information, which we assume that only ternary gradient information available, i.e., I E 1,,, The last the optimizer states, where we focus specifically on the ternary states with E (-1,0, 17d. This extreme regime poses new challenges for training quantized models. Despite the restrictions, the described setting is increasingly relevant for training under tight memory and computational budgets, where every bit of storage matters. To address this gap, we introduce a tailored optimization method that harnesses low-precision gradients to perform momentum-like updates, making quantized optimization viable even under such conditions.
The contributions of our work can be listed as follows:
Stochastic Ternary Momentum. We introduce stochastic ternary momentum, an optimization method which includes novel modification of first-order momentum methods that enables the optimizer state and updates to be represented entirely with ternary values. The key innovation is to replace deterministic momentum and parameter updates with random update rule. At each iteration, we sample Bernoulli masks that determine which coordinates of the momentum and parameters are updated. Instead of applying full-precision magnitudes of the updates, they are realized through probabilistic masks that respect ternary constraints with updates that are limited +1, -1 or O. Our method builds on the availability of ternary gradient approximations, such as sparse zero-sign representations (Zhang et al. 2024 Chen et al.] 2024
Zhou et al. 2025 Bar & Giryes] 2025), discrete non-backward approximations
Di Castro al. 2024 and
the seminal TernGrad approach [Wen et al. 2017).
2

---

Published in Transactions on Machine Learning Research (03/2026)
Expected Updates Match Deterministic Momentum. Despite relying discrete updates, our design is principled: we show that the expected updates of both momentum and parameters exactly match their deterministic counterparts when relying on ternary gradient information. This guarantees that the stochas- ticity introduced for quantization compatibility does not bias the optimization process, while still enabling convergence with low-memory and fully quantized updates.
ceantizaationPPeeevviii Optimization. Beyond extreme memory efficiency, a major advantage of our method is that it natively preserves quantization. Unlike conventional approaches, it does not require mixing full-precision and quantized variables, thereby eliminating the repeated overhead of re-quantizing parameters or optimizer states. This property ensures both consistent quantization throughout training and robustness under tight bit budgets.
Theoretical Convergence Guarantees, To further motivate our ternary momentum optimization algo- rithm, we establish convergence guarantoes for our method assuming deterministic gradient sign access in two different settings. (i) In the case of separable convex functions over discrete and bounded domain, our algorithm converges to the global optimum. (ii) For minimizing more general smooth non-convex functions over the unbounded integer set, provide convergence guarantee to local minimum, i.e., critical point.
We empirically validate our method across diverse architectures and tasks, including image classification on ImageNette and language model fine-tuning with ToRA.. Our experiments span ternary parameters, various gradient approximation techniques, and datasets such as IMDB, SST2, COPA, and WinoGrande, using fully connected networks, ViTs, OPT-1.3b and Llama2-7b. Our results demonstrate a reduction in optimizer state memory consumption by factor of (3)g(3))32 which is equivalent to = 5% compared to the full-precision firs-mmomentum counterpart. Our evaluation demonstrates that our algorithm is both effective and robust, even under extreme memory constraints. These findings highlight the practicality of our approach and open the door to fully quantized optimization in low-resource training environments.
Experimental Results.
2 Related Work
Quantization. Neural network quantization is primarily used for inference, where post-training quantiza- tion (PTQ) methods dominate. In these approaches, models are first trained in full precision using large computational resources and are then quantized and deployed on resource-limited devices (Jacob et al. 2018:
Xiao et al. 2023; Frantar al. 2023 Han et al. 2016 Liu et al. 2021 Wang et al. 2019b Li et al. 2016 An alternative paradigm is quaatizationnawarre waroo---wwa----rniinggggg (QAT), where parameters are repeatedly quanlized during training to better simulate inference-time behavior. However, QAT methods typically assume access to full-precision gradients and rely on optimizers designed for full-precision updates, which therefore require
maintaining high-precision optimizer states in memory (Esser et al. 2020 Zhou el al. 2016 Rastegari et al.
2016 Choi et al., 2018 Courbariaux al. 2015 Dettmers et al., 2022 Wu al. 2018).
Memory Efficient Optimizers. Recent work has also explored reducing the memory footprint of opti- mization algorithms. One line of research designs low-dimensional optimizers, where gradients and optimizer states are stored or updated in a low-rank or sparse subspace while maintaining full-precision representations
(Zhao et al.] 2024 Zmushko et al.
2025
Lialin et al. 2024 Luo et al.) 2024; Zhang et al., 2025) He et al.
2025 Zhu et al. 2025; Hao et al.]
2024
Liang et al. 2024 Yao et al., 2022 Lin et al. 2018 Stich et al,
2018 Alistarh et al. 2018 Wangni et al.
2018). Another line of work focuses on reducing the number of
trainable parameters rather than the optimizer states, a common strategy in fino-tuning. Paramette-Efffiicnt
Fine-Tuning (PEFT) methods (Han et al.
20244a| Houlsby et al. 2019 Ben-Zaken al.) 2022 Hu et al., 2022), among which LoRA (Hu al. 2022 is particularly influential (see Han et al. (2024b) for compre- hensive survey), introduce lightweight low-rank adapters or other compact modules into pretrained models to minimize memory usage during fine-tuning. Although PEFT methods do not directly address optimizer
quantization, they naturally result in lower memory usage for gradients and optimizer states due to the reduced number of trainable parameters.
The works most relevant ours are those that explicitly quantize the optimizer states, inchuding methods
designed for Adam (Zhang et al. 2025: Modoranu et al. 2024) and other optimizers (aang et. al.) 2024
3

---

Published in Transactions on Machine Learning Research (03/2026)
al. 2023 However, these approaches typically assume access to full full-precision gradients and often rely relatively high bitwidths (e.g., bits), which limits their applicability in highly resource-constranned training settings.
Di Castro et al. 2024 Bar & Giryes 2025 Zhou et al. 2025
3 Background and Problem Setup
In the following section we list the relevant background for our ternary momentum algorithm.
Full-Precision Optimization with Momentum. We begin by recalling the standard optimization setup involving gradient access and momentum. The objective is to minimize a differentiable loss function L R R. Starting from initial paraneters 6g E F at each iteration t= = 1....... .T. we obtain noisy gradient estimate 9t Rd of the true gradient VL((0-1) and determine the next iterate 0g based on the previously observed (noisy) gradients. Typically, this noisy estimate is computed using randomly sampled mini-batch of training data.
To mitigate the stochastic noise in this process, the heavy-ball method (Polyak) 1964) maintains a running average of past gradients, m, = 3m4-1 + (1 f)gt, with my = and a momentum paramcter 3 E (0, 1). These updales require high-dinensional real oplimizer states that are stored in floating- pointt memory. The parameter update rule then given by 0, 0:- ne,,, where Th denotes the learning rate at step
Quantized Setting. In the quantized optimization sctting, the parameter space is restricted to discrete values. Specifically, assume 0, zá, with each coordinate bounded between some integers Tmin and Tmax- Throughout this work, we mainly focus on the ternary case, where Tmin = -1 and Tmax = 1, that 0, E {-,,0, In addition, we assume access to quantized gradient information oracle, denoted by TernGrad(0), (0), which given the current parameters 0, returns a ternary gradient approximation ğ E (-1, 0, 1)dd The optimization objective remains the same-to minímize the loss [(0) but under the strict constraint that the parameters must remain within the discrete and bounded range at all times.
In this work, propose an optimization scheme that employs ternary momentum, offering high memory efficiency while preserving optimization effectiveness. We consider the setting where the momentum is restricted to discrete ternary values, my E 1, and depends solcly on ternary gradient information, z E (-1,0, 134. Furthermore, our objective is to learn discrete model parameters, 0, E [fmin; Tmax] C Z, that remain discrete throughout the optimization process.
3.1 Naive Quantized Optimization with Ternary Gradients
Optimization without momentum. A straightforward approach to quantized optimization with ternary gradient information is to update the parameters directly using only the ternary gradient, 9, f--1 9t- However, such updates can be overly aggressive, often causing instability in the optimization process Bernstein et al. (2018): Karimireddy et al. (2019b). A natural remedy is to scale the update by learning rate, 8, = 0, ne9. but this introduces a weakness: it requires mixed-precision operations between the quantized 0, and the floating-poin Nh. E .dd. Hence, the parameters 0g must be repeatedly quantized after each step, reintroducing instability and computational overhead [Curbarriux et al. (2016).
Optimization with momentum. Another natural attempt to stabilize training is to incorporate mo- mentum into the ternary gradient approxination. In this case, the moving average produces real-valued momentum states, m 8m,-1 + (1 8)ğ. Idd, which in turn yield real-valued parameter updates, 0, 0,-1 7)m, E Rd. To enforce quantization, both the parameters and optimizer states must be re- quantized after every update, Yet, under the aggressive quantization regime considered in this work, repeated re-quantization amplifies the mismatch between the continuous dynamics and their quantized counterparts, leading to updates that are either vanishingly small or excessively large, depending on the hyperparameters. This ultimately undermines stability and prevents reliable convergence.
4

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm 1 Ternary First Moment
Input: 0n E friins... ] 1 with integers Tiin < mmax, learning rate {n Jn- momentum coefficient 3 TernGrad ternary gradient algorithm, and bounds Tmin: Tmax- Initialize: mo = 0
for = I,.,, do
Compute ternary gradient approximation: 9. TernGrad((..1)
Update momentum:
Sample vector eTH)) Bernoulli(3) i..d. clement- wise where e(m)) {0,01) 13 Momentum Update: m, = = (m) m + (1 et (m²) @ gr.
Update parameters:
Sample vector e, Bernoulli(ne) i..d. element wise where (0) {0.1)* (0)
Updale rule: O2 clamp/ [raan /max] (0.-1 es ① m+}
end for
Return optimized parameters 0r
4 Optimization with Ternary Gradient and Momentum
In this section, we present our momentum-basee optimization method designed specifically for the setting of quantized parameters, ternary gradients, and quantized first-order momentum. The method is detailed in Algorithm
The required imputs to our algorithm are quantized initial parameters that are bounded by Tiin and Tmax: learning rate πI that can be adapted over time, a momentum coefficient, and a function for acquiring the ternary approximation of the gradient. At each iteration t, we get the ternary information of the gradient using the TernGrad function. Then, unlike standard momentum updates that involve floating-point arithmetic (e.g. m = 3m3-1 + (1 3)g.), our approach employs a ternary Bernoulli approximation of the momentum. Specifically, for each entry, we sample a binary random variable e(T)) E {0, from a Bernoulli distribution with parameter B. This forms vector (mn) E [0,1]4 which then dictates element-wise the momentum update: if (m) 1, the i-th component of the momentum my retains its previous value m4-1 otherwise, if (m) = 0, it is replaced by the corresponding component of the current ternary gradient Jr. This is formally expressed as:
m = (m)) c 0m-1++ + (1-ee (mm) 2 Dt,
(m) ~ Bernoulli()).
(1)
The hyperparameter controls the probability of which we rely on past. approximations that are stored the momentum, for higher values of the updates rely more heavily on the history. This stochastic design is critical because it intrinsically ensures that the momentum m, remains confined to ternary values, thus avoiding the need for high-precision storage and mitigating memory demands.
Next, we describe the parameter update step. Instead of directly applying a floating- learning rate (e.g.. 6. = - - m));) we again leverage a Bernoulli approximation. For each parameter ot, a binary random (9)
variable e E {0, 1} sampled from a Bernoulli distribution with parameter 7), (the learning rate). This generates vector e!") E 10, The parameter update then becomes 0, 0.- 0-11-e00 @ m, This effectively means that each parameter is updated by its corresponding momentum component only with probability equal to the learning rate 7/--
Unbiased Updates Property. Our stochastic update rules of the momentum and parameters are moti- vated by their expected behavior. Despite the use of integer values, the stochastic nature of the updates maintains consistency with full-precision optimization. In expectation, the updates for both the momentum and the parameters preciscly match their full-procision counterparts. Formally,
m1m 9m,-1 + B)gr, and E000,0,-1, me] 0,-1 m/
5

---

Published in Transactions on Machine Learning Research (03/2026)
This theoretical equivalence is crucial as it suggests that the algorithm may achieve similar convergence properties to traditional methods with access to lernary gradients while operating under severe memory and precision constraints.
Our proposed ternary momentum algorithm enjoys several significant strengths, particularly for constrained environments. By storing momentum states in ternary form (requiring log2(3) bits) rather than full-precision floats (32 bits), the memory footprint is reduced by a factor of 1ogg(3) ~ 0.048. Moreover, our approach maintains integer-valued parameters 0 and momentur updates throughout training, eliminating the need for muxxed-precisiog operations and avoiding the overhead and potential precision loss of repeated quantization. Updates are performed using randomized masks e(m) and ( with integer arithmetic, ensuring fully integer- only optimization.
32
In addition, the expected number of parameter updates be controlled, providing a proxy for energy efliciency on low-precision hardware. Specifically, for momentum coelficient. 3 and learning rale T, the expected number of optimizer state changes per iteration is approximately (1, where d is the number of parameters. Smaller πt or larger 3 reduce the variability of updates, allowing principled regulation of both update sparsity and energy consumption (Stutz et al. 2021
5 Theoretical Analysis
Next, we present a thcorctical convergence analysis our ternary first-moment algorithm (Algorithm 1 under different assumptions on the loss function and parameter domain. Throughout, we assume that the loss function satisfies the following element-wist smoothness assumption.
Definition (Element-wise smoothness). A differentiable function f Re + R is element-wise L (Lı...., Lmo-sh for Li...., ......,,, La 2 0, if for any x, E R
d
(5()) - (f(x) + VS(x) (y- x))1 S
Li(: - -x;)².
This assumption, previously used in the analysis of SignSGD (Bernstein et al. 2018 naturally captures heterogeneous curvature across coordinates and is well suited to sign-based methods, where each coordinate update has a fixed magnitude.
In addition to the smoothness assumption, we distinguish between two settings:
Separable, convex, and smooth setting. We assume that the objective is separable, i.e., of the form ((0) = Ei=, (0,,,, where each L, IR R convex and L,-smooth for i 1, I,...,,. d. This is equivalent to assuming that the separable function L(0) itself is convex and L-smooth, where L (LI...., La). In this setting, convergence is measured in terms of loss suboptimality, [(0) - ming (()').
Non-convex and smooth setting. Here we assume that L Rd + is element-wise L-smooth but not necessarily separable or convex. To accommodate this broader function class, we analyze Algorithm D in the case where Tmin -0 and Tmax co, i.e., no clipping applied. As standard in non-convex optimization, convergence is measured in terms of the gradient norm, which reflects convergence to a critical point.
We focus on these two sottings, rather than more general convex or non-convex formulation, due to the intrinsic limitations of sign-based methods with quantized updates of the form 0, 0,-1 E (- 1]d. As demonstrated by Karimireddy et al. (2019b), even the simpler SignSGD method fails to converge in certain worst-case instances for more general classes of such problems. While avoiding clipping departs from strict memory efficiency and separability is unlikely to hold in practical scenarios, convergence guarantees established in the above settings provide theoretical support and a foundation for general analyses.

---

Published in Transactions on Machine Learning Research (03/2026)
Finally, for ternary gradient information, we use the element-wise deterministic sign operator over the de- terministic gradient at time 4
+1 if VL(()) > 0:
9t =
= 0 if VC((0) = 0;
VL())) <
We focus on the deterministic setting for simplicity. In stochastic settings, a common strategy to mitigate gradient noise to employ very large batch size (Berustein al.) 2018), effectively reducing the problem to (semi-)ddtterminstiic problem. A possible alternative is to adopt an average-case framework, such random least-squares problems, which can provide more refined characterization of gradient noise (Paquette
ct al. 2021 Paquette & Paquette 2021 This promising direction lies outside the scope of our work.
In Appendix D we present synthetic linear regression simulations to illustrate the behavior of our method in both separable and non-separable settings under gradient noise induced by mini-batching batching. These simulations demonstrate that full-batch training yields the best performance, and that our method remains effective with moderately sized mini-batches, though gradient noise can have a large effect on the final performance for small mini-batches, highlighting the importance of stable ternary gradient estimation.
5.1 Separable, Convex and Smooth Objectives
Following, we formally detail the high-probability suboptinality guarantee for Algorithm given separable, convex and clement-wise smooth objective.
Theorem 1. Let 8 (0, 8 (0, 1), integers Tmin < Tmaxx separable, convex, and element-uiise L-smooth i= (,d 0, E :
For any T 2 Tmax - Tmin + Ag, where Ag & [log (dT) / 1))(1/3)1, consider the output 0r produced by Algorithm with porameters Po mmins mmax: 3, 1T for 1. I....,T, T, and the deterministic signed gradient as TernGrad. Then with probability at least 8,
function ((0) =
(8gg(8/(dT)) Σi=
1 [(Or) - ming "mx) S 5 Lll (ogT///////g(133]
where the probubility is with respect to the random masks d.m)) for t I.
For a sufficienlly large number of steps, Theorem 1 establishes convergence rate of O(I Up to logarithmic factors, this rate is essentially tight: in the worst case, the distance from the minimizer along each coordinate is lower bounded by half the update magnitude, resulting in a worst-case suboptimality of Dl1) For example, this lower bound is attained for simple quadratic objective E L,(x; 0.5)²
The proof of Theorem 1 provided in Appendix C.1 relies two key observations. First, due to separability, the gradient with respect to each coordinate always points toward the minimizer. Second, with high prob- ability, the momentum term cannot drive the iterate away from the minimizer for more than a logarithmic number of steps; afterward, the coordinate necessarily returns toward the minimizer.
5.2 Non-Conver Smooth Objectives
The result in Section 5. addresses relatively restrictive family of objective functions. We now turn to a more general class of non-convex smooth functions. In this broader setting, we necessarily provide weaker form of guarantee. Specifically: (1) we no longer assume bounded coordinates, that is, Tmin = -o and Tmax = 0o, resulting in optimization over 24; and (2) the convergence guarantee concerns minimizing the average f--orm of the gradients rather than the loss suboptimality
Theorem 2. Let L Rdd + R be an element-wic wise L-smooth function lower bounded by some L E R. Let fg nd and consider the sequence (0i) generated by Algorithm ] with porameters lg Tmin 00, Tmax
CConvergence in non-convex optimization commonly expressed terms of minimizing the gradient norm, which sponds to finding an approximate critical point (Ghadimi Lan 2013 Arjevani al. 2022

---

Published in Transactions on Machine Learning Research (03/2026)
00, (0, 1p for = I....,T,, T, and the deterministic signed gradient TernGrad. Then it holds that
[(00)- - C* (1 [1+,
T B[
(1 - ByT + -
Similar to Theorem [ the result above guarantees convergence to a neighborhood of size O(I which is unavoidable due the fixed magnitude of the updates. On the other hand, Theorem 2 complements Theorem 1 by showing that the algorithm converges, in terms of the average gradient norm, for broader class of objectives. Note that while Thcorem 2 fixes ?oe 1 for = 1...., T, we prove
general version that supports arbitrary "f, including the schedules 7/2 = 1/1 and t)t 1/V1, although these schedules yield slower convergence rate. See Appendix C.2 for full details.
Taken together, these results help explain the practical success of Algorithm | on favorable objectives- when the optimization landscape exhibits separability, or when convergence to a critical point corresponds strong empirical performance, as is often implicitly assumed when applying gradient- badie methods non-convex problems.
6 Experiments
We evaluate our proposed method across both image and language domains using various models to demon- strate its effectiveness and versatility. We include various datasets, lasks, and ternary gradient approximation methods.
Datasets and Tasks
1. Sentiment Analysis with IMDB:
We perform sentiment analysis using the IMDB dataset, lever- aging BERT features (Devlin et al. 2019). The model architecture consists of fully-connected network with three layers, each having hidden width of 4096.
2. Image Classification with ImageNette: We utilize a subset of ImageNet, known as ImageNettee (oward 2019), which consists of 10 classes. For this task, we employ the Vision Transformer (VIT) architecture (Dosovisskiy) 2020). We investigate two variants of our method: Hybrid Variant: In this configuration, we replace the feed-forward network in each transformer layer with ternary weights. Hybrid 2 Variant: This variant further involves replacing the key (K), query (Q), and value (V) matrices in the attention block while retaining the prediction head in full-precision. The exact mumber of ternary parameters used in both variants is detailed in Table
3. Fino-Tuning of Language Models:
We fine-tune OPT-1.3b (Zhang al. 2022 and Llama2-7b COPA (Wagg et al. 2019a) and WinoGrande (Sakaguchi et al., 202T)datasets using LoRA. A. In this setup, all learned parameters are ternary, while the parameters that remain frozen are maintained in full precision (unless stated otherwise). LoRA which lowers the dimension of learned parameters during fine- tuning highly compatible with our method's aim to reduce memory during training. Importantly, since our mothod requires the initial parameters to be discrete fine-tuning all the parameters with our algorithm require some post training quantization. Hence, we LoRA fine-tuning which introduces new parameters during fine-tuning so initialize them in quantized manner.
Touvron et al. 2023 language models for with SST-2 (Socher et al. 2013
We use the experimental setup and the code of Di Castro et al. 2024 for IMDB and ImageNette (results in Table D. Furthermore, we use the code of Zero-Order Optimization Benchmarking framework (Zhang al., 2024h which include code for fine-tuning LLMs with zero-order optimization method. We specifically used it for LorA fine-tuning (results in Table 3
Ternary Gradient Approximations
Our method requires access to ternary gradient approximations that can potentially be acquired efficiently, However, in experiments we mostly simulate this approximation. We use different method for ternarizing
8

---

Published in Transactions on Machine Learning Research (03/2026)
the gradient during optimization and asses the robustness of the benefits of our ternary momentum to different approximation methods. We use the following methods:
Deterministic Ternarization: In this approach, 10% of the full gradient entries with the smallest magnitude set to 0, while the remaining entries are assigned values based on their sign. For this approximation calculated the gradient of the current batch using backward pass.
2. Stochastic TernGrad (Wen et al. 2017): : widely known approach samples the spansity mask according to the gradient magnitudes, after which a sign operator is applied the selected non-zero entries. This method is well-known, it was designed for efficient distributed optimization where the batched gradients is computed in full-precision and then quantized to ternary values.
3. No Backward (Di Castro et al,, 2024): This method docs not involve a backward pass and relies on a discrete algorithm to assess the contribution of each parameter. The parameters are then updated using only {+1} values for influential parameters, yielding a ternary update step. As result, the method provides an efficient alternative for gradient estimation. Since this method aims to maintain parameters within bounded integer domain without FP operations, it does not employ sophisticated optimization schemes. Our method, which is designed for optinization in the same constrained regime, naturally complements their ternary gradient approximation, making it well-suited for ovaluation in this sctting.
4. Sparse Zero-Order (ZO) Zhang et al. 2024): : This method employs zero-order optimization approach (Liu et al. 2019 to estimale the gradient sign. Specifically, it perturbs the parameters by adding and subtracting small noise and determines the gradient sign based on the corresponding increase or decrease in the loss value. The sparsity arises from zeroing a randomly selected subset of parameters, where 30% of them are non-zero. Full-precision operation are required for the pertur- bation and then a full-precision forward is needed. The Backward pass not needed for zero-order approximation method.
Deterministic and TernGrad ternary gradient approximation methods require full access to the gradients and are therefore used primarily as simulations. In contrast, the latter two mothods do not rely on gradient information, making them more suitable for the studied use case. Additionally, although the Sparse 70 approximation does not require full gradient access, the parameter perturbations it introduces still necessitate storing full-precision copy of the model, which increases memory requirements.
Optimizer Baselines. Since current optimization schemes rely on high-precision gradient information rather than ternary one and are focused on high-precision updates of the parameters, using off-the-shelf optimizers requires additional adjustments to the sctting and honce we leave that for future work. Honce, we compare our method with two optimizers which are very relevant to the scenario we are focused on.
Stochastic Updates without Momentum: The update steps are chosen according to a randomly sampled masks, but no momentum used. The update step is 8 = 8;-1 e ÿ. where ea Bernoulli(Ht) (as in the parameter update step Algorithm 1 Note that this baseline memory efficient since it does not require storing 1-bit optimizer states (see Table 2
2. Full-Precision Momentum: The update steps are applied according to full-precision momentum. The momentum updates are m. = 3m4-1 + (1 - }) and the parameter updates are performed with stochastic rounding 0, SR(0.-1 e, gr) where el") Bernoulli(ye. Note that the original optimizer relies full-precision gradients, and its effectiveness under limited gradient information, such as in the case of ternary gradient approximation, remains unexplored. Most importantly, additional full-precision optimizer states are required to be stored.
In all experiments, we use a learning rate of 0.75/t, which depends on the itcration number, following Di Castro et al. (2024). In Appendix A we provide additional details on the hyperparameter used and hyperparameter search for the initial learning rate, learning rate scheduler, and momentum coefficient..
9

---

Published in Transactions on Machine Learning Research (03/2026)
Ternary Approx.
Optimizer
IMDB
ImageNette Hybrid
ImageNette Hybrid 2
of parameters
19.9M
38.33M
38.33M
of Ternary
19.9M
25.1M
37.3M
FP model+gao grad
AdamW
87.3
90.8
90.8
No Momentum
86.75+0.0s
83.6=0.48
78.32+0.a7
Deterministic
FP Momentum
86.19
86.78
$2.8920..
Ternary Momentum
86.8610.05
87.69 20.45
81.44
No Momentum FP Momentum
85.2710.18
89.5820.6
85.4620.06
Stochastic
#0.9
89.58=0.15
85.29
Ternary Momentum
85.28 1.288...0
89.62
86.3510.29
No Momentum
83.04
85. 20
75.020.17
No Backward
FP Momentum
85.29
83.97
71.44
Ternrry Momentum
85.57
88.23 :0.23
79.4
Table 1: Accuracy for training from scratch ternary parameters with multiple ternary gradient approxi- mation methods. The FP baseline refers to standard practice of training full-precision models which utilize AdamW. denotes the standard deviation, and bold indicates the highest average accuracy for each baseline.
Theoretical
Fully Connected
Hybrid
Hybrid
OPT-1.3b LORA
Llama2-76 LORA
No Momentum
0
0
0
FP Momentum
d
79.6
95.75
142.2
6
16.78
Ternary Momentum
d/ log, (3)
3.94
4.74
7.1
0.3
0.84
Table 2: Optimizer states memory MB.
6.1 Results
Training from scratch. Our initial experiments on standard classification datasets (Table 1 demonstrate that the proposed ternary first momentum method ef- fectively learns model from scratch across diverse tasks. We observe that our method, employing all ternary ap- proximation strategies, generally outperforms the quan- tized baselines, affirming the benefits of incorporating even an extremely low precision momentum for effective optimization.
100
deieministic drieministi:
Loss
stechetic
slochati mamerium
As expected, all tested configurations exhibit some degra- dation in accuracy compared to their full-precisin train- ing (parameters and optimizer states) counterparts. How- ever, these differences are often small, suggesting that models trained with ternary momentum can achieve com- petitive performance on standard classification tasks. Specifically, when both parameters and momentum are ternary, observe accuracy degradation of up to 2% for IMDB, 3.1% for ImageNette with Hybrid 1, and 9.5% for ImageNette with Hybrid 2. This pattern suggests that the degradation tends to increase as the number ternary parameters being optimized grows.
10 20
30
Step
Figure 1: Training loss curves (cross-entropy loss) on the IMDB dataset using deterministic and stochastic ternary gradient approximations. Using momentum results in faster convergence of the training loss. FP line represents the standard practice of training full-precision with AdamW.
Notably, in the ImageNette Hybrid task with determin- istic ternary approximation, our ternary momentum method showed some degradation compared to the
10

---

Published in Transactions on Machine Learning Research (03/2026)
Ternary Approx.
Optimizer
SST2
COPA Llama2-1 7b
WinoGrande
OPT-1.3b
Llama2-7b
FP param-ggrad
AdamW
96.7
85.0
69.5
No Momentum
93.0+0.30
73.0
68.6+0.28
Deterministic
Ternary Momentum
23.4
76.0
70.210.28
No Momentum 93.620.24
84.0
66.620.0m
Stochastic
Ternary Momentum
93.710.0
85.0
66. 1 10.14
No Momentum 93.210:39
80.0
68.520.5
No Backward
Ternary Momentum
93.5
83.0
68.9#0.07
No Momentum
93.4 20.41
84.0
67.4
Sparse ZO
Ternary Momentum
23.4
84.0
68.32..39
Table 3: Accuracy results of fine- fine-tuning ternary parameters of LoRA with SST2, COPA, and WinoGrande. FP baseline refers to standard practice for fine-tuning with LoRA which utilizes AdamW. Higher is better and the highest of each baseline is marked in bold. The median accuracy is reported for COPA. + denotes the standard deviation.
Method
IMDB
ImageNette
Method
SST2
Ternary
83.04
85.46
Ternary
93.58
Ternary
85.57
86.35
Cernary+MMmenne
93.69
Ternary Quant. Act.
79.86
39.42
Quantized Weights
93.12
Ternary+Mom++uunn, Act.
79.92
42.67
Quantized Weiggts++Momentum
93.27
Table 4: Comparison of accuracy results with quan- tized activation to 8 bits. Our ternary momentum enjoys accuracy benefits which suggests it can be used with activation quantization for memory re- duction.
Table 5: Effect of pretrained parameters quantiza- tion (to 8 bits) and ternary momentum SST2 fine-tuning accuracy. The results suggests that ternary momentum can be used with quantized pre- trained parameters.
full-precision baseline, likely due to higher sensitivity to gradient quantization noise in this larger and deeper model. Nevertheless, it still performmeedd the no-momentum baseline, highlighting its robustness and strong munorr-efficiencyy trade-off as detailed below. For other baselines, including ImageNette Hybrid-1 and IMDB with stochastic ternary gradient approximation, the observed performance gains are more modest. This may be attributed to suboptimal hyperparameter choices or to the increased noise introduced by the stochastic nature of the gradient estimates.
Furthermore, Fig. demonstrates that momentum effectively accelerates loss convergence, showcasing an additional benefit of its use. The observed gap between full-precision and ternary optimization is expected, quantization introduces inherent approximation error in both the parameters and the optimization trajectory. While the approximation error introduced by quantized models is unavoidable, it remains unclear whether improved algorithms can mitigate the degradation in convergence caused by quantized optimization.
Fine-TTiee Tine-Tuning LLMs with LoRA. The results for fine-tuning LLMs with LoRA (Table B further underscore the practical utility of our ternary momentum method. We observe that our approach generally achieves su- perior performance compared to optimizing ternary models with ternary gradients without momentum across various downstream tasks. This demonstrates that the stochastically approximated momentum effectively enhances optimization stability and performance, even when applied to large, pre-trained models.
Our method's inherent memory elficiency complements LoRA, which reduces the number of trainable param- eters. By minimizing the optimizer state footprint, the ternary momemommntum method enables fine-tuning of larger models under tight memory constraints. Combined with ternary gradients, achieves predomi- nantly ternary memory usage for both gradients and momentum, yielding substantial overall memory savings during training.
11

---

Published in Transactions on Machine Learning Research (03/2026)
Notably, observed minor performance degradation specifically for the WinoGrande dataset when using stochastic ternary approximation. We hypothesize this particular instance of degradation may be attributed to the inherent noise and adversarial nature of the WinoGrande dataset interacting unfavorably with the stochastic momentum. For other baselincs, specifically SST2 and COPA with Sparso-ZO, the more limited improvements may be explained by the aggressive stochasticity inherent the gradient approximation scheme.
Memory of optimizer states. Table 2 comprchensively details the memory required for optimizer states across various models. The results clearly demonstrate the drastic reduction of memory footprint compared to full-precision momentum optimizers. Specifically, our method is using only (og/(3)/32 = 5% for the first momenlum buffer compared to a full-precision momentum. While naturally requiring more memory than an optimizer completely devoid of momentum, the ternary momentum memory cost is demonstrably minimal for the performance gains provides, presenting an optimized tradc-off. This magnitude of memory savings becomes particularly critical for very large models, where memory constraints are often the primary training bottleneck.
Ternary momentum with other memory reduction methods. In Tables 4 and we report addi- tional memory reduction experiments, focusing on activation quantization and weight quantization of the pre-trained parameters. Across all tested settings, incorporating ternary momentum consistently improves accuracy. The effect is particularly pronounced on ImageNette, where aggressive memory reduction leads to substantial performance degradation, and the proposed method mitigates this loss. The results suggest that our method can be used as complementary approach for reducing memory consumption during training, alongside other quantization methods.
These findings demonstrate the ternary momentum method's position a highly effective and memory- efficient optimization strategy for large-scale machine learning, particularly beneficial in resource-constrained environments.
7 Conclusion, Limitation, and Future Work
In this work, introduce a ternary firt--momentum optimization algorithm, provide theoretical molivation for its design, and demonstrate its empirical potential. Our results show that optimizer states can retain meaningful information even under an extremely low bit-width sctting with ternary quantization.
While the proposed method demonstrates promising results, its practical applicability is currently limited by the lack of efficient and accurate ternary gradient approximation mechanisms that operate in fully quantized manner. Furthermore, using ternary values current hardware. To deploy our method on real edge devices chips, the gradient sign approximation must be fully quantized, and additional hardware-speciffc adjustments are required to enable forward (and, if necessary, backward) passes that align with the device's capabilities. Moreover, the model architecture, or context length, may need to be chosen carefully to satisfy memory constraints. Although our method shows benefits several settings, it is not intended as replacement for full-precision industrial-scale pre-training, which is beyond the scope of this work.
do not account for the practical overhead typically associated with
This work opens several directions for future research. One promising avenue is extending the approach to stochastic and quantized variants of second-moment optimizers like Adam (Kingma & Ba 2014 The algo- rithmic changes required for second moment with ternary gradient approximation are non-trivial due to the nature of powers of ones, i.e. (-1)²- = 1 = which limits the ability to weight parameters differently. Addi- tionally, combining low bit-width quantization with low-dimensional optimization techniques could further enhance memory and computational efficiency. Another interesting direction to explore adaptive masking strategies, where the update mask is sampled based on a dedicated saliency score per parameter or adjusted over time (see Paquette & Paquette (2021)) rather than being uniformly applied across parameters.
12

---

Published in Transactions on Machine Learning Research (03/2026)
References
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Commmication-cffcian sgd via gradient quantization and encoding. Advances in neural information processing systems, 30, 2017.
Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli, The convergence of sparsified gradient methods. Advances in Newral Information Processing Systems, 31, 2018.
Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, pp. 1-50, 2022.
Noga Bar and Raja Giryes. Zoqq: ZZrooooder quantized optimization. In ICASSP 2025- 2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficint fine-tuning for transformer- based masked language-models. In 60th Annual Meeting of the Association for Computational Linguistics, ACT. 2022, pp. 1-9. Association for Computational Linguistics (ACT), 2022.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Com- pressed optimisation for non-convex problems. In International conference on machine learning, pp. 560- 569. PMLR, 2018.
Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Konstantinos Parasyris, Jiancheng Liu, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order opti- mization for deep model training. In The Twelfth International Conference on Learning Represenlations, 2024.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. anXiv preprint Xrx: 06085, 2018.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural net- works with binary weights during propagations. Advances in neural information processing systems, 28, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprinl 160: 02830, 2016.
Tim Dettmers, Mike Lewis, Youncs Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8- matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318 30332, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: fummnn language technologies, volume 1 (long and short popers), pp. 4171-4186, 2019.
Dotan Di Castro, Omkar Joglekar, Shir Kozlovsky, Vladimir Tehuiev, and Michal Moshkovitz. Gradient- free neural network training on the odge. arXie preprint 10.09744. 2024.
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale, arXiv
preprint ar trXx:2010011 11929, 2020.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswany, and Dharmendra Modha. Learned step size quantization. In International Conference on Learning Representations, 2020.
URL http::///pppameeieee..ee////rrrr
Elias Frantar, Salch Ashkboos, Torsten Hocfler, and Dan-Adrian Alistarh. Optq:
Accurate post-training quantization for generative pre-trained transformers. In 11th International Conference on Learning Rep- resentations, 2023.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Saced Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Oplimization, 23(4):2341-233) 22668, 2013.
Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Kumar Jawanpuria, and Bamdev Mishra. SItrain: a sparse plus low rank approach for parameter and memory efficient pretraining. Advances in Neural Information Processing Systems, 37:1 11111182711188 118295, 2024a.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In ICLR, 2016.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-effiiient fino-tuning for large models: comprehensive survey. arxiv prepriul arXiv:2403. 14608, 2024b.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. In International Conference on Machine Learning, pp. 17554-17571. PMLR, 2024.
Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp.
PMIR, 2025.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficint transfer learning for nlp. In International conference machine learning, pp. 2790-27900 2799. PML.R, 2019.
Jeremy Howard. Imagenette: A smaller subset of 10 easily classified classes from imagenet. GitHlb, March 2019. URL
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient only inference. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2704 2713, 2018.
Wei Jiang, Sifan Yang, Wenhao Yang, and Lijun Zhang. Efficient sign-based optimization: Accelerating convergence via variance reduction. Tn The Thirty-eighth Annual Conference on Newral Information Pro- cessing Systems, 2024. URL http:///wnnneeeeeeeeeeeerrrradaa
Sai Pranceth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International conference on machine learning, pp. PMLR, 2019a
Sai Pranceth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error foodback fixes SignSGD and other gradient compression schemes. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2252-3261. PMLR, 09-15 Jun 2019b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ hbs/1412.6690 2014. URL https://api. somaaiiccoaaa wrg/CorpusIDD 6628106
Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems, 36:15136 15171, 2023.
Fengfu Li, Bin Liu, Xiaoxing Wang, Bo Zhang, and Junchi Yan. Ternary weight networks. orXiv preprint arXiv:1605.0477 2016.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Vladislav Lialin, Sherin Muckatira, Namrata. Shivagunde, and Anna Rumshiskk, ReloRA: High-rank training through low-rank updates. In The Twelfth International Conference on Learning Regresenlalions, 2024. URL https://openrevierr
Kaizhao Liang, Bo Liu, Lizhang Chen, and Qiang Liu. Memory-efficient 1lm training with online subspace descent. Advonces in Neural Information Processing Systems, 37:64412 64432, 2024.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the com- munication bandwidth for ristributed training. In International Conference on Learning Represenlations, 2018. URL
Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong- signsgd via zeroth-order oracle. In International conference learning representations, 2019.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092 28103, 2021.
Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter optimization method for large language models. Advances in Neural Information Processing Systems, 37:24926-24998 2024.
Iomut Vlad Modoranu, Mher Safaryan, Grigory Malinovsky, Eldar Kurtić, Thomas Robert, Peter Richtárik, and Dan Alistarh. Microadam: Accurate adaptive optimization with low space overhead and provable convergence. Advances in Neural Information Processing Systems, 37:1 43, 2024.
Yurii Nestcrov. A method for solving the convex programming problem with convergence rate o1/h²)) Proceedings of the USSR Academy of Sciences, 269:543-5/4 1983. URL httptps://ppi.ssmtant htt///csp..mmmmiiccccccooolla htgg/oorpusID:14511100
Yurii Nesterov. Introductory lectures on conver optimization: basic course, volume 87, Springer Science & Business Media, 2013.
Courtney Paquette and Elliot Paquette. Dynamics of stochastic momentum methods on large-scale, quadratic models. Advances in Neural Information Processing Systems, 34:9229 14:9229-9240, 2021.
Courtney Paquette, Kiwon Lee, Fabian Pedregosa, and Elliot Paquette. Sgd in the large: Average-case analysis, asymptotics, and stepsize criticality. In Conference on Learning Theory, pp. 3548 3626. PMLR, 2021.
Boris Polyak. Some methods of speeding up the convergence of iteration methods. User computational mathematics and mathematical physics, 4(5):1-17, 1964.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. TEEE, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European, conference on computer vision, pp. 542-542. Springer, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge al scale. Communicattins of the ACM, 64(9):99- 106, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. Tn Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Sebastian U Stich, Jean- Baptttt Cordonnier, and Martin Jaggi. Sparsified sgd with memory. Advances in neural information processing systems, 31, 2018.
15

---

Published in Transactions on Machine Learning Research (03/2026)
David Stutz, Nandhini Chandramoorthy, Matthias Hein, and Bernt Schicle. Bit error robustness for energy- efficient dnn accelerators. Proceedingg of Machine Learning and Systems, 3:569-500 598, 2021.
Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang. Momentum ensures convergence of signsgd under weaker assumptions. In International Conference on Machine Learning, pp. 33077 33099. PMLR, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumyy Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint orXiv:3307.09288 2023.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding sys- tems. Advances in neural information processing systems, 32, 2019a.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantiza- tion with mixed precision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4612-8621 2019b.
Sike Wang, Pan Zhou, Jia Li, and Hua Huang. 4-bit shampoo for memory-cfficiet network training. Aduances in Neural Information Processing Systems, 1222997-127007 2024.
Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication-effiicen distributed optimization. Advances in Neural Information Processing Systems, 31, 2018.
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. Advances in neural information processing systems, 30, 2017.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. In International Conference on Learning Representations, 2018.
Guangxuan Xiao, Lin, Mickael Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099. PMLR, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zero- quant: Efficient and affordable post-training quantization for large-scale transformers. Advances in neural information processing systems, 35:27165 68-27183, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint ar Xrv::2200.00 01068, 2022.
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yi Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficien Ilm fine- tuning: A benchmark. In International Conference on Machine Learning, pp. 59173-50 5919909 PMLR, 2024.
Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. In The Thirteenth International Conference Learning Representations, 2025. URL https://openreviert wot.:
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memorr--fficient IIm training by gradient low-rank projection. In International Conference on Machine Learning, pp. 61121-61143. PMLR, 2024.
Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, and Zheng Zhang. Quzo: Quantized zeroth-order fine-tuning for large language models. arXiv preprint ar Xiv:2500. 12946, 2025.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1rXi::1600 061 60. 2016.
Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. APOLLO: SiD-like memory, adamw-level performance. In Bighth Con- ference Machine Learning and Systems, 2025. URL
Philip Zmushko, Aleksandr Beznosikov, Martin Takáč, and Samuel Horváth. FRUGAL: Memory-efficient optimization by reducing state overhead for scalable training. In Forty-second International Conference on Machine Learning, 2025. URL http:://ppaaareeeee.eeeerrrrrras
A Experiment Details
ImageNettee VIT
IMDB Fully-Connected
LM Fine- Sine-Tuning w. LoRA
Batch Size
200
Batch Size
256
Batch size
16
FP LR
6 x 10
Epochs
50
Iterations
20000
WD
0.05
Layers
3
WD
0
Iterations
5000
Input dim
768
LoRA a
16
Embed. dim
512
Hidden dim
4096
LoRA r
8
Layers
12
Init. density 0.9
Init. density
0.9
Heads
16
Init. density
0.9
Figure 2: Hyperparameters used with different settings.
Hyperparameters Search. Results of grid search with multiple values of momentum coefficient B and initial learning rate no. The results show the benefits of the chosen hyperparameters.
85.46
85.4
85.33
85.29
Accuracy
85.21
B
Figure 3: Momentum coefficient, B, hyperparameter search.
Accuracy
85,46
85.35
84.89
83.78
0.25
0.5
0.75
Initial Learning Rate
Figure 4: Initial learning rate search.
17

---

Published in Transactions on Machine Learning Research (03/2026)
In Table G and Fig.
compare the inverso-time learning rate scheduler with the constant schedule 1/vi.
n/vl
n/b
80.79
80.95
85.57
Table 6: Accuracy results with different learning rate schedulers with IMDB and No Backward sign approx- imation. For constant we use = 0.1 for all other scheduler we use Y/ = 0.75.
0.70
w/vi
0.60
Loss
0.55
0.45
0.40
0.0
2,5
5.0
7.5
10.0
12.5
15.0
17.5
Step
Figure 5: Cross entropy test loss with different learning rate schedulers.
B
Additional Results
Additional comparisons with the use of our ternary momentum.
FP AdamW
No Backward
No Backward FP Momentum
No Backward
Ternary Momentum
Optimization parameters [#M]
38.9
0
19.9
19.9
Changes [#B]
585.06
0.19
195.21
19.69
Energy [J]
2.851
0.936
1.084
1.010
Table 7: We report the number of parameter changes summed with the optimizer-state changes, together with the corresponding expected energy consumption. For our method with momentum, the reported number of changes constitute an upper bound, since updates are driven by the momentum term and the exact number of zero updates cannot be directly approximated. In contrast, without momentum, the expected number of changes is explicitly controlled through deterministic masking, while momentum be interpreted as additional masking mechanism applied on top of this deterministic scheme. The approximated difference in energy consumption when using ternary momentum is small, while accuracy improves at the cost of a modest additional energy overhead.
C Proofs of Section 5
C.1 Proof of Theorem
Note that for all E [T]. since Q) = 1, it holds that et = 1. Hence,
0,: 0.-1. me,i, and 10t.: 0--1,5 1.
18

---

Published in Transactions on Machine Learning Research (03/2026)
160
deterministic
deterministic momentum
140
stochastic
stochustie-momn.onne
120
fp-agd
fp momentum
100
Loss
60
40
0
10
20
30
40
50
Step
Figure 6: Training loss curves (cross-entropy loss) on the IMDB dataset using deterministic and stochastic ternary gradient approximations. FP lines represent full-precision training with SGD, with and without momentum. Momentum accelerates convergence the ternary case, while in the full-precision case it improves the final training loss but does not noticeably accelerate convergence.
deterministic
deterministic cenemiiitti--mmmmetuum
stochastic
stochustic-mom.ane
Loss
0.5
0.4
0.3
10
20
30
40
50
Step
Figure 7: Test loss curves (cross-entropy) on the TMDB dataset using deterministic and stochastic ternary gradient approximations. The FP line represents full-precision training with AdamW. While in Fig. [ the training loss demonstrates large gap between FP and ternary optimization, the gap in test loss is smaller, explaining the small accuracy difference reported in Table
19

---

Published in Transactions on Machine Learning Research (03/2026)
Method
Params
Grads
Optim
Activation
Total
IMDB
FP
79.60
79.60
159.20
5.0
323.40
Ternary
3.94
3.94
0.00
5.0
12.89
+ Mom.
3.94
3.94
3.94
5.0
16.83
VIT
FP
149.20
149.20
298.40
21 620.0
22216.80
Ternary
7.46
7.46
0.00
21 620.0
21 634.92
Ternary+ Tern. Mom.
7.46
7.46
7.46
21 620.0
21 642.38
OPT1-36
FP
5335.04
6.1440
12.2880
1024.0
6377.47
Ternary
5324.80
0.3072
0.0000
1024.0
6349.11
+ Mom.
5324.80
0.3072
0.3072
1024.0
6349.41
Table 8: Memory Footprint Comparison (All values in MB). For IMDB our method show clear benefits, for other settings the benefits are shown in the optimization memory. Those setting are presented mainly to demonstrate the benefits in accuracy, see Tables and
Deterministic
Stochastic
No Backward
FP (with Adam)
87.3
No Momentum
86.75
85.27
83.04
Ternary Momentum (quant. after step)
76.88
85.47*
84.4*
Ternary Momentum (stochastic, ours)
86.86
85.28
85.57
Table 9: Performance comparison using non-stochastic updates and momentum on the IMDB dataset. Momentum is updated according to m, = -m,.-++0 (1- and the parameters are updated 0, = 0:-1 m, and then both momentum and parameters are quantized to lernary values using stochastic quantization. Results indicate that while accuracy gains are observed in some specific configurations, the benefits of non-stochastic momentum and updates inconsistent across gradient approximation. Furthermore, the observed training instability (* denotes settings where only converged seeds reported) suggests that additional regularization or optimization measures are required for reliable convergence.
Step (i): the good event. Fix i [d]. We define the following good event:
E A Yt T A + 1] e 1..1....,4 As - 1} s.t. Ctd,s = 0).
Fixing some E [ Ag + 11,
(m) Pr(e!") e =... 1+ = =1 3 (dT)).
where the last inequality follows from the definition of Ag. Performing a union bound over at most T windows, Pr(F.)) 2 5/d. Defining FG Nietal E; and performing union bound, Pr(Ec) 2 1-8.
The rest of the proof will follow deterministically,, conditioned on Eg.
Step (ii): distance from the minimizer. Let θ = [Fminn Tmax and let 0; A arg argminggce (.))) For any E [T), let (,(),) be the distance of or,; to the set of minimizers O L.c.,
D2(0,) A min 10,, - 071.
Our goal is to bound D()+.) < Ag for all t [Ag + Fmax Ymin. T7. Let [t] be the maximal iterate with (0,))) 1. To establish the existence of such T, first, note that conditioned on E,, there is S Ag with
20

---

Published in Transactions on Machine Learning Research (03/2026)
Ia., m. Second, since Iz.s sign(0.-1.1 ) (for all 8 from convexity in 1-d, m,, sign(0.-1, -0;) after which (0,)) D:(0++1),-.. must decrease by in each step until reaching some s' with (2(0,) < (as all the gradients are aligned with the momentum until the iterates cross the set of minimizers). According to the domain size, it must happen after at most Ag + Tmax
Tmin < steps.
Now, t < As, then (..) S ((+))) + Ag. Otherwise, conditioned on Ei, there some s [T 1, Asl with ma sign(0.-1,i 05) (for any 0 0)), and D;(O,), Di(0++),.... must decrease by 1 in cach step until reaching below the 1 threshold, either contradicting the maximality of enforcing (0.))) & D.(0,-)) Ag. Hence, D.,),) S Ag.
Hence, we have established bounds for each i E [d] that hold with probability at least 1 - (the probability of Ec), such that
1 111,433
[(OT) min L()) S ege
C.2 Proof of Theorem
Here we prove a more general version of Theoreme supports then follows immediately by substituting γ = for t = I...,,T
suppots an arbitrary sequence "h...,, Tpr Theorem
Theorem 3. Let C Id + R be an element- wise L-smooth function lower bounded by some L E R. Let Bg E zd and consider the sequence (0+) generated by Algorithm | with porameters l.. T'in -00 Tmax 0 (0, 1), some .......77 .rr > 0, and the determånistic signed gradient as TernGrad. Then it holds that
C(0)) -
(1+
Σ m El (((O--))
E1=1 11
(1- 3) Ei=1"h t1
2(1 - 3)2
In order to prove the theorem, we will use the following lemma. Its proof follows.
Lemma 1. Let f E Rd, A E-1.1.0,11- L = (L,) E R4, and let L Rd + R be differentiable function satisfying
Vx,y Rd
(C(y) - (C(x) vC(2) (v - x))1 s Li - xz)²
Then it holds that
- - s 311
Proof. By the smoothness assumption,
((0.) - L(0:-1) S VL((0-1) (0. - 0z-1) +
Σ
14(0,,- - 1,)2
2
i=1
=-VC(0-11) (e) (0) mm)) +
1
[e(e
(0)
m,
2
Summing for t = 1..... T, replacing L(0r) 2 L" and rearranging,
T
T E|VC(( (el (0) 2 mm)) < ((b)) - L ΣΣ
m. m.E((epmm.)))
1=1
By the law of total expectation, L[V (e (0) m:)] = 1, V ((0,-1) m.). and noting that (0) m.,5 E {-1,0, 1}, El o m, s ne Hence,
T ||
LE -m] S L(bb)
L"
m--
2
1=1
21

---

Published in Transactions on Machine Learning Research (03/2026)
Next will focus the E[V2(((-1) mel-
= - VL(0--11) - (mi 3c)]
(VL(0--1) Dr (2(0--1)11)
= -
(c, (m)) ② (m,-1 - 9c))]
EUIIV (m4-1 - 9t)!
00--1;14-1,,8 Dt independent from eg e(()) (VL(0t-1) -: [V
[ - +) (V m
[ - > [ -m--1 (
m4-1 ((V vL((,-2)) -m,-1]
where the last incquality follows by Hölder's incquality and Lemma and using the initialization min 0,
Applying the incquality until t =
E[V(((0-1) mi) 2 E[3VL(01-2) - 14-1 + -1- 3) - -1
2 E[(1 3) Σ (Z( - - -11 Σ gt-++1]
k=1
k=1
3/31
E[(1 3) (B) 12 1)
1 3
E((1 (3]1--)]][((-))) 3,911
1-8
Thus,
C(0%) L-
(1 (1+53))144
(1- 3) Σ 11
2(1 101--9- 3)2
11= 1/
C.3 Proof of Lemma 1
Proof. Let sign A) V((x)). Hence, from the smoothness assumption,
(()) VL(x) s 2 D
((z -s- 1 Σ
a)
Li(s--AN - A.)2 >
1=1
and
C(x) - C(x - v (-A) < 5
Summing the inequalities,
(VL(x A) VL(x)) 8 < $31
D Synthetic experiments
In this section, we study synthetic linear regression tasks with both separable and non-separable objectives. These experiments are designed to illustrate the behavior of our method under mini-batch optimization, where the gradient sign is affected by sampling noise.
Our setup uses dataset X Raxd containing n 1024 samples of dimension d 128. Ground-trute predictions E R are generated by first sampling ground-truth solution 0* Iddx1 and then setting
22

---

Published in Transactions on Machine Learning Research (03/2026)
y X0 The loss of parameter vector 0 RR associated with row z2 of X is given by L,(0) (0.2; ya))
We consider the following two tasks:
Separable objectives. To construct a multi-sample separable objective, we proceed as follows. The matrix X is constructed have orthonormal columns by taking the Q matrix from a QR dccomposition of random matrix with i.i.d. normal entries. To account for different coordinate scalings, multiply Q by /n/d E Rdxd, where is sampled from exponential dis- tribution with unit scale and the square root is applied element-wise In this case, the objective satisfies
1 - (0 (0- 0-) x 00)
((0) A (,(
- 2n
|X0 1X00-x011--
2n
i=1
1 2d
(0 0 diag(s) 0*) 2d
s; 02)²
To match Theorem [ we use Tmin and Tmax 1, and apply clipping. The initial weights are sampled uniformly from {-1,0, 0, 13 while the ground-truth vector sampled uniformly from []1
General linear regression. In this sctting, the entries of are sampled independently from a normal distribution and scaled by 1/vd. To match Theorem 2 we Tmin -oo and Tmax 00, and perform no clipping. The initial weights sampled uniformly from {-3, 2, 1, while the ground-truth vector is sampled uniformly from [_3.
1,
Optimization is performed using parameters 8 = 0.9 and V = 0.75/t, matching the setup in Section We run the algorithm for T = 2048 optimization steps with varying mini-batch sizes. All experiments are conducted using ternary gradients (obtained by applying the sign operator the mini-batch gradient)).
Loss Iterations
Loss Iterations
0.50
TFM Buch Sizr
4.0
TEM Bush Sioe 16
Siez
Beac
3.5
TEM
7FM Pull (1926
TRM Buch (100
0.40
3.0
0.35
2.5
Loss
Loss
0.30
2.0
0.25
1.5
0.20
1.0
0.15
0.5
500
1000
1500
2000
0
500
1000
1500
2000
Iteration
(a) Separable objective with ternary weights.
(b) General linear regression without clipping.
Figure 8: Loss of Ternary First Momentum (Algorithum) 1 on synthetic linear regression with different batch sizes.
Fig. g illustrates the convergence behavior of Algorithm [ under different batch sizes, which induce varying levels of noise in the ternary gradients. Full-batch optimization yields the best performance, with consistent improvement over the iterations considered. Nevertheless, we observe decrease in loss across iterations even when using mini-batches. Tn the separable case, batch size of 256 achieves performance comparable
23

---

Published in Transactions on Machine Learning Research (03/2026)
to full batch, whereas it degrades more noticeably in the general linear regression setting. Smaller batch sizes of 64 and especially 16 lead to substantial degradation in performance.
These observations indicate that gradient noise plays a crucial role in the optimization of Algorithm highlighting the importance stable ternary optimization.
24


Reviews:
[{"review_id": "me1jgZNkv2", "reviewer": "Reviewer_nAXh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a simple ternary momentum mechanism that maintains a ternary first-moment state and applies stochastic updates whose expectation mimics classical momentum whle avoiding FP optimizer state. It argues that this yields substantial theoretical reductions in optimizer-state memory and provides convergence to a neighborhood guarantees under idealized assumptions with exact sign gradients. Empirical results suggest that ternary momentum can improve stability and performance over no-momentum baselines in quantized training settings albeit the experiments primarily illustrate feasibility rather than tightly validating the theory.\n\nStrengths\n- Good, simple optimizer idea. Intuative and lightweight way to reintroduce momentum behavior while keeping the state ternary.\n- Theoretical analysis with nontrivial guarantees\n- Empirical breadth coudl be improved but evaluates across multiple tasks/settings (from scratch and finetuning), showing the method can be competitive relative to simple baselines.\n- A relatively clear systems motivation focusing on optimizer related memory (a bottleneck in large-scale training).\n\nWeaknesses\n- Theory vs. practice gap. Theory assumes access to exact sign/ternary gradient information and idealized storage, while several experimental gradient approximations may rely on FP components or overhead (this seems standard in the field but remains a key issue for full impact of the papers ideas).\n- Baseline/attribution clarity: comparisons and “following prior work” framing would benefit from more explicit mapping of baselines to specific prior algorithms and fuller metric alignment.\n- Limited ablations/robustness reporting: sensitivity to factors like batch size/gradient noise and variability across runs is not deeply explored in the current presentation.", "claims_and_evidence_yn": "No", "claims_explanation": "Overall, while I think the paper proposes an interesting idea and provides relevant theory, I find the current paper suffers from an underdeveloped experimental evaluation, lack of deep insights into the (practical) properties of the proposed algorithm, a gap between theoretical assumptions and practical implementation and thus practical relevance. Addressing the major concerns below would be necessary to provide enough insights to backup the general claims of relevance and correctness. \n\n\n1. Redundancy and structure (Sections 1-4) - Moderate concern: \nWhile the introduction is generally clear, I feel it contains substantial redundancy across Sections 1-4. In particular, the three long summary bullet points largely restate issues already discussed earlier in the text, resulting in an unnecessarily verbose presentation. The introduction would benefit from restructuring and consolidation; it feels overly long relative to its informational content.\n\n2. Lack of a synthetic experiment validating theory and demonstrating properties - Major concern: \nThe paper presents non-trivial theoretical results, yet misses an opputunity to provide a (potentially synthetic) optimization experiment designed to validate these results under the stated assumptions. A controlled synthetic experiment would be helpful to demonstrate that the algorithm behaves as predicted by the theory (see also 2b) and to clarify the relevance of the theoretical analysis to the proposed method. Without this, the theory remains largely disconnected from the empirical evaluation (e.g. the theory assumes access to determistic gradients, but the experiment do not).\n\n2. Batch size and gradient noise are insufficiently explored - Major concern: \nTernary and sign-based optimization methods are well known to be sensitive to gradient noise, making batch size a critical factor. Although briefly mentioned, this issue is neither analyzed nor experimentally evaluated. The absence of a systematic discussion or ablation significantly weakens the empirical section. Targeted experiments (e.g., low-dimensional or synthetic settings) woudl be very helpful to assess stability and behavior across batch sizes.\n\n3. Weak justification of experimental use cases - Major concern \nI did not find the experimental choices convincingly motivated specifically for studying ternary momentum/ternary optimization. The paper often relies on “following prior work” (e.g., Di Castro et al.) rather than clearly stating what each setting is intended to test or why it is particularly diagnostic for the proposed method (e.g., memory constraints, sign noise, stability). As a result, the experimental section feels under-contextualized and would benefit from clearer motivation and framing.\n\n4. Baseline positioning and comparison to prior work - Major concern: \nAs far as I can tell, the baselines are not clearly and explicitly mapped to specific prior methods/papers, making it difficult to assess novelty and progress relative to existing literature. Explicitly identifying which algorithms or papers each baseline corresponds to is necessary for easy comparison. \n\n5. Theory–practice mismatch and memory claims - Major concern: \nThe narrative surrounding the memory efficiency claims is a bit confusing to me. The theoretical analysis assumes idealized storage (e.g., log2(3) bits per parameter with no overhead), whereas the experimental setups necessarily has memory overhead and in some cases even rely on full-precision gradients. Even highly optimized implementations would require at least two bits as far as I know. While the paper acknowledges some of these limitations, the implications are not sufficiently reflected in the framing of the results, which appear closer to simulations of an idealized regime than realizable systems - perhaps the authors can help me to better understand this slightly unusual approach (as an outsider to the ternary research area).\n\n6. Missing variability and statistical reporting - Moderate concern: \nTables 1 and 3 report single accuracy values without any measure of variability (e.g., standard deviation, confidence intervals, or multiple runs). Given the modest effect sizes and the stochastic nature of the methods, this omission makes it difficult to assess robustness or statistical significance.\n\n7. Omission of SparseZO in Table 1 - Moderate concern: \nSparseZO is discussed in the paper but is not included in Table 1 (only Table 3), without explanation. This omission is confusing and I recommend clarifying and ideally including it.\n\n8. Incomplete alignment with Di Castro et al.'s metrics - Moderate/Major concern : \nThe paper claims to follow the experimental design of Di Castro et al., yet omits several metrics used in that work (e.g., number of updates and other efficiency-related measures). To me, this weakens the comparability of results and makes the “following” claim questionable. Either the framing should be adjusted, or idealy the missing metrics should be included.\n\n\n\n9. List of minor comments:\n - p2, mid $r_{max=1}$-> $r_{max}=1$\n - log2(3) perhaps spell out where this comes from \n - p4. I’d strongly suggest providing references inline for the claims made in “Optimization without momentum” paragraph. \n - p5, last paragraph. “motivate” -> “motivated”\n - p9: “that is possible acquired in an efficient way.” -> wording\n - Table 3: Please include the actual metric and preferably also if higher is better or lower is better.\n - Table 3: For consistency, I’d suggest having the same number of decimals for the three fine-tuning tasks.", "audience_yn": "Yes", "audience_explanation": "The paper addresses memory efficient optimization via ternary momentum, a topic relevant to readers interested in quantized training, optimization methods, and efficient learning. Its combination of algorithmic ideas, theoretical analysis, and empirical evaluation would - subject to the suggested corrections - be of interest to a subset of TMLR’s audience, particularly those working on low-precision and resource-constrained training.", "requested_changes": "See the list of concers above (items listed as Major are items I feel the authors need to address)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BT7iHOGxMk", "reviewer": "Reviewer_5N4S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary of Contributions:\nThis paper introduces a stochastic ternary momentum mechanism for training neural networks under extreme quantization, where parameters, gradients, and optimizer states are restricted to the ternary set {-1,0,+1}. The method replaces continuous momentum with a coordinate-wise Bernoulli masking rule that, in expectation, recovers the behavior of heavy-ball momentum while using only integer-valued operations. The authors provide convergence results for both convex and smooth non-convex objectives, and present experiments on IMDB, ImageNette, and ternary LoRA finetuning of large language models. Across these settings, ternary momentum consistently outperforms no-momentum baselines and approaches the performance of full-precision momentum with significantly reduced optimizer memory.\n\nKey strengths:\n1. The paper proposes a clean and intuitive discrete momentum mechanism that maintains the expected behavior of classical momentum while operating entirely in the ternary domain.\n2. The update rule requires only integer arithmetic and Bernoulli masks, making it conceptually appealing for low-precision or hardware-constrained training environments.\n3. Theoretical results cover both convex and smooth non-convex scenarios, providing justification that the proposed stochastic ternary updates remain stable and converge to approximate minima or stationary points.\n4. The empirical evaluation spans multiple architectures and tasks, including MLPs, ViTs with ternary components, and ternary LoRA finetuning of LLMs, demonstrating consistent improvements over no-momentum baselines and competitive performance relative to full-precision momentum.\n\nKey weaknesses:\n1. Although optimizer memory is a primary motivation, the paper does not provide detailed system-level measurements (e.g., end-to-end memory usage or runtime behavior), so the practical impact remains somewhat unclear.\n2. Experimental settings are relatively small in scale; results on industrial-scale or pretraining-scale workloads would strengthen the claims about practicality.\n3. The theoretical assumptions (e.g., separability or smoothness) do not fully capture modern deep neural network training dynamics, leaving a gap between the theoretical guarantees and real-world practice.\n4. The method is not compared against stronger low-precision optimizer baselines (such as 8-bit or 4-bit Adam), which makes it harder to position ternary momentum within the broader landscape of quantized optimization methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims are reasonably well supported. The ternary momentum update is clearly specified, and its connection to heavy-ball momentum in expectation is mathematically justified. The convergence results, under standard smoothness assumptions, are consistent with the proposed algorithm. Experiments on IMDB, ImageNette, and ternary LoRA finetuning show consistent gains over no-momentum baselines and competitive performance relative to full-precision momentum, supporting the core claims. Some aspects (e.g., large-scale system measurements and comparisons to stronger low-precision optimizers) are less thorough, but this does not undermine the primary conclusions.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a focused but relevant problem in low-precision and memory-efficient training, an area that has an active community within TMLR’s readership. Researchers working on quantized optimization, hardware-aware training algorithms, or resource-constrained deep learning would likely be interested in the proposed ternary momentum mechanism and its theoretical and empirical findings.", "requested_changes": "1. Provide more detailed reporting of the optimizer memory savings in practice. Since memory efficiency is a primary motivation, adding clearer measurements (e.g., total training memory usage with and without ternary momentum) would strengthen the empirical section.\n\n2. Include a brief comparison or discussion relating the proposed method to existing low-precision optimizers such as 8-bit or 4-bit Adam. This would help situate ternary momentum more clearly within the broader landscape of memory-efficient optimization.\n\n3. Clarify the experimental setup for the ternary gradient approximations (e.g., TernGrad, sparse ZO, No-Backward). A short explanation of how each surrogate gradient is computed would make the experiments easier to interpret.\n\n4. Add a brief discussion about limitations, such as the scalability of the method to very large-scale pretraining settings or potential issues with highly noisy ternary gradients. This would improve clarity without requiring additional experiments.", "broader_impact_concerns": "This work focuses on an optimization algorithm for low-precision training and does not raise clear ethical or societal risks. The method is technical in nature and does not involve sensitive data, deployment decisions, or applications with direct human impact. A short statement clarifying this, along with noting that more efficient training may indirectly reduce energy usage, would be sufficient. No major broader impact concerns arise from the current submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jm3YZLxLWs", "reviewer": "Reviewer_adFL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an optimization approach, based on sampling Bernoulli masks, where the values of both the parameters and a momentum state are kept ternary. The motivation is training on resource-constrained devices.\nThe paper presents experiments that show some gain compared to omitting the momentum ingredient from the optimizer, and some theoretical results under simplifying assumptions.", "claims_and_evidence_yn": "No", "claims_explanation": "Following are my comments regarding the proposed method.\n\n1. What is the precision of the model's activations?\nThe memory-efficient motivation of the paper suggests that it should be taken into account, but this aspect seems to be overlooked.\n\n2. It is not clear to me how to draw Bernoulli masks of arbitrary real p \\in (0,1) (namely, beta and eta_t in the context of the paper) without high precision computation or high computational complexity (e.g., loop) per entry of the Bernoulli mask. Please elaborate on this.  \nAs the masks are computed per optimization update, what is precisely the complexity gain compared to quantizing the parameters after each update?\n\n\n3. The assumption in setup 1 of the theoretical analysis, namely that the loss function is separable with respect to the parameters, is not practical for nonlinear models. This should be emphasized.  \nThe assumption in setup 2 of the theoretical analysis, namely that no clamping is done, leads to the fact that each entry of theta_t can have any positive/negative integer value. This is in contrast with the motivation for ternary parameters, and should be emphasized as well.\n\n4. Please report training time and present training loss curves for the proposed approach with and without momentum and for plain full precision training.\n\n5. I would also like to see comparison with non-sampling based optimizers where the parameters/momentum are quantized to ternary values after each step.\n\n6. In the fine-tuning experiments you use pertained networks and tune only LoRA. I would like to see an examination of the approach after the pretrained models are quantized.  \nOtherwise, what is the storage gain during training/inference? You will still need to store many full precision parameters/activations. Present the parameter count for these setups (as done in Table 1 for other setups).\n\n\n7. Given the memory-efficient motivation of the work, it would be interesting to see the performance of the models when the activations are being quantized as well. How do they perform compared to methods that take the quantization of both the weights and the activations into account during training?\n\n\n8. In some setups (not that few, in both Tables 1 and 3) there is only minor/negligible gain from using the ternary momentum compared to not using it. Elaborate on configurations of the setups under which the gain is higher/lower.", "audience_yn": "Yes", "audience_explanation": "The work may be relevant to audience interested in training networks on resource-limited devices.", "requested_changes": "Please address the above comments.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)