Arbitrary-Order Block SignSGD for Memory-Efficient LLM Fine-Tuning
Authors: Yijie Zhou
Shi Pu
Venue: ICLR 2026 Poster (2026)
Keywords: Block-Coordinate Optimization
SignSGD
Large Language Models (LLMs)
Memory-Efficient Fine-Tuning
Primary area: optimization

Abstract: We propose \textbf{ABSignSGD}, a block‑coordinate variant of sign-based descent with flexible block selection that enables memory‑ and runtime‑efficient full‑parameter fine‑tuning of large language models. We present a unified convergence analysis under mild conditions, covering both the base method and a \textit{majority‑vote} extension for distributed training. The latter improves communication efficiency by aggregating only gradient signs rather than averaging full gradients. Experiments on \textcolor{blue}{Qwen3‑8B, Llama3-8B, and Qwen3-32B}, spanning mathematical reasoning and general instruction‑following tasks, show that ABSignSGD converges faster per iteration and delivers superior downstream performance while reducing both runtime and memory usage compared to existing methods. Ablation studies further indicate that the memoryless sign-based update naturally complements block‑wise updates, explaining the method’s strong empirical performance.

Published as a conference paper at ICLR 2026
ARBITRARY-ORDER BLOCK SIGNSGD FOR MEMORY-EFHICIENT LLM FINE-TUNING
Yijie Zhou & Shi Pu"
The Chinese University of Hong Kong (Shenzhen) rijiezhoue mnkccuhk edu. cn, pushi edu.cn
ABSTRACT
We propose ABSignSGD, a block-coordinate variant of sign-based descent with flexible block selection that enables memory- and runtime-efficient full- paraameter fine-tuning large language models. We present a unified convergence analysis under mild conditions, covering both the base method and a majority-vore exten- sion for distributed training. The latter improves communication efficiency by ag- gregating only gradient signs rather than averaging full gradients. Experiments on Qwen3-8B, Llama3-8B, and Qwen3-32B, spanning mathematical reasoning and general instruction-followe
tasks, show that ABSignSGD converges faster per iteration and delivers superior downstream performance while reducing both run- time and memory usage compared to existing methods. Ablation studies further indicate that the memoryless sign-based update naturally complements block-wise updates, explaining the method's strong empirical performance.
INTRODUCTION
Large Language Models (LLMs) achieve state-of-the-art results in reasoning, dialogue, and code generation (Achiam al. 2023), but specialized applications still require task-specific adapta- tion (Ding et al. (2023). Fully retraining is prohibitively expensive, making fine-tuning the prac- tical route for domains such as biomedical text (Singhal et al., (023) or legal text (Chalkidis al., 2020) and for aligning behavior like multilingual support. Yet even fine-tuning imposes heavy GPU memory demands (Han et al.. 2024). motivating methods that reduce memory and runtime without degrading performance. Existing strategies to reduce the memory footprint of LLM fine-tuning span multiple directions. System-level techniques such as quantization (Dettmers et al. 2022 2023 Lin
et al. 2023) modify the numerical representation of model parameters or activations (e.g., storing
weights in lower precision), while offloading (Ren et al.) 2021 Rajbhandari et al. 2021 changes
the storage location of tensors within the hardware memory hierarchy (c.g., moving optimizer states to CPU or NVMe). Complementary to these system-level approaches, algorithmic optimizers form the second major axis of memory-efficien fine-tuning. Among them, zeroth-order methods (Zhang et al. 2024 Liu et al.) 2024) represent an important line of work, eliminating backward passes and achieving inference-leve memory usage. However, their slow convergence often limits practical applicability for LLM fine-tuning.
In this work, we focus on first-order algorithmic methods that aim reduce memory and runtime without sacrificing performance. Existing methods can be broadly grouped into three families.
(i) Parameter-efficient fine-tuning (PEFT). These methods reduce memory usage by training only a small set of additional parameters while keeping the base model frozen. Representative techniques include prefix-tumii tuning (Li Liang] (022), prompt-tuning (Lester al., 202I), and adapter architec- tures (Houlsby et al. 2019 Pfeifter et al. (021). The most widely adopted strategy is Low-Rank Adaptation (LoRA) (Hu et al., (021). which reparameterizes weight matrices using low-rank fac- tors. PEFT reduces memory and makes fine-tuning feasible on limited hardware, yet usually yields lower performance than full- pallameereter training.
(ii) Low-rank projection for full-parameter training. Methods such as GaLore (Zhao et al.) 2024 Fira (Chen et al. 2024), Flora (Hao et al. 2024), and GoLore (He et al. 2024) cut optimizer
*Corresponding author. The code is available https: t/ubb
thub. hwtuub.com//// com www.c//c/....o ji jzc.cn////cccc AHSI gnSGD

---

Published as a conference paper at ICLR 2026
memory by projecting gradients into low-rank subspace via SVD or faster/cheaper random pro- jections, sometimes applied intermittently to reduce cost. They preserve fall-parameter updates but may face: (a) performance gaps compared to strong baselines like AdamW, (b) incompatibility with standard gradient accumulation under layerwise updates for maximal memory saving, and (c) slow runtime when frequent costly decompositions are needed.
(iit) Block-wise optimization. This approach updates only a subset of parameters per iteration. For example, BAdam (Luo al. 2024) combines block-coordinate updates with Adam, saving memory by storing optimizer states only for the active block and reducing runtime by halting backpropagation that block. However, Adam's 's dependence on first- and aecond-moment estimates conflicts with block switching, requiring frequent state resets that, as shown in our ablations, degrade convergence relative to Adam with full-model updates.
Motivated by these trade-offs, we propose Arbitrary-order Block SignSGD (ABSignSGD), which combines block-coordinate updates with sign-based descent. Leveraging the simplicity of SignSGD, our method offers greater memory savings and extra runtime gains from arbitrary-order updates while maintaining competitive performance. We further introduce ABSignSGD-MV, communicaiion-efficeent data-paralle variant that transmits only block gradient signs (1 bit per co- ordinate) cutting communication cost by 960x (with 30 blocks) over standard DDP (Paszke et al.) 2019) without harming convergence. As shown in Table [ ABSignSGD achieves the lowest mem- ory overhead, smallest communication budget, and fastest runtime.
Table I: Memory-efficient optimizer comparison. M: model parameters (billions); r: rank for low- rank methods; m: weight matrix dimension (assumed square); N: layers (for layer-wise update) or blocks. Memory/communication in GB. ABSignSGD achieves the lowest memory and communi- cation costs and the fastest runtime. See Appendix A for derivations and details.
Memory Overhead!
Comm Budget
Gradient Accum.
Runtime
Method
Speedup
ABSignSGD
M BN 16M N 36
M 8N 4M
BAdam
N
LoRA
SMr
m
m
GaLore
8M
12Mr
4Mf
X
X
N
Apollo
8.M
+
SMT
4Mr
X
N
m
m.
Excludes the 2M GB half-precision weights stored by all methods. For low-rank projection methods, original papers omit communication budgets; sending full gradients costs 1M GB-orders magnitude higher than others-and even low-rank gradients remain comparable to LoRA and far above ABSignSGD. Double checkmark denotes additional runtime speedup from arbitrary-order block updates.
1.1 CONTRIBUTIONS
(i) We introduce ABSignSGD, a block updates, allowing to tailor the update policy for maximal efficiency (c.g., depth-biased updates; see Contribution (iii). This design delivers substantial memory and runtime savings while preserving competitive convergence and downstream performance. We further extend the method to distributed training with ABSignSGI-MMV which aggregates only gradient signs via majority vote, thereby achieving extreme communication efficiency.
variant of SignSGD that enables arbitrary-order
(ii) We establish theoretical convergence guarantees under mild assumptions, providing a unified analysis for ABSignSGD and ABSignSGDD-WV. Specifically, they achieve O under arbitrary block selection schemes given bounded update intervals.
convergence
VR
(iii) We introduce depth-biased update that prioritizes deeper layers, providing runtime speedup without sacrificing performance. Extensive experiments on fine-tuning Qwen3-8B and Llama3-8B for mathematical reasoning and cntruction-followinn show that ABSignSGD achieves the lowest memory footprint, fastest rumtime, and superior downstream performance among memory- optimizers. A targeted ablation study further pinpoints the factors driving its effectiveness.
2

---

Published as a conference paper at ICLR 2026
Next, we formalize the fine-tuning setting and present the ABSignSGD algorithm.
ALGORITHM DESIGN
2.1 PROBLEM SETTING
We consider the general unconstrained optimization problem
min f(()). =ERd
In the case of LLM fine-tuning, the objective fx) Ex, E), where F is the loss function and D is the data distribution. And one uses batch gradient (zz) to estimate ff(()).
Notation: [n]] ;= 21,2...., n2 denotes the set of integers from 1 to Let P = { TN) be partition of [d] into N blocks; that is O π Ø for i j and U T [d]. We write (ITz -TnN) where Ir, E RRd, collects the coordinates of indexed by я; For notation convenience, we denote the i-th block by 2 == TT,, while 2j denotes the j-th coordinate x for j E [d]. For memory analysis, assume the model has M billion parameters.
2.2 PROPOSED METHOD: ABSIGNSGD
The most widely used optimizer for LLMs Adam (Kingma & Ba, 2013), which stores first-order and second-order momentum for each parameter. The update rule scales the learning rate adaptively based on these estimates. While effective, maintaining these optimizer states nearly triples the mem- ory required for parameters, posing significant bottleneck. In contrast, SignSGD (Bernstein al., 2018) is a stateless optimizer that discards gradient magnitude entirely, relying only on the sign of the gradient for parameter updates:
k+1 = x a. (ggn(g(""))
This memory-efficient approach remains competitive because its dynamics share similarities with Adam (Kunstner et al., 2023 2024). Sign-based principles have been successfully incorporated into modern optimizers for LLMs, such as Lion (Chen et al., 2023), which leverages the sign of the momentum term. Crucially, recent empirical evaluations (Zhao et al. 2025) show that sign-based methods are comparable to AdamW in both performance and hyperparameter robustness. However, previous applications have focused on full-model updates; combining the efficiency of SignSGD with block-updaa framework remains unexplored.
We adopt SignSGD with coook-onod updates to solve Problem I At iteration k the algorithm selects block Fi, and updates its coordinates using stochastic gradient signs:
X++1
sign(gt, (x*)),
(2)
where 9h, (x*) is the block gradient estimate. We refer to update 2 as Arbitrary-order Block- Coordinate SignSGI (ABSignSGID. Algorithm presents its complete procedure, along with communication- effficient variant (see Section 2.2.2). The name reflects its tolerance for flexible block selection: each block only needs to be updated least once every B steps (see Section [.T]. This property enables customized update rules that speed up training without degrading performance. As one example, we propose a depth-biased selection strategy (see Section 4) that updates deeper layers more frequently, yielding additional runtime savings while maintaining strong accuracy.
2.2.1 ABSIGNSGD MEMORY- AND EFFICIEFT
M GB of memory (exclud- ABSignSGD offers notable memory savings, requiring only 2M EN ing activations) for training. In contrast, Adam with mixed-precision training requires 18M GB of memory. The savings mostly stem from storing optimizer states only for the active block. For com- parison, BAdam (Luo et al. 2024), Adam with block updates, consumes 2M + 16M GB memory. ABSignSGD achieves further savings by (i) avoiding moment storage and (ii) only using signs for updates. For an 8B model with N 36, this yields extra 3.5 GB memory reduction.
3

---

Published as a conference paper at ICLR 2026
Second, block-update method, ABSignSGD also enjoys backpropagation runtime savings when blocks align with neural network layers. As observed in (Luo et al. 2024 computing the gra- dient for given layer (block) allows the backward pass to terminate at that layer; updating only the final layer thus eliminates nearly all backpropagation cost. BAdam uses cyclic block updates, yielding about 50% reduction in backpropagation time. ABSignSGD extends this advantage by allowing arbitrary (see Assumption 3.3 block updates, enabling deeper layers to be updated more frequently and further reducing backward-pass computation. In the extreme, updating the last layer for B (N - 1) consecutive iterations before updating each remaining layer once reduces the av- erage backpropagation time approximately N-1 for large In practice, we develop strategy
(see Section that delivers an additional R 20% runtime saving without impairing performance.
Algorithm ABSignSGD and ABSignSGD-MV (local view)
Require: Initial point x0 IR; partition { Øy}; ]: stepsize a block rule. I: for k == 0, 1. ......do
2: Select block in
3: if single-agent then
ABSignSGD
4:
f Sigu(gm,, (x*))
5: else multi-agent
> ABSignSGD-MV
6:
f sign Eソ= sign (ott (a*)))
Aggregate signs with majority vote
7:
end if
8:
.kl +- rkk Ri
mig
~
9:
Iog s++1 f- skk for all i ix
10:
end for
2.2.2 A COMMUNICATION-EFFICII EXTENSION
We further extend ABSignSGD to the data-parallel setting, yielding extremely communication- efficient multi-agent variant: Arbitrary-rdder Block-Coordinate SignSGD with Majority Vote (ABSignSGID-MV) This method inherits all the memory and runtime benefits of ABSignSGD while substantially reducing inter-agent communication.
In ABSignSGD-MY, agents compute stochastic gradients in parallel. At iteration k, all agents update the same block Ti, according to
k+1 J sign Σ sign (oi (a*))
where oi (x*) denotes the block stochastic gradient computed by agent j. Unlike the standard approach, which applies sign(E)= oh (a*)), ABSignSGI-MVV first takes the sign of each agent's block gradient, then aggregates these signs via majority vote.
With this design, each iteration requires agents to exchange only the signs of the block gradient, amounting to just bit per coordinate, rather than full prllisreeision values (32 bits per coordinate). For N 30 blocks, this reduces communication volume by 960x relative to the standard PyTorch implementation (Paszke et al. 2019). Under the same setting, the reduction is 32x compared BAdam, and 4.5x compared to LoRA with rank 8 and internal dimension m 4096. Moreover, the MV estimator is asymptotically more robust under heavy-tailed noise, as indicated by Theorem 3.5
Di wwwribbudDDaaPPa..
2.2.3 PRACTICAL CONSIDERATIONS COMPATIBILITY AND LIMITATIONS
ABSignSGD inherently applies extreme gradient quantization by reducing each gradient coordinate to a single bit, making any additional gradient quantization redundant. However, weight quantization and activation quantization remain fully compatible and can be combined with ABSignSGD to further reduce memory footprint and runtime. Similarly, offloading weights or activations is compatible, though optimizer-state offloading is irrelevant due to ABSignSGD's statelessness.
4

---

Published as a conference paper at ICLR 2026
Convergence of sign-based updates requires the sign-agreement probability to exceed 0.5. Prior work Safaryan Richtárik (202I) shows divergence on toy problems when this condition fails, so such methods may underperform under certain regimes, e.g. with extremely small batch sizes. Ap- pendix G.T shows that ABSignSGD is indeed more sensitive to noise (i.e., decreasing batch sizes) than the baseline. Crucially, however, it avoids breakdown and maintains a faster convergence rate than BAdam even under the extreme noise of batch size 4. Furthermore, in our main experiments (Section 4 with a relatively small batch size of 16, ABSignSGD demonstrates substantial perfor- mance lead. The observed sensitivity can be partially attributed to the absense of orthogonal mech- anisms (e.g.. momentum and adaptive learning rate) rather than the sign update itself. A promising future direction is to incorporate such stateful techniques via system-level offloading. Since block- coordinate methods require only the active block's ' state each iteration, the I/O bandwidth demand is minimal. This allows for the offoading of optimizer states to enable momentum-base variance reduction without compromising the method's ultra-low memory footprint or runtime efficiency.
THEORETICAL ANALYSIS
Having introduced the algorithm and its distributed extension, we now present theoretical analysis, establishing convergence guarantees for both under mild conditions within a unified framework
3.1 ASSUMPTIONS
Assumption 3.1 (L-smoothness and Lower Boundedness). The function f is L-smooth and lower bounded, i.e. fVv ffly)l L||= vlla and f(x) 2 f", Vx.
Assumption 3.2. For each element in the gradient estimator g(x), its sign aligns with that of the ground truth gradient V f() with a probability larger than 1/2. Namely,
p((x) = P(sign(a((x)) = sign( (ign(V..(())) 2'
Vr E Idd and all i [d].
Assumption 3.2 also referred to as the Success Probability Bound (SPB) in (Safaryan Richtárik (qquivvllll. the sign-agreement probability bound) has several sufficient conditions. One such condition holds when the gradient noise is unimodal and symmetric, property observed in many deep learning tasks (Bernstein al. 2018). Another guarantec arises if the gradient noise vari- ance satisfies the element-wise bound of (x) S Cs of (x) and the mini-batch size exceeds max; 4-
Assumption 3.3 (Bounded Update Interval). There exists a positive integer B such that, for every t, each block index j E [N] is selected at least once within the interval [t..... t+ B 1111. 1}. Equivalently, for every and every j [N], there exists some E (t....,t 1
- 1] with ig =
This assumption affords the algorithm substantial flexibility in block selection, which, analyzed in Section 2.2.1 and corroborated by later results, can further reduce runtime.
3.2 CONVERGENCE RESULTS
We establish convergence theorems for both ABSignSGD and its Majority Vote variant under this customized alignment norm, following the formulation in (Safaryan & Richtárik 2001)..
Definition 3.1 (Alignment norm). Let g(x) E Rdd and (wi(x)) be alignment weights with 0 S wx(z)= < 1. Define the alignment norm
llg())]lv==
Σ wx(xx) (x)(
For ABSignSGD, use wi(x)) 2p;(x) - 1. where pz(x) is from Assumption ABSigHSGD-MV, (x(x) = 21 (p(())); 1, where [(n + 1)/2] and I(5:.)) the regu-
3.2 For
larized incomplete beta function.
Although termed a "norm." " ||. |Il is not true mathematical norm; it is weighted & measure in which each coordinate's contribution is scaled by its likelihood of sign agreement with the true
5

---

Published as a conference paper at ICLR 2026
gradient. Coordinates with higher (z(z) receive greater weight, reflecting their higher expected contribution to descent and their importance for convergence guarantees. In the single-agent case (n 1). the ABSignSGD-MV weights reduce to those of ABSignSGD.
We now present a unified convergence guarantee for both ABSignSGD and its Majority Vote (MV) variant under this alignment-norm framework. The result applies to both single- and multi-agent settings, differing only in the definition of the alignment weights.
Theorem 3.4 (Unified Convergence of ABSignSGID and ABSignSGD-MVV, Given Assumption 3.1 to 3,3 and assuming identical block size, ABSignSGI) and ABSignSGD-MV converge as follows:
Σ Lk=o
E B S _(aa) - + aLLd + 2N
N+1
N
2 ))
K
aK
with different weights wi as defined in Definition
When the smoothness and lower-boundedness SPB, and bounded-update interval conditions hold, and block sizes are identical, both ABSignSGD and its Majority Vote (MV) variant converge up to steady-state term determined by the step size, the block-update interval, and the number of blocks, with convergence defined in the alignment norm. The single- and multi-agen settings are unified within common proof framework, differing only in the definition of the alignment weights. With proper step size aggressive reductions in memory, runtime, and, via MV, communication cost.
/V we arrive ( (1/KK) convergence rate, which persists despite
Although MV discards gradient magnitude information, it match averaging in certain regimes, particularly under heavy-tailed noise (a common phenomenon in deep learning (Gurbuzbalaban misaligned outliers, as it ignores gradient magnitudes and thus avoids amplifying their influence. This is validated by the following result.
outperform gradient
al., (02T)). This robustness stems from its resistance confidently
Theorem 3.5 (Informal Statement of Theorem
Under heavy-tailed noise, Majority Vote is asymptotically a superior sign estimator compared to aggregation by arithmetic mean.
Together, these results establish that ABSignSGD and its MV variant retain strong convergence guarantees while offering robustness under heavy-tailed noise, providing the theoretical foundation for the empirical studies in Section
4 EXPERIMENTS
We now evaluate ABSignSGD against leading memory-efficient fine-tuning optimizers, measuring memory usage, runtime, convergence speed, and downstream performance.
4.1 EXPERIMENTAL SETUP
Tasks and datasets. We fine-tune QWEN3-88 -BB on OpenMathInstruct-2 (Toshniwal et al.) 2024) (50K samples) for mathematical reasoning and Stanford- (Taori al. (2223) (35K samples) for general instruction following. The 8B scale is the primary focus, as it represents the most com- case for full-parameter fine-tuning. Each dataset is fine-tuned separately and evaluated on task-specific benchmarks: mataaiiauuiioo -uatinntaaeeessssssss- (Gou & Zhang 2025) for math rea- soning, and MT-Bench with a GPT-5 judge via FastChat (Zheng et al., 2023) for instruction following, following the official protocol. Results on LLAMA3- -8BB-BBBBBBBB show similar trends and are re- ported in Appendix E] Scalability is further validated by fine-tuning QWEN3-322 on the math task, where consistent performance gains are observed; these results are provided in Appendix E
Baselines. We compare A ABSIGNSGD with leading memory-efficient optimizers from three fami- lies: low-rank adaptation (LoRA), low-rank projection (GALORE, APOLLO), and block-coordinate (BADAM) methods. To narrow the comparison, we exclude any method that inherently uses orthog- onal techniques, like quantization//ffloaddiin
as part of its core design, such as QLoRA (Dettmers) et al. 2023). We also exclude sign methods that rely primarily on momentum buffers, such as Lion Chen et al. 2023), since these ineur substantially higher memory costs unless paired with orthogonal techniques. Hyperparameters follow each method's 's official implementation where pos- sible, with all other settings matched. We adopt gradient checkpointing while disabling gradient

---

Published as a conference paper at ICLR 2026
accumulation (even when possible) to avoid runtime bias. No offloadingg quantization, other saving techniques are applied. Further details are deferred to Appendix D
Block partitioning and selection strategy. The model is partitioned into N blocks at the layer level, where each Transformer layer (including attention and FFN modules) constitutes a single block (resulting in N = 36 for Qwen3-8B). Building on this layer-wise partition, our framework supports arbitrary block selection. As discussed in Section 2.2.1 prioritizing deeper layers yields measurable runtime gains. We also hypothesize that this may help mitigate catastrophic forgetting, as shallower layers tend to encode more general features (Howard & Ruder 2018). We leave a thorough empirical validation of this effect future work. Moreover, we seek to avoid repeatedly updating the same block in succession, which risks premature convergence to poor local minima. To balance these considerations, we adopt an event-driven depth-biased update rule.
To implement this strategy while satisfying the bounded update interval assumption (Assump- tion 3.3), an event-driven update rule is adopted. Each block is assigned a fixed "virtual update cost" Tg. which serves as hyperparameter to control the relative update frequency. A "next-ready" virtual timestamp T, is maintained for each block, initialized to T.. At each iteration k. the algorithm selects the block with the minimum timestamp, i arg min, T, performs the update, and incre- ments the timestamp: Tú, <- Ti, + Ti.- This scheme ensures that every block is updated at least once within fixed interval B (the derivation of B and concrete execution trace provided in Appendix D.2). The costs are defined as T, = N +( - i + 1), where is the block index (1 being shallowest) and is bias coefficient. In the experiments, c 10 is used, ensuring deeper blocks are updated more frequently. Note that ABSignSGD and its MV variant remain fully synchronous.
4.2 MEMORY FOOTPRINT AND RUNTIME
Table reports the peak reserved GPU memory during training and the wall-clock runtime. AB- SignSGD attains the smallest peak memory usage at 20.29 GB, about 2 GB lower than LoRA and Apollo, and nearly 3 GB lower than BAdam and GaLore. In terms of runtime, ABSignSGD is R200 faster than BAdam and roughly twice as fast as LoRA, with even larger gains over projection-base methods. The speedup relative to BAdam indicates that the depth-biase block-selection scheme delivers additional runtime benefits beyond those of block updates alone. Overall, the empirical evi- dence reinforces ABSignSGD's suitability for large-model fine-tuning under tight memory budgets.
Table 2: Maximum reserved GPU memory and runtime for fine-tuning Qwen3-8B on 50K OpenMathInstruct-2 samples for 3 epochs. ABSignSGD achieves the lowest memory footprint and fastest runtime among all compared methods.
Metric
ABSignSGD
LoRA
GaLore
BAdam
Apollo
Mem. Reserved (GB)
20.29
22.54
23.47
23.19
22.58
Runtime (h)
2.66
5.51
12.77
3.32
6.64
4.3 CONVERGENCE SPEED
We present convergence curves for fine-tuning Qwen3-8B on the OpenMathInstruct-22 dataset, using each method's optimal step size determined via grid search (details in Appendix D). As shown in [igure[ ABSignSGD reduces training loss more quickly than all baselines, both in terms training token count and wall-clock time.
This improvement arises from combining sign-hased updates, which deliver strong per-iteration progress, with depth-biased update rule that reduces runtime. Together, these features allow AB- SignSGD to achieve lower loss in fewer updates and finish training sooner, making it well-suited when both convergence speed and runtime are critical.
The distributed Majority Vote (MV) variant is further analyzed in Figure Although not empha- sized in the caption, F (Left) includes the convergence curve for ABSignSGD-M. running on four agents with an identical global batch size to the single-node version. The MV variant tracks the non-MV baseline closely, confirming that the sign-based aggregation preserves convergence speed.
7

---

Published as a conference paper at ICLR 2026
LeRA
LoRA
ABSignSGa-BSS #C16
BAdamm Apollo
ABSign
Apolo ABSignSti
ABSign ScDgnsooooaoe #$128
Traning
ABSigniCDDW
Traiiing
Ansign -81-8516
ABSignSGD
Apsigr SCD-8N--61222
ABSignS MN-8564
Tokcen
Time (5)
iter.
Figure 1: Training loss for Qwen3-8B on OpenMathInstruutt2. Left Middle: Convergence com- parison against baselines in terms of tokens and wall-clock time (ABSignSGD-MV is excluded for time comparison as uses multiple agents). ABSignSGI achieves the fastest reduction in both metrics. Right: Robustness of ABSignSGD-MV. The curves show loss iterations the number of agents increases from to 32 (fixing local batch size). The MV variant closely tracks the single- agent baseline regardless of the agent count, demonstrating high scalability.
Figure ( (Right) further highlights the method's robustness to scaling. With the local batch size fixed at 4, the number of agents is increased from 1 to 32 (scaling the global batch size from 4 to 128). ABSignSGD-MV consistently tracks the single-agent baseline across settings, confirming the predictions of Theorem 3.5 and demonstrating that the method remains stable even the number of voting agents increases. Further results on a fixed global batch size deferred to Appendix G.2
4.4 DOWNSTREAM TASKS PERFORMANCE
To assess the practical impact of faster convergence, we evaluated the fine-tuned models on down- stream tasks. The results show that ABSignSGD's optimization efficiency translates into stronger generalization across both specialized and general domains.
Mathematical Reasoning. On diverse mathematical benchmarks, Qwen3-8B fine-tuned with AB- SignSGD achieves substantial and consistent accuracy gains over all baselines (Figure [ These improvements highlight the method's ability to produce models that handle complex, specialized reasoning more effectively, benefiting from faster and more stable optimization.
Base
ABSignSGO
LoRA
Galore
BAdam
Apollo
Accuracy
46
ago
Figure 2: Accuracy on diverse mathematical reasoning benchmarks for Qwen3-8B fine-tuned on OpenMathInstruct22 with different optimizers. ABSignSGD consistently outperforms basclines across tasks and achieves 6% accuracy improvement against the second best, indicating that faster convergence during training translates into stronger task-level generalization.
General Instruction-Folllvee
On MT-Bench (Table 3). ABSignSGD attains the highest overall average score (8) across eight categories, leading five and remaining highly competitive the rest. This breadth of strength shows that the method enhances not only domain-specific reasoning but also broad, multi-skill instruction-follwiii capabilities.
Across specialized and general tasks, ABSignSGD consistently surpasses strong memory-efficient baselines. By combining sagn-based updates with depth-biased block selection, it accelerates con- vergence without loss of accuracy, producing high-performing models with limited resources.
8

---

Published as a conference paper at ICLR 2026
Table 3:
MT-Bench scores (higher is better) for Qwen3-8B fine-tuned on Stanford-Alpaca with different optimizers. ABSignSGD attains the highest overall average and leads in five categories.
Method f Writing
Roleplay
Reasoning
Math
Coding
Extraction
STEM
Humanities
Ave.
4.64
5.04
5.12
4.93
5,07
4.66
4.88
4.59
4.87
Ours
5.77
6.31
6.39
6.56
6.06
6.12
6.04
6.20
6.18
LoRA
5.82
5.81
5.34
5.87
5.37
5.77
5,48
5.66
5.64
GaLore
5.80
5.11
5.32
5.02
5,58
4.74
6.10
5.15
5.48
BAdam
5.79
5.75
6.08
5.21
6.11
5.54
5.25
6.00
5.72
Apollo
5.63
5.43
5,51
5,56
5.11
5.45
5.08
5.43
5.40
4.5
ABLATION STUDY
The preceding results establish ABSignSGD as a powerful, efficient method for fine-tuning LLMs. To identify the key design components behind its success, we perform thorough ablation study. Unless stated otherwise, all ablations use the smaller Qwen3-1.7B model with the same training configuration as the main experiments, reducing computational cost while preserving the relative behavior of variants. Additional results are provided in Appendix F.3
Justification for SignSGD. To isolate the effect of the optimizer, compare ABSignSGI with BAdam and BlockSGD (the block-coordinate extension of SGD that updates a single parameter block per iteration) under identical block-selection scheme with their optimal learning rates, and also include their base optimizers (SignSGD, Adam, and SGD) for reference.
As shown in Figure 3 (Left & Middle), combining a block update strategy with Adam impairs convergence, likely because its adaptive step size relies on past gradient history that block switching erases. In contrast, SGD and SignSGD are not affected as much because they only depend on current gradient information, making them inherently more compatible with the block-coordinate approach.
BlockSCD
Adam 54D SignSCD
Black5CD
Adam S6D
DS-1
BB-1
0.4
ABSignSDD
ABSigrSDD
DS 10
0B5
BAdbm
BAdbm
SignSGD
D5 50
DB:10
Taaiing 1:50
DS-200
0B-20
Taising
0.2
Edown
4807
0.0
8.50
Figure 3: Convergence curves from two ablation experiments on fine-tuning Qwen3-1.7B with OpenMathInstruct-2. Left & Middle: Token- and time-wise training loss for core optimizers (SGD, SignSGD, Adam) and their block-update counterparts under an identical block-selection scheme. Adam degrades under block switching, whereas memoryless methods (SGD and SignSGD) remain compatible; SignSGD further outperforms SGD, yielding faster iteration- wise convergence. Right: ABSignSGD under different update rules exhibits similar convergence.
Figure Left illustrates why SignSGD converges in our setting: the iign-agreemet probability dis- tribution is sharply skewed toward 1. Only around 1% of coordinates have agreement probabilities below 0.5, while for large fraction the sign is almost always correct. This closely matches the method's ''s core assumption and supports its stable convergence behavior.
We now turn to explaining why sign-based methods outperform SGD (Figure Two factors are central. i) regularization: prior work shows SignSGD induces an Adam-like regularization ef- fect that is beneficial under heavy class imbalance, which we quantify via the token-class frequency distribution. For example, in the Stanford-Arl Allaccaaaaaaaaaa dataset the most frequent token appears roughly 10x more than the second most frequent (Figure Middle) ii) noise resilience: the relative gradi- ent noise magnitude remains consistently > 1 and can exceed 10 (Figure Right); such noise impede SGD but is naturally damped by sign-bases based updates.
On block update rule. We validate the necessity of our flexible block selection scheme by com- paring (1) depth- depth-biased biased updates (DB-c), where is the relative backprop. time ratio; (2) deep-to- shallow selection (DS-K), K,), deterministic scheme updating from the deepest to the shallowest layer

---

Published as a conference paper at ICLR 2026
0040
Histagram
Probability
Hestogram Noise Signal
Cum Lumulovve Probabiity
Parameter Count
0.5
Para meteer Count
Nolse 0.00%
Sign Agreement Probabilit
Class index (Trquencyy rank)
10"
11a
15 10
Log Reilative Magniludee
11
Figure 4: Factors explaining ABSignSGD's success (Qwen3-1.7B on Stanford-All -Alpaca; details in F.. Left & Right: Metries derived from comparing full gradients with multiple mini-batch gradients over the same parameters. The left panel shows the sign-agreement probability histogram, while the right shows the relative gradient noise magnitude histogram. Middle: Token-class fre- quency distribution, revealing severe class imbalance the training set.
with flock-switching interval K; and (3) uniform random selection (UR). Consistent with (Luo et al. 2024). Figure 3 Right shows that, within moderate hyperparameter range, the selection choice minimally affects convergence. Appendix G.3 further verifies that the choice scheme has negligible impact downstream generalization. Consequently, the depth-biased method's primary advantage efficiency- by updating shallower layers more often, achieves substantial runtime savings without performance loss.
CONCLUSION
We introduced ABSignSGD, an arbitrary-order block-coordinate extension SignSGD for efficient full- aala-martetter tne-tuning of large language models. The framework supports diverse block selec- tion strategies, with the depth-biased scheme as one effective example, and includes majority-vote (MV) variant for data- mata-paalllll training. We provide unified theoretical convergence guarantees for both methods under the SPB assumption. Empirical results show that ABSignSGD surpasses strong baselines in convergence speed and downstream accuracy while reducing memory footprint and wall-clock runtime.
REPRODUCIBILITY STATEMENT
We have taken steps to ensure the reproducibility of our results. All implementation details, includ- ing hyperparameters, learning rate schedules, optimizer configurations, dataset splits, evaluation protocols, and hardware specifications, are provided Appendix [ We also rely on official im- plementations for baseline methods and evaluation methods ensure fairness and comparability. Together, these details should enable independent researchers to replicate our experimental findings without ambiguity.
ETHICS STATEMENT
This work focuses on developing memory- and runtime-efficient optimization algorithms for fine- tuning large language models. All experiments were conducted publicly available datasets (e.g., OpenMathInstructt2, Stanford-Alpace with open-source models (e.g. Qwen3, Llama3) that do not contain personally identifiable information. We do not foresee direct risks of harm arising from methodology. Nevertheless, as with any system that improves the efficiency of LLM fine-tuning, there exists the potential for downstream misuse, including generating harmful biased content. We emphasize that our contributions are intended to advance research in optimization and efficiency, and encourage responsible and ethical use of the resulting models and techniques.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported in part by the National Natural Science Foundation of China under Grant 62373316 and Grant 62336005, and in part by the 1+I+1 CUHK-CUHHK(Z--GDSTCCC Joint Collab- oration Fund under Grant 2025A0505000049.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Arizzadenesheli, and Animashree Anandkumar. signSGD:: Compressed optimisation for non-convex problems. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 560-569, PMLR, 10-15 Jul 2018. URL
www.mmr..e.esssseeeeeee v80 w80/baa htt//bennsteiing 8a. html
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androut- sopoulos. LEGAL-BERT: The muppets straight out of law school. In Trevor Cohn, Yulan He. and Yang Liu (eds.). Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2898 2904, Online, November 2020. Association for Computational Linguistics. doi: 10. 111618661/22200 httt://0000.iinnn---ee2222222.2. Findings emn 2617
URL httph:tpp..... //aaaaaaaahhlooyyy........ org/2020.
Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang- Fira: Can we achieve full-rank training of ilms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V Le. Symbolic discovery of optimization algorithms, Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.in8(): 8-bit matrix multiplication for transformers at scale, arXiv preprint arXiv:2208.07333) 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. arXiv preprint 2rXiv:2305.14314 2023.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature machine intelligence, 5(3):220-233) 2023.
Paul Embrechts, Claudia Klüppelberg, and Thomas Mikosch. Modelling extremal events: for insur- ance and finance, volume 33. Springer Science & Business Media, 2013.
Zhibin Gou and Yue Zhang. math-evaluatton-aee marking Ilms mathematical reasoning tasks.
A simple toolkit for bench- com. ZubinGou/
mathheva 1 uuat on ha rness 2025. MIT License.
Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference Machine Learning, pp. 0964-3975. PMLR, 2021.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficien fine-tuning for large models: A comprehensive survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL/btttss:/ppperreee
net htt://ooommmmmdidsseese
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters secretly gradient compressors, 2024. WttLlhhtts://arrataa org/ abw//aa/555 2202 03293
Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. arXiv preprint 10Xiv:2410. /289, 2024.
11

---

Published as a conference paper at ICLR 2026
Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American statistical association, 58(301):13-30 1963.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-cffiiient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research. pp. 2790-2799, PMLR, 09-15 Jun 2019. URL https: hatps::/ppoceecdnnss mmr.
press/v97/houlee shy.elsssyylaa. html
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprini arXiv: 0101100001.66,.. 2018
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
//arxiv.crg/aaaaa.iv. /2t.:22266... 09685
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua uni-trie de/
Bengio and Yann LeCun (eds.), ICLR (Poster), 2015. URL
Sbb/conf//ccr//ccr200000 html #iimmlKiin #KingmaB14
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise the main factor behind the gap between sgd and adam on transformers, but sign descent might be. arXiv preprint arXiv:2304.1300 2023.
Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. Advances in Neural Information Processing Systems, 37:30106-30145 2024.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Maric-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 22.11863111222..com wwap:pmmmmmmmmmmmmmmm.mmmm URL
wtg. emnlp ma main.
243/
Xiang Li and Percy Liang. Prefix-tuning: Optimizing pretrained generative models with prefix- continuous prompts. In Proceedings of the 2021 Conference on Empirical Methods in Natural
Language Processing (EMNLP 2021), 2021. http://ppss//aaaaathhllooo org/2021. lemmlpppee in. 727
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. AWQ: Activation-aware weight quantization for L.LM compression and acceleration. arXiv preprint 1.Xi::2306000978, 2023.
Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsich, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order I1m fine-tuning, 2024. URL https:
farxiv.org/abs.. 2402. 15751
Qijun Luo, Hengxu Yu, and Xiao Li.
Badam: A memory efficient full parameter optimization method for large language models. arXiv preprint arXiv:2404.02827, 2024. URL https://
arxiv. org. abs/ / 2404.04420 02827
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019. URL
httpp::/pytorchh org/)
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: -on-destruetive task composition for transfer learning, 2021. URL https: ///ariv... htr/abb/20055.00 00247
12

---

Published as a conference paper at ICLR 2026
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. ZeRO- Infiniinfn Breaking the GPU memory wall for extreme scale deep learning. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2021.
Jie Ren, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. ZeRO-Offlod:: Democratizing billion-scale model training. In USENIX Annual Technical Conference (ATC), 2021.
Mher Safaryan and Peter Richtárik. Stochastic sign descent methods: New algorithms and better theory, 2021. URL http://///aaal.ooorr////////...
Karan Singhal, Shekoofch Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):122-180 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An costructionfoollowing llama model. https: //gtt/b/tttbb./.. com/ tat 1ab/ st anfoandnoaaeeee allpaalpaaaaaa 2023.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction 2024.
data. arXiv preprint arXiv:2410.011
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang. Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient Ilm fine-tuning: A benchmark, 2024.
URL
wttp://..rr//b/////222..2 11592
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient IIm training by gradient low-rank projection, 2024. URL.
stttps://axxiv.org//ssss
Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstruct- ing what makes a good optimizer for language models, 2025. URL httts::/aaxii..oo abs/ 2407 07972
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging auuueee- with mt-bench and chatbot arena, 2023.
13

---

Published as a conference paper at ICLR 2026
A MEMORY AND COMMUNICATION ANALYSIS
We first analyze the memory efficiency of different optimizers from two perspectives: block-wise storage during the update of a single block and the global memory footprint across the entire model.
A.1 BLOCK-WISE STORAGE
Table Block-wise storage requirements (in parameter counts) different methods during up- date of a single block. The table reports the number of parameters that must be stored for weights, gradients, optimizer states, and projection matrices. Precision (e.g.. FP16 vs. FP32) is not consid- ered here; actual memory usage can be derived by multiplying these counts by the storage size per parameter.
Method
Weight
Gradient
Opt. State
Proj. Matrix
ABSignSGID
mm
mn
BAdam
17
mn
2mn
LoRA
mE + MF + ni
MT + T2
2mr + 2nr
GaLore
mn
mn + nr
2nr
mr
Apollo
mn
m7 + T2P
2nr
Table reports the storage requirements incurred when updating one block of parameters. The values expressed in terms of the number of parameters that must be stored (weights, gradients, optimizer states, and projection matrices). These counts are independent of numerical precision; actual memory usage in bytes can be obtained by multiplying the counts by the storage size per parameter (e.g., 2 bytes for FP16, 4 bytes for FP32).
We assume block weight W e R with denoting the low-rank dimension and m > For all low-rank methods, we assume they use Adam as the base optimizer.
ABSignSGD requires only the block weights and their gradients, with no optimizer states, since updates rely solely on gradient signs.
BAdam stores both gradients and optimizer states just as Adam.
LoRA reduces gradient storage by restricting updates to low-rank matrices. Therefore, only store two low-rank gradient matrices and their corresponding optimizer states.
GaLore and Apollo requires the gradient to conduct low-rank projection. Moreover, Ga- Lore needs to store a projection matrix, while Apollo uses random matrices for projection.
At first glance, low-rank methods appear advantageous because they store only compact matrices. However, block-based methods like ABSignSGID scale more favorably, since they only maintain variables for the currently updated block, whereas low-rank methods must maintain auxiliary states across all blocks.
A.2 GLOBAL MEMORY FOOTPRINT
Table{]exteedds the analysis to the full model (excluding activations), assuming M-B parameters and rectangular weights of dimension m. We assume the low-rank projection methods to use layer- wise update, as they otherwise require much more memory because they need to store full gradients. N represents the number of blocks layers for different optimizers. Here we incorporate precision assumptions: we assume model weights to be stored in half-precision, while gradients, optimizer states, and other parameters maintained in full precision. The total memory footprints
ABBignnGD: weights in FP16 plus cached block sign gradients:
M
2M
+
8N
14

---

Published as a conference paper at ICLR 2026
BAdam: weights in FP16 plus block gradients and optimizer states in FP32 (and an FP32 copy the block weight for the update):
16M
2M
N
LoRA:
copies of the low-ranl weights and low-rank matrices in FP16 plus, matrices, gradients, and optimizer states in FP32:
36Mr
2M
m
GaLore: weights in FP16, block gradients and projected gradients in FP32, plus projec- tion/auxiliary terms:
SM
12Mr
4Mr
2M +
N
Apollo: weights in FP16, block gradients and projected gradients in FP32:
SM N
8Mr
4Mr
2M
+
+
mN
Overall, ABSIGNSGD achieves the lightest footprint by eliminating FP32 optimizer states and ing sign-only cached information, while block-wise storage ensures that only the currently updated block contributes transient overhead.
Table 5: Global memory footprint comparison of different methods.
Method
Weight
Gradient
Opt. State
Other
Total
ABSignSGD
2M
2M
M 16M
BAdam
2M
4M
iM
SMM
2M
LoRA
2M
12Mr
SMr
16Mr
2M
+
36Mr
GaLore
2M
4Rt
4M
+
4MT
Ki r
4Mr
2M +
8M
12X0
4Mr
Apollo
2M
LM
+
&Xir
2M +
8M
mN
m.
A.3 COMMUNICATION BUDGET
In distributed implementations, the communication budget is determined by the amount of gradient information that must be exchanged across workers each synchronization step.
For ABSIGNSGD-MY BADAM, and LoRA, the communication cost is directly proportional to their gradient storage requirements in Table 5 That is, each worker must transmit the same number of parameters as it stores locally for gradients, ensuring consistency across replicas.
In contrast, GALORE and APOLLO require transmitting the full gradient of size 4M maintain mathematical equivalence during distributed updates. This communication volume is orders of mag- nitude larger than that of the other methods, and can quickly become the dominant bottleneck in multi-GPU or multi-node training.
While might be possible reduce this overhead by transmitting only the low-rank projected such modification would alter the update rule and falls outside the scope of mN
gradient of size 4Mr this paper.
B CONVERGENCE PROOF
B.1 AUXILIARY LEMMAS
Lemma B.I. The alignment norm || || satisfies the following conditions
15

---

Published as a conference paper at ICLR 2026
Triangular inequality: ||2 yllv 2 || - llvllar Var, y-
Upper bounded by l norm: ||~|| S ||0/112, Va.
Proof. Note that
| lIulla- = waa -wlaraa we 19
S w/ [2; Vil
= 1| y LLv
and
|
We (ara
|x:|]
|0]|1
Lemma B.2, For all x E TR.,, I-norm and 12-norm have the following relation
||>|| I vd[[=ll2.
B.2 PROOF OF THEOREM 3.4
The proof is an extension of the one in (Safaryan & Richtárik 2021) to the block-update scheme. We first provide a block descent lemma, which arises naturally after the definition of the customized norm || lle-
Lemma B.3. [Block descent lemma] Given Assumption 3.1 and 3.2 sizes, the updates of ABSignSGD and ABSignSGD-MV satisfy
and assuming identical block
ΣK- Vi, Mln
1xx
adL. aK
(4)
2N
Proof. We first prove for case of ABSignSGD. By L-smoothness of the objective function f()),
f(xh+1) - f(x*))(*)>))))))))()))) $ ff(=")" (x
x²) 1| |1
k+1
V. sign( (I*) )) + a²di, L
2
Note that by the definition of sign-alignment probabilities Pi: we get
V (x sign(gh, )x] = (x*"" Elsiin((n (x*)))
vif(a ((i()))))nn)) (v(((()((())() - (((="))sign(V..((a))
iEM
Σ (2p((x))
NERi
I/V llv.
Combining the above and take full expectation, we arrive at
V (a*>) f(x²) f(a adiz L
(5)
a
For ABSignSGD-MV, we define the majority of block gradient signs
Tr. (a Σ sign (gf (z*)).
16

---

Published as a conference paper at ICLR 2026
Update B
can be rewritten
The a. sign(ina(*)))
1x
Following an identical reasoning as in ABSignSGD,
sign( + addie L
f(aht1) - =( -avin((a")
(6)
2
By Lemma 13 in (Safaryan & Richtárik 2021 we have
E E (x*)) sign(Gin (a
Vin VinJ (x*))²
Elsign (x*))]
Σ
[nn E (isinn (x²)) Vin fx((()))
Issd Vi f(xx) )
Σ Isied Viy((" )/0
(2T(())) 1, 1, 1)
Combining equation (6) with the above, we reached equation (
Sum over equation 5 over K iterations, we have
Via f (x*)
f(x0) - 5r + k=
aL
EK-1
K-1
di
k=
K
aK
2K
1(x0) 8r adL
aK
2N
Lemma B.3
states that the average of block gradient alignment norm converges under proper step size. Now, we analyze the relation between the block gradient and the full gradient to show the convergence of the latter.
Lemma B.4. Given identical assumptions in Lemma B.3 and Assumption SignSGD and ABSignSGD-MM -V satisfy
3.3 the updates of AB-
N +
|Vf('') Σ [v.f(=')l Lad(B
2
Proof. From Assumption 3.3 each block updated at least once for every B iterations. Therefore, pick N steps in which each step updates different block and we have
k+B-1
Σ
[L.f(')
N Σ I(Vvll('')ls
tES,
where S, (k...., ls + B 1), |S
N, and (ihes [N].
17

---

Published as a conference paper at ICLR 2026
We compare the RHS to the full-gradee graddeed at step k as follows
f E fV''l(')
K
(IVIS((**) -1 1
(IVif((**) vef(o
(ValIViS(at) - Vis(s 1)(2
(VTIIs((>>) - ([(ctter-))12
Σ -
where the updating block at k t is the i-th block.
Note that the second last inequality above is necessary, as the difference between æ² and - crosses multiple blocks, so
[V - -i )12 S Illaf - 1i k+h 1122
does not hold. Note that
h+4 2 2 Σ
Σ Vd.Lil=*> - - S
x² 112
N
j=k
k+ 2
a
N
j=k
N La (t--1)
1=1
Block updates at iteration 1: + f, - 1. However, there is only one block updating at each iteration. The worst is when all blocks update once within the last N iterations in the [k: k + B 1] window. Therefore,
N (t.
N-1 1) S
Σ
(B - N +
3=0
NIB N +1
2
Now bound the staleness error, or the distance traveled between the steps
kth
||
- 112
|
++
xa1l2 1|2
t=k
k+r,
= Σ a /
t=k
d
< aB
N'
18

---

Published as a conference paper at ICLR 2026
Combining the above, arrived at the conclusion.
Theorem 3.4 is straightforward combination of Lemma B.3 and B.4
Sum equation over KB iterations, we have
K
(K-1)B
N+1
VV IIl
Σ
2
Combine the above with equation
we have
11 11B E V
+ +LBdd
K
K
[(x)) -S + BBL 2
N
aK
2N 2
C ROBUSTNESS OF MAJORITY VOTING
Safaryan & Richtárik (2021) states that the Majority Vote (MV) estimator's eгrоr rate converges exponentially to O. But we are not sure how the MV estimate compared to averaging the gradient. Here we prove that under heavy-tailed noise, the MV estimator is asymptotically infinitely more accurate than the standard Summation (or Averaging) estimator.
C.I PROBLEM SETTING
With a slight abuse of notation, we consider the problem of estimating the sign of a true signal f e R from set of M independent observations. Each observation generated from signal-plus-noic model:
9i f G2 for i . M,
where { are i.idd. random noise variables with E[e,] = 0. We assume each observation provides weak but better-than-random signal about the sign of f. This captured by the per- worker accuracy
= P(sign(g)) sign(f)) 1/2.
We analyze and compare two sign estimators:
1. Majority Vote (MV): The sign estimated by aggregating the signs of the individual observations. Let sggn(g.) E (1,,+1}.
fMv sign ΣS S
2. Summation (SUM): The sign is estimated from the sum of the raw observations.
fsum sign Σ
Without loss of generality, let f 0. The error probabilities are then PMV(M) P S, s 0)
and PSUM(MM = P I 0) = P & Mf.).
C.2 ASYMPTOTIC ANALYSIS OF ESTIMATOR ERROR
We first establish the benchmark performance of the MV estimator, which is known to be robust regardless of the noise distribution's 's tail behavior.
Theorem C.1 (MV Error Bound). The error probability of the Majority Vote estimator converges exponentially to zero:
PMV(M) S exp(-CM),
where C (2p 1)2 /2 is positive constant.
19

---

Published as a conference paper at ICLR 2026
Proof. The result is direct application of Hoeffding's inequality (Hoeffding 1963) to the sum of ii.d. bounded variables S; [-1,1] with positive mean E[S.] 2p 2 0.
Next, we analyze the SUM estimator under the condition of heavy-tailed noise, modeled as a regu- larly varying distribution.
Assumption C.2 (Heavy-Taiee -ailed Noise). The noise distribution has regularly varying left tail. Let
Y; -i. The tail probability P(Y; > 1) is regularly varying at infinity with index a > 1:
P(Y: > t) =)==t((())
where L(t) is a slowly varying function.
Under this assumption, the SUM estimator's performance degrades significantly. The following theorem formalizes this by showing that its error, relative to the MV estimator, diverges.
Theorem C.3 (Asymptotic Dominance of Majority Vote). Under Assumption C.2 the ratio of the error probabilities for the Summation and Majority Vote estimators diverges infinity:
(SUM(M)
M-+00 DC.
PMV(M)
Proof. We establish an asymptotic lower bound for the ratio. The error of the SUM estimator determined by the "big-jump principle" for subexponential distributions (Embrechts et al. 2013), which states PSUM(M) ~ M IP(Y1 > Mf) when M + co. Under Assumption C.2 this yields the asymptotic lower bound:
PSUM(M) Z M
for some constant 0. Combining this with the upper bound for the MV error, we have for all sufficiently large M:
PSUM(M)
Ci MM (M-Mo--a((MM)))
PMV(M) 2
= exp(CM) M- L(MJ).
The limit of this expression as M + oc is determined by the competition between the exponential growth term, eX)( and the product of the polynomial decay term, M-aa-1), and the slowly varying term, L) As exponential growth dominates both polynomial decay and slowly varying functions, the lower bound diverges to infinity. Consequently, the ratio itself must also diverge to infinity. This demonstrates that the MV estimator asymptotically infinitely more accurate than the SUM estimator in this setting.
D DETAILS ON EXPERIMENTAL SETUP
To ensure fair and reproducible comparison across all methods, standardized the training and evaluation pipeline. Below we describe the configuration in detail.
D.1 LEARNING RATE SCHEDULE AND HYPERPARAMETER SEARCH
All methods adopt a linear learning rate schedule with a warmup phase covering the first 10% of total training steps. Learning rates were selected via logarithmic grid search in the range 3 10 to x 10 with a multiplicative step size of The chosen learning rates for each method-modi pair are summarized in Table 6
D.2 DETAILS ON DEPTH-BIASED UPDATE RULE
Derivation of the Bounded Update Interval B. Recall that depth-biased rule selects the block with the minimum timestamp Т, and updates it as T; + T, + TG. This is equivalent to a weighted round-robin schedule where the frequency of block proportional 1/7,-
20

---

Published as a conference paper at ICLR 2026
Model
ABSignSGD
LoRA
Gal.ore
BAdam
Apollo
Owen3-8B-Man
3e-5
3c-6
le-5
le-5
le-5
Owen3-8B-Alpan
3e-5
le-5
le44
le-5
le-5
Alama3-8B-Matt
le-5
3e-6
3e-5
3e-6
3e-5
Llama3-8B-
le-5
3c-6
3e-5
3e-6
3e-5
Llama3-32B-Math
3e-5
le-6
3e-5
le-5
le-5
Table 6: Final learning rates for each method across models.
To satisfy Assumption 3.3, we must show there exists finite integer B such that every block is visited least once any window of B iterations. Consider the "slowest" block sslow which has the largest virtual cost Tmaz max, Ti- In the worst-case scenario, between two consecutive updates of block slow any other block j can be updated at most Tima/TTyl times. Therefore, the maximum number of iterations between two updates of any block is bounded by the of these worst-case update counts:
N
Tmaz
B
(8)
Given that all Tj 2 and N is finite, B is finite, thereby satisfying the bounded update interval assumption.
Execution Trace.
To clarify the mechanism described in Section 4.1 provide concrete ample of the depth-biased selection strategy. This strategy is purely algorithmic and does not depend real-time hardware measurements.
Consider simplified model with N = blocks. We define the virtual update cost 7; such that deeper blocks have lower costs (higher frequency). Let assume the costs calculated for blocks through 4 respectively (where Block 4 is the deepest).
[6, 5, 4,
The algorithm maintains virtual timestamp vector T, initialized T т. At every step, the block with the lowest 7, is selected, and its value incremented by its cost
Table 7: Trace of block selection over the first steps.
Step (k:)
State vector T (before selection)
Min Value
Selected Block (ix)
Action
(T; + Ti, + Tia
6, 5, 4,3)
Block
Ta + 3 6
1
6, 4,6
4
Block 3
T3 +- 4 8
2
6, 5, 6
Block 2
Tg 5- 10
3
[6. 10, 6,
6
Block 1
Ti + 6 12
[12, 10,
Block
T4 + 3 =9
Note: In event a (as seen in Step where Ti = and Ta = 6), we prioritize the shallower block ensure coverage, though any consistent tie- frebbrn rule works.
As shown, Block 4 (the "fastest" deepest block) is updated again at Step 4, while Block has not yet been revisited, Over many iterations, the update count for block i converges to be proportional 1/T,
D.3 OPTIMIZERS AND PRECISION SETTINGS
AdamW: Used whenever "Adam" is referenced, with PyTorch default hyperparameters (81, B2, e).
LoRA: Configured with rank consistent with its common lightweight adapter set- ting.
Low-Rank Projection Methods (GaLore, Apollo, Flora): Configured with rank r = 128, aligning with their reported accuracy-efficiency trade-off.
21

---

Published as a conference paper at ICLR 2026
Precision:
ABSignSGD run in half- colf-precision as it does not benefit from mixed seeeed-preiision training due to its sign-based update rule.
Low-rank projection methods (GaLore, Apollo) run in half-precision following their official implementations.
LoRA and BAdam follow their officiall/poplar implementations, using mixed preci- sion training.
D.4 TRAINING CONFIGURATION
Epochs: All models are trained for 3 epochs, with the final checkpoint used for evaluation.
Batch size 16, sequence length 128.
Gradient checkpointing: Enabled.
Gradient accumulation: Disabled. Note that ABSignSGD, BAdam, and LoRA are compat- ible with gradient accumulation for much larger batches. But this option is off for fair comparison.
Layer-wise updates:
Low-rank projection methods (GaLore, Apollo) employ layer-wise updates for maximum memory save, which are incompatible with standard gradient accu- mulation.
Trainable weights:
All methods update only the transformer layers leaving other weights
intact.
Offloading: Disabled.
Quantization: Disabled.
D.5 HARDWARE ENVIRONMENT
GPU: All experiments are conducted on a single NVIDIA RTX 3090 GPU (24 GB VRAM).
CPU: Intel(R) Xeon(R) Silver 4310 @ 2.10GHz.
Repetition: Each configuration is repeated times with different random seeds; we report mean + standard deviation.
Isolation: No GPU interconnect used, ensuring results influenced by distributed hardware variability.
D.6 EVALUATION PROTOCOLS
Evaluated using GPT-5 as a judge via FastChat, with fixed decoding pa- rameters (temperature 0, top-p fixed seed, official prompt templates).
MT-Bench:
Math-Eval-Harnes:: Official dataset splits with standardized prompt formatting and de- coding parameters (temperature top-p 1, max new tokens 256, fixed seeds).
D.7 RUNTIME AND MEMORY MEASUREMENT
Runtime: Measured using t time. time () and torch. cuda. synchronize () before and after the timed region.
Memory:
Record
peak
reserved
GPU
memory after
ing
torch. cuda. manoda.max..o www.aaxmmattt www.mmawwmmmr.ccccmmmmm reservedd
calling
corch.ouda. resereseeteeeee weeeeetppaak..eeeee ptw.eeteeeaammmmmrp.ee memorr.emmry.como ()
D.8 BASELINE IMPLEMENTATIONS
To ensure reproducibility, we on the official implementations of baseline methods without mod-
ification:
22

---

Published as a conference paper at ICLR 2026
Method
Official Repository
LoRA (PEFT)
GaLore
SaaiiaaaoGGaarrrooo n..pss//gottnbb..com//le/////Ba GaLore
https:/ htt.:s//ggtchbbb.ccmm///ss/22222
BAdam
Apollo
Table 8: Official implementation links for baseline methods.
E ADDITIONAL RESULTS FOR MAIN EXPERIMENTS
E. CONVERGENCE
We provide extended convergence curves for Qwen3-8B on Stanford-Alpaca, Llama3-8B on both OpenMathInstruct-22 and Stanford-Alpaca, and Qwen3-32B on OpenMathInsttrutt2 (see Figure [ Across all tasks, ABSignSGD demonstrates consistently faster iteration wise convergence compared to LoRA, GaLore, BAdam, and Apollo. Moreover, ABSignSGD's lead widens with respect to time. These findings reinforce the claim that the sign-based block updates are particularly well-suited for large-scale fine-tuning, where both iteration count and runtime are critical bottlenecks.
E.2 DOWNSTREAM PERFORMANCE
We next evaluate downstream task performance of Llama3-8B and Qwen3-32B fine-tuned with different optimizers mathematical reasoning (See Figure 6 and general instruction-following following (see Table 9 benchmarks. The trends are similar to those of the main text.
Table 9: MT-Bench scores (higher is better) for Llama3-8B fine-tuned on Stanford-Alpaca with different optimizers. ABSignSGI) attains the highest overall average and leads in four categories, while remaining competitive in the rest.
Method f Writing
Roleplay
Reasoning
Math
Coding
Extraction
STEM
Humanities
Ave.
Base
4.85
5.37
4.92
4.72
4.65
4.57
4.49
4.87
4.81
Ours
5.52
6.35
6.15
6.39
5.90
5.71
6.22
5.66
5.99
LoRA
5.40
6.03
5.29
5.39
5,08
4.71
5.31
5.16
5.30
GaLore
5.17
5.76
5.06
5.35
5,50
5.40
6.01
6.30
5.57
BAdam
5.35
5.72
6.03
5.78
4.99
6.38
5,48
5.19
5.62
Apollo
5.58
5.78
5.60
6.09
5.92
5.78
5.23
5.37
5.67
F DETAILS ON ABLATION STUDY
In this section, we detail how the ablation statistics in Figure were computed and how to inter- pret them. We also include additional results across more models and datasets, demonstrating the generality of our findings.
F.1 METHODOLOGGY
We create a set S, of 20,000 parameters uniformly sampled across transformer layers (attention, MLP, embeddings). For each parameter Sp. compute (i) the full-batch gradient V.f by aggregating over the entire training set and (ii) 500 mini-batch gradients {g} 500 from randomly sampled mini-batches.
From these, we derive:
1. Sign agreement probability
Σ
1 sign sign( V.f) sign(V
Pi
500
1=1
23

---

Published as a conference paper at ICLR 2026
0.50
0.50
LoRA
0.55
Galore
0.55
Galore
BAdam
BAdam
Training Loss
Apollo
Apollo
ABSign SGD
ABSignSGD
0.45
Training
0.43
0.40
0.50
0.0
0.2
0.5
2.6
0.8
3.0
1.2
0.30
5040
15000
15005
20000
25200
32000
35000
Taken
Time (6)
(a) Qwen3-8B (Alpaca) Token
(b) Qwen3-8B (Alpaca) Time
LoRA
LoRA
0.45
Galore
0.45
Galore
BAdam
BAdam
Training Loss
Apollo
Training Loss
Apello
ABSi gDDD
ABSignSGD
0.36
0.30
0.25
0.03
0.25
5.50
0.75
1.00
1.25
3.50
175
33000
20000
20000
40003
Token
Time (s)
(c) Llama3-8B (Math) Token
(d) Llama3-8B 3BB (Math) Time
LoRA
LoRA
0.35
Galore
0.35
Galore
BAdam
BAdam
Taining Loss
0.50
Apollo
0.50
Apollo
ABSI gnDDD
Training
ABSignSGD
0.40
0.40
0.0
2.6
3.0
35000
25:00
Token
Time (s)
(c) Llama3-8B (Alpaca) Token
Llama3-8B (Alpaca) Time
LoRA-ir1e-06
LoRA-Ir2e-06
Galore-Ir3e-05
Galore-Ir3e-05
BAdam-ir1e-05
BAdam-Ir1e-05
Training Loss
0.40
Apollo-Ir1e-05
0.45
Apollo-irie-05
ABSignSGDiir3e-05
ABSignSGD-Ir3e-0s
Training
0.25
0.2-
0.02
60000
Token
Time (5)
(g) Qwen3-32B (Math)
Token
(h) Qwen3-32B (Math) Time
Figure 5: Additional Training Loss Curves. Left column: Loss vs. Tokens. Right column: Loss vs. Wall-clock Time. Rows represent different Model/Task combinations. ABsignSGD consistently outperforms baselines, with the advantage widening in terms of wall-clock time.
24

---

Published as a conference paper at ICLR 2026
ABSignSGD
Apollo
Accuracy
Llama3-8B on OpenMatthnstrrutt22
Base
ABSignSGD
Apollo
Accuracy
(b) Qwen3-32B on OpenMathInstruct-2
Figure 6: Accuracy diverse mathematical reasoning benchmarks. Top: Llama3-8B results. Bot- tom: Qwen3-32B results. In both scales, ABSignSGD achieves a significant accuracy improvement against the second-best baseline (e.g., 5% improvement for both models).
which measures how reliably the mini-batch gradients align in direction with the full gra- dient.
2. Relative noise magnitude
300
of V.l
500
V.fl
which quantifies the average deviation of the mini-batch gradients relative to signal strength.
Figure 4 Left and Right plot the distributions P; and mes respectively. Figure Middle shows the class index histogram for all tokens in the training set, which exhibits pronounced long-tail distribution.
F.2
KEY OBSERVATIONS
Across all primary settings, we observe:
Stable signs: Most coordinates have large P; (e.g. > 0.7), indicating that gradient signs provide meaning information guide training. This matches Assumption 3.2 for theoreti- cal analysis.
Long-tailed token frequencies: Class frequency histograms show a heavy head and long tail, evidencing severe class imbalance.
Noise-dominated magnitudes: 7) is typically > 1, implying that raw gradient magnitudes unreliable compared to their signs.
25

---

Published as a conference paper at ICLR 2026
F.3 ADDITIONAL RESULTS ACROSS MODELS AND DATASETS
To assess robustness and demonstrate generality, we extend the ablation additional model-dataan pairs, including Qwen3-1.7B. Llama-3,2-1B. and GPT-Neo-1133 on OpenMathInstructz/ and
Stanford-Alpaca. shown Figure 7 the same qualitative patterns persist across architectures and corpora: gradient signs remain highly reliable, token distributions are strongly imbalanced, and noise magnitudes exceed signal strength in most coordinates.
1192
Histogram
Hislegram
Cum Cumulativ uouuoeeeeeeeeoooo Probobiitty
Count
0.5
Parameter Count
Moise Signal
0.5: 7.74%
Signal
32"
10"
10"
167
Agreement Probabilit
Class Index (frouuncyy rank)
Relattve Noice Magnitude
(a) Qwen3-1.7B on OpenMathInstrrutt2
2009
Histogram
Cumulative Probaai lllla
Count
Parameter Count
Vone Signal
p.s. 0.S: 4.47%
Noice Signal 0.08%
1280
Cumnulstvv
Sign Agreement Probability (o)
Index (frequency rani)
Log Relative Noise Magnitude
(b) Llama3-IB on OpenMathInstruutt22
Histogram
Hestogram
Obmu Obmuuaotiv botve Probability
Parameter Court
p- 0.5
Probabiit
Parameter Count
Noise Signal
p= 1.24%
Noise Signal 0.00%
1009
Sign Agreement Probability (e)
Index (fraquency
Log Relative Neise Magnitude
(c) Llama3-IB Stanford-Alpaca
2009
Histogram
337
Hletsgram
Cumubaiio ubve Picbabiitty
Perameterr Countt
p.5.5
Persmeter Count
Noise Signal
po:s
2.38%
Noise Signal 0.00%
OUmmulttion
250
Sign Agreement Probability (e)
(Frequency rank)
Log Relative Noise Magnitude
(d) GPT-nco-1.33 on OpenMaahlnsttuut-2
120K
Histgraa
Poybobiitt
Hastqgram
Curuuative Pww.baaaooooo... biity
0.0.5
Porameter Count
Noise Signal
p=0.5: 1.15%
32"
Noise Signal 0.00%
Sign Agreementt Probability (el
(Frequency rank)
Log Relative Noise Magnilude
(c) GPT-neo-1.3B on Stanford-Alpaca
Figure 7: Additional ablation results across pairs. The qualitative trends match those Figure
26

---

Published as a conference paper at ICLR 2026
G ROBUSTNESS AND SCALABILITY ANALYSIS
In this section, we provide additional empirical results assessing the robustness of ABSignSGD under diverse conditions. Specifically, we evaluate: (I) the method's sensitivity to extreme noise via batch size reduction, (2) scaling behavior distributed settings, and (3) the impact of different block-switching schemes on downstream generalization.
G.1 NOISE SENSITIVITY (BATCH SIZE LIMITS)
As discussed Section 2.2.3 sign-based methods can theoretically diverge if the agggemeeement
probability drops below 0.5. To empirically test this limit, we fine-tune QWEN3-8B on the OpenMathInstruut22 dataset while decreasing the batch size from 16 down to To strictly iso- late the effects of noise, we fix the learning rate to the optimal value identified for the baseline configuration (batch size 16, non-MV) across all experimental runs. As shown in Figure although the convergence speed degrades noise increases, the method remains stable.
ABSignSGD
AltSignSGD-BS B516
SigrSGD-B8 86
ABSignSGD B58
SigneGD
BAder-854
SigrSGiD-BS -858
Traninn Loss
0.03
BAdam-858
0.25
Figure 8: Convergence comparison under decreasing batch sizes (increasing noise). While sign- based methods (ABSignSGD and SignSGD) exhibit higher sensitivity noise than full-precision baselines (BAdam) as batch size decreases (16 - 4), ABSignSGD avoids divergence and maintains performance lead even at the extreme batch size of 4.
G.2 DISTRIBUTED SCALABILITY (FIXED GLOBAL BATCI)
we demonstrated scalability by fixing the local batch size. Here, we present additional scaling results for ABSignSGD-MV in Figure 9 where the global batch size is fixed at 16 while the number of agents varies. Similar to the sensitivity analysis, we use QWEN3-8B on OpenMathhnstrrutt22 with the fixed optimal baseline learning rate.
In the main text (Section 4.3
0.59
ABSignSGD-MVI ABSignSGD-MV2
ABSigHSGD-MVA ABSignSGD-MVS
Training Loss
0.39
0.23
0.00
0.25
0.30
0.75
1.50
1.25
3.30
3.75
Taken
Figure 9: Convergence of ABSignSGD-MV with fixed global batch size of 16. As the number of agents increases, the convergence trajectory remains virtually unchanged, confirming that the Majority Vote aggregation is robust to the number of voters.
27

---

Published as a conference paper at ICLR 2026
G.3 ROBUSTNESS TO BLOCK SWITCHING SCHEME
We further investigate whether the choice of block update rule impacts the final model quality. Fig- ure [0 compares the downstream accuracy of models trained with different schemes (Depth-Biased, Cyclic, Uniform Random). The results are from finetuning QWEN3- 1.73311.BB on OpenMathlnstruct222 Taken together with F these results confirm that ABSignSGD is robust variations in block-switching strategies, thereby offering flexibility and tunability without compromising perfor-
mance.
Base
DB-20
DB-5
DS-200
DS-10
Accuracy
GEM
MAWPS
Figure 10: Downstream generalization performance across different block switching schemes. The consistent accuracy confirms that the runtime efficiency gained from the depth- depth-biased update rule does not come at the cost of generalization capability.
H LLM USAGE
In preparing this manuscript, we made limited use of Large Language Models (LLMs) solely for minor text polishing. Specifically, the LLM was employed improve grammar, clarity, and read- ability of certain sentences. All conceptual development, theoretical analysis, experimental design, and result interpretation were conducted entirely by the authors without assistance from LLMs. The scientific content remains the authors' original work.
28


Reviews:
[{"review_id": "iB5Ox799f4", "reviewer": "Reviewer_LQY1", "summary": "The paper combines several elements (SignSGD, block-coordinate updates, majority vote) for improving the efficiency of fine tuning LLMs along various dimensions. I suspect the paper is mainly of interest to academic researchers studying optimizer design, compression, or distributed training efficiency. I suspect that few LLM training groups are likely to use sign-based methods in production due to the likely need for substantially more hyperparameter tuning even in cases where they can achieve comparable results to FP-based methods, undoing any efficiency gains.\n\n- The paper consider the fine-tuning step of LLMs.\n- Proposes ABSignSGD, combining blockwise updates, 1-bit sign gradients, and arbitrary (depth-biased) scheduling.\n- Extends to ABSignSGD-MV for distributed training via 1-bit majority-vote aggregation.\n- Provides standard convergence guarantees under sign-agreement assumptions.\n- Shows empirical gains in VRAM, runtime, and fine-tuning accuracy vs. LoRA, GaLore, Apollo, and BAdam.\n- Lacks validation at larger scales where the claimed advantages would truly matter.", "questions": "- Can you respond about the concerns about hyperparameter tuning and the overall process efficiency?\n- Why did you not try to fine-tune larger LLMs? It seems to me this paper would be much stronger if you could directly demonstrate that the method allows some fine tuning of large models on limited hardware that are impossible to fine tune using any of the other methods.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear exposition and empirical reproducibility: Algorithms, tables, and ablations are well presented.\n- Consistent, modest memory/runtime improvements relative to known baselines at 8B scale.\n- Simple implementation concept—no optimizer states, potentially useful for educational or constrained hardware studies.", "weaknesses": "- Limited practical relevance: SignSGD and its derivatives are rarely, if ever, used in large-scale LLM tuning, and this is an incremental improvement over SignSGD, making widespread adoption unlikely.\n- Limited scope: this paper only addresses fine tuning, a step that takes up only a small part of the compute budget of an LLM.\n- Scale mismatch: Experiments are confined to 8B models, but memory and communication constraints dominate only at tens or hundreds of billions of parameters. Small differences on 8B model performance may make the difference between a competitive and uncompetitive large model.\n- Questionable net compute advantage: These kinds of methods introduce their own set of hyperparameters, which also require tuning. The paper does not show that the advantages remain when that tuning is taken into account. That is, I may need to conduct quite a few runs of the method in the paper in order to be fairly confident that I have a network that performs as well as straightforward tuning with one of the other methods would have yielded.\n- In practice, this would likely be used as just another fine tuning candidate, and people would use its output only if it happens to yield better accuracy; in effect, a method intended for generating for improving compute performance would end up being just another ad hoc fine tuning variant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NspLk0fIcC", "reviewer": "Reviewer_qPmE", "summary": "The authors propose an algorithm for memory-efficient finetuning of LLMs by combining SignSGD and block-coordinate descent. It has good empirical performance compared to many state-of-art finetuning methods, and the authors also provide convergence proofs for their algorithm.", "questions": "- What is the rank of LoRA used in the experiments? How are they selected? \n\n- What are the block size used in the experiments? Are they as small as individual Q,K,V weight matrices or groups of them? \n\n- Where are the results for the block-update rule ablation study at the end of Section 4? I just see the discussion but no tables or figures for that.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The algorithm is simple and effective. It combines SignSGD with block-coordinate descent to produce a memory efficient algorithm for finetuning LLMs. It is about 10% more memory efficient than competing methods and also converges faster. It also achieves good generalization performance on many finetuning benchmarks. \n\n- In addition to the base algorithm, the authors also suggest a distributed version that employs majority voting to reduce the communication cost. \n\n- The authors provide convergence proofs of their algorithms, based on previous works from Safaryan & Richtarik 2021.", "weaknesses": "- Although the authors propose the communication efficient version of their algorithm using majority voting, I don't see any empirical evaluations on that in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NaXXDwyicQ", "reviewer": "Reviewer_UBss", "summary": "This paper studies the memory-efficient optimization methods for LLM training. Related works have adopted block-coordinate descent to reduce the memory cost of gradients and optimizer states of the popular optimizers. This paper proposes an algorithm ABSignSGD which introduces SignSGD into the block-coordinate descent to further reduce the memory cost since the signal consumes only 1 bit for one coordinate. \n\nABSignSGD also adopts a flexible rule of block selection: each block is selected in at most B iterations. The theoretical analysis shows that if the Success Probability Bound is satisfied, ABSignSGD achieves the $O(1/\\sqrt{K})$ convergence rate in the form of alignment norm. The experiments of fine-tuning LLMs show that ABSignSGD consumes the least memory and runtime among the baselines. It also converges faster and performs better in downstream tasks than other memory-efficient algorithms. The ablation studies indicate the reason why SignSGD is better than SGD and Adam when coupled with BCD.\n\nThe main contribution of this paper is that it goes a step further based on BCD and BADAM by adopting SignSGD as the optimizer in the Block-Coordinate update. The experiment results verify that this is a significant solution for memory-efficient training.", "questions": "This work proposes a “depth-biased” update rule in the experiments, but this part is somewhat confusing. Specifically, ABSignSGD selects the block with the minimal timestamp to update. However, it also says that “This event-driven rule mimics asynchronous execution, where each block becomes eligible for update once its gradient is available”. This seems somewhat contradictory since the former statement indicates that ABSignSGD updates the blocks in a serial way, why it mimics asynchronous execution. In addition, how is the gradient-computation latency obtained?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "SignSGD has been mainly studied in distributed learning or federated learning to reduce the communication cost. On the other hand, recent studies about using Block-Coordinate gradient to reduce the memory cost mainly adopt SGD or Adam as the optimizer. Thus, the originality of this work is good. The authors provide the theoretical analysis for the proposed algorithm and conduct detailed ablation studies to explain why SignSGD performs well in BCD. I think this work is of high quality and significance. Finally, most content of this work is clear and easy to follow.", "weaknesses": "This work proposes an extra extension of ABSignSGD, i.e. its distributed version. However, there is no further analysis, including both convergence analysis and experiments for it in the following contents (only a robustness analysis in the supplementary material). The meaning of proposing such an algorithm is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "COwqprLdM2", "reviewer": "Reviewer_WMLw", "summary": "This paper introduces ABSignSGD, which bridges sign-based optimization and block-coordinate training through an arbitrary-order block update rule and a distributed 1-bit majority-vote aggregation scheme.\nThe method achieves strong empirical and theoretical performance, offering substantial memory, runtime, and communication efficiency while maintaining competitive accuracy.\nComprehensive ablations and analyses confirm its practical effectiveness for resource-constrained LLM fine-tuning.", "questions": "Q1. In ABSignSGD, each iteration updates only one active block using the sign of its gradient. Could you clarify whether this *sign-only* update may lead to misalignment between active and inactive blocks across layers during training? How do you ensure stability when different blocks are updated at different frequencies?\n\nQ2. How is a *block* defined in your implementation? Does one block correspond exactly to a Transformer layer, or can it be a smaller unit (e.g., Q, K, V projection matrices) or a larger group of layers? How did you determine the optimal block size (N) in practice? Within a selected block/layer, are all submodules (attention projections, output projection, and feed-forward networks) updated uniformly with the same sign-based rule, or do you apply differentiated treatment among them?\n\n\nQ3. The paper introduces an “arbitrary-order” block selection strategy with a bounded update interval assumption. Could you elaborate on how the *event-driven* depth-biased rule is implemented in practice? Specifically:\n  * How are the latency parameters ( \\tau_i ) estimated for different layers?\n   * What determines the update readiness timestamp (T_i)?\n   * How does this scheduling compare empirically to random or cyclic updates in terms of convergence stability?\n\nQ4. This paper mention that deeper layers are updated more frequently. Could you provide more detail on the precise mathematical rule or heuristic that governs this frequency difference? For example, is it proportional to the inverse of the estimated backward latency or some adaptive function? Since deeper layers are updated more often than shallow ones, does this induce any gradient misalignment or optimization imbalance between early and late layers? Have you observed any degradation in generalization or representation consistency due to this update asymmetry?\n\nQ5. ABSignSGD is designed as a *stateless* and block-switching optimizer. In contrast, widely used adaptive optimizers such as Adam and AdamW maintain two exponential moving averages (the first and second moments), which require continuous accumulation of gradient history for each parameter. Given that block switching interrupts this continuity, is ABSignSGD fundamentally limited to *SGD-like* (stateless) optimizers?\nHave you explored—or do you foresee the feasibility of—extending your approach to *stateful* variants that preserve per-parameter moment information while still supporting arbitrary-order block updates?\n\n\nQ6: The current paper investigates ABSignSGD primarily in the fine-tuning setting, where model parameters start from pretrained weights and the optimization trajectory remains relatively close to the original model. In contrast, pretraining begins from random initialization and requires learning the full parameter distribution from scratch.\nGiven that ABSignSGD discards gradient magnitude information—updating parameters solely based on gradient signs (±1 per coordinate)—do you expect this sign-only update rule to maintain sufficient precision for large-scale pretraining?\nIn particular, could the absence of gradient magnitude information hinder convergence when the optimization landscape requires large, magnitude-sensitive adjustments early in training? Have you considered or tested any hybrid variants (e.g., partial sign scaling or adaptive step-size modulation) to mitigate this potential limitation?\n\nQ7: The paper establishes an (O(1\\sqrt{K})) convergence rate under mild assumptions using the alignment norm. Could you clarify how sensitive this convergence behavior is to the *sign-agreement probability* (ρ > 0.5) in large-scale, high-noise training regimes?\nIn particular, do you observe any empirical breakdown when ρ approaches 0.5 (e.g., with small batch sizes or unstable gradients), and how does ABSignSGD behave in comparison to standard SignSGD or BAdam under such conditions?\n\nQ8: For ABSignSGD-MV, you mention a 960× reduction in communication cost compared to standard DDP by transmitting only gradient signs.\nCould you elaborate on:\n\n* Whether the MV aggregation is performed per-block or across the full model?\n* How sensitive convergence is to the number of participating agents ?\n* Whether partial synchronization (e.g., delayed or stale majority votes) degrades convergence stability in multi-node environments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper bridges sign-based optimization with block-coordinate methods, introducing an arbitrary-order update rule and 1-bit majority-vote communication that collectively advance memory- and communication-efficient training.\n\n2. Its technical quality is supported by a theoretical convergence analysis, extensive empirical validation on LLMs, and comprehensive ablation and memory analyses that substantiate the design choices.\n\n3. The work offers a simple yet effective solution for large-model fine-tuning under tight hardware constraints.", "weaknesses": "1. The method’s reliance on *sign-only updates* discards gradient magnitude information, which may limit precision and hinder convergence in magnitude-sensitive or pretraining scenarios.\n2. The approach is primarily evaluated in fine-tuning settings on 8B-scale models, leaving its scalability and effectiveness for full pretraining or larger architectures less explored.\n3. The convergence analysis assumes a fixed sign-agreement probability and bounded block-update interval, conditions that may not strictly hold in high-noise or asynchronous environments.\n4. While the arbitrary-order rule is conceptually appealing, its implementation details (e.g., latency estimation, update scheduling) could benefit from deeper empirical or theoretical justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)