Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation
Authors: Lun Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Micro-batch Clipping
Convergence Analysis
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We revisit micro-batch clipping, a technique to improve memory efficiency in differentially private stochastic gradient descent, and find that it can also provide performance boost.

Abstract: Micro-batch clipping, a gradient clipping method, has recently shown potential in enhancing auto-speech recognition (ASR) model performance. However, the underlying mechanism behind this improvement remains mysterious, particularly the observation that only certain micro-batch sizes are beneficial. In this paper, we make the first attempt to explain this phenomenon. Inspired by recent data pruning research, we assume that specific training samples may impede model convergence during certain training phases. Under this assumption, the convergence analysis shows that micro-batch clipping can improve the convergence rate asymptotically at the cost of an additional constant bias that does not diminish with more training iterations. The bias is dependent on a few factors and can be minimized at specific micro-batch size, thereby elucidating the existence of the sweet-spot micro-batch size observed previously. We also verify the effectiveness of micro-batch clipping beyond speech models on vision and language models, and show promising performance gains in these domains. An exploration of potential limitations shows that micro-batch clipping is less effective when training data originates from multiple distinct domains.

Published as a conference paper at ICLR 2025
REVISIT MICRO-BATCH CLIPPING: ADAPTIVE DATA PRUNING VIA GRADIENT MANIPULATION
Lun Wang
Google Jonwang@google..com
ABSTRACT
Micro-bater clipping, gradient clipping method, has recently shown potential in enhancing auto-speech recognition (ASR) model performance. However, the underlying mechanism behind this improvement remains mysterious, particularly the observation that only certain micro-batch sizes are beneficial. In this paper, we make the first attempt explain this phenomenon. Inspired by recent data pruning research, we assume that specific training samples may impede model convergence during certain training phases. Under this assumption, the convergence analysis shows that micro- batch clipping can improve the convergence rate asymptotically at the cost of an additional constant bias that does not diminish with more training iterations. The hias is dependent on a few factors and can be minimized at specific micro-batch size, thereby elucidating the existence of the sweet-spot micro-bated size observed previously. We also verify the effectiveness of micro-batch clipping beyond speech models on vision and language models, and show promising perfor- mance gains in these domains, An exploration of potential limitations shows that micro-batch batch clipping is less effective when training data originates from multiple distinct domains.
INTRODUCTION
Micro-batch clipping (McMahan al., 2018 Ponomareva et al. 2023) was initially introduced as a memory optimization technique for differentially private stochastic gradient descent (DP- SGD) (Abadi al., 2016). The method groups per-example gradients within a mini-batch into smaller micro- batches, clips the average gradient of each micro-batch, and updates the model with the mean of the clipped micro-batch gradients to avoid materializing all per-example gradients in memory. Recent studies (Wang al. 2024a b) have surprisingly revealed that micro-batch clipping can enhance the performance of automatic speech recognition (ASR) models when used outside DP- SGD. This finding has sparked interest in exploring micro-batch clipping as standalone optimization technique, offering potential benefits in model performance. However, the underlying mechanism driving this improvement remains poorly understood.
In this work, we aim to elucidate the behavior of micro-batch clipping through a combination of theoretical analysis and empirical evaluation. Specifically, we conceptualize micro-batch clipping as a specialized form of data pruning (Sorscher et al. 2022). Unlike traditional data pruning techniques, which deterministically exclude redundant data, micro-batch clipping adaptively suppresses samples that hinder convergence, referred to as "draggers", recognizing that a data sample's helpfulness can change throughout training. Guided by this intuition, we introduce Assumption [4.4 to capture certain properties of the draggers' gradients, which are later empirically verified in Section 5.I Based on the assumption, we analyze the SGD and micro-batch clipping smooth loss manifolds and summarize the results in in Table [ We can observe that the introduction of draggers slows down the convergence rate for SGD by constant factor, while the usage of micro-batch clipping asymptotically accelerates the convergence rate, at the cost of an additional constant bias term. The bias term provides an explanation for the phenomenon observed in prior works (Wang et al. 2224a b) where performance improvements are only seen with specific micro-batch sizes. Concretely, because this term does not vanish with increasing iterations, the performance gain is contingent upon the magnitude of this term being
rate for both standard

---

Published as a conference paper at ICLR 2025
Table 1: Convergence Rate
Dragger
Clipping
Convergence Rate
TVs 2L - L.) o
Thm 9 Bu al. 2024)
x
Thm 4.1
/
Ti/A (1-c)² (Lu - L.)
2xa
(1-t)B
Thm 4.2
VT OLo - C.
(vle ve(L-e)))((tt))
sufficiently small. According to Theorem 4.2 only certain micro-batch sizes minimize the bias term, thus explaining this effect.
To test the effectiveness of micro-batch clipping outside the speech domain, we apply it to vision and language models and observe promising performance improvements. Another interesting observation is that micro-batch clipping's efficacy diminishes when facing multi-domain training data, particularly when data sizes between domains are unbalanced. This may attributed to the method hindering the convergence of domains with fewer samples by treating them draggers.
BACKGROUND
2.1
MICRO-BAR BATCH CLIPPING
While originally introduced in MMcMahan et al. 2018 as transition from record-level differential privacy to user-level differential privacy in federated learning, micro-batch clipping's 's widespread adoption in differential privacy libraries stems primarily from its memory efficiency (Ponomareval et al. 2023 Wang et al. 2024a b). By requiring fewer gradients to be materialized, it reduces memory consumption, especially for large models, at the expense of lower agnal-to-noise ratio under the same differential privacy budget. Notably, per-core clipping (where the miero-batch size equals the per-core batch size) achieves complete memory parity with non-private training when data parallelism is employed.
The potential of micro-batch clipping as an optimization technique to improve model performance emerged from observations in (Wang et al. [2024a] b), where it's shown to reduce word eттоr rate (WER) when training Conformer-buis based ASR models (Gulati al., (020). However, whether this benefit generalizes beyond specific model architectures and datasets remains an open question that this work seeks to address.
2.2 DATA PRUNING ACTIVE LEARNING
growing body of research (Toneva al. 2018 Paul et al. 2021 Sorscher al. 2022) highlights the detrimental effect of unhelpful even harmful training data model performance, particularly in large-scale datasets. Sorscher et al. (Sorscher et al.) 2022) propose metric-based approach to prune such data and alleviate inefficient power law scaling in theory. Although the approach effectively reduce the size of the data to store, the method overlooks the dynamic nature of data utility, which can evolve throughout the training process.
Active learning (Bordes et al. 2005 Settles 2009; Sener & Savarese 2017 Birodkar et al. 2019
Mirzasoleiman al. 2020 Emam al. 2021 Karamcheti al. (202) offers alternative approach to address this challenge by actively selecting the most beneficial training samples. Our work shares similarities with active learning in that data importance determined dynamically during training based on gradient information. However, our approach is more fine-grained, implicitly adjusting data influence through adaptive gradient clipping rather than explicit sample selection.
METHODOLOGY: ADAPTIVE MICRO-BATCH CLIPPING
This section formally defines micro-batch clipping, focusing on the adaptive variant proposed in (Wang al. 2024b), where the minimal I.2 norm of all average miero-batch gradients is used
2

---

Published as a conference paper at ICLR 2025
the clipping bound, thus ensuring that all micro-batch gradients clipped. We choose this variant because avoids introducing clipping bound as an extra hyper-parammter, which requires further tuning, and (Wang et al., 2024b) shows that it performs comparably to clipping with manually selected clipping bound. For brevity, subsequent references to "micro-batch clipping" will refer to this adaptive variant.
Let DD be the training set and we want to run stochastic gradient descent with adaptive micro-batch clipping. The mini-batch size B and learning rate 17. In each iteration, mini-batch of training examples are picked and sharded into micro-batches of size b. The average gradient on each micro- batch will be calculated and clipped using the minimum L2 norm within all the micro-batch gradients. Then the clipped micro-batch gradients are summed to get the mini-bate gradient which is then used to update the model (or passed to the optimizer if adaptive optimizers are used.). The process formalized in Algorithm
Algorithm Pseudocode for SGD with adaptive micro-batch clipping. 9t represents mini-batch gradients in the 1 iteration. g' represents the micro-batch's gradient in the 1th iteration.
Input: initial parameters wy, loss function L, training data D, #iterations T, micro-batch size #mini-batch size B, learning rate 7).
for -,1,2 2, ... do
2:
{d]) (I.....) B) +- D
sample a mini-batch
3:
for j = 1, 2........ B (parallelly) do
4:
Load micro-
{d] (M0-)))))))).
5:
9/ = -VL(w;-1, 1001,,14)eee -D).....))))
> obtain average gradient of a micro-batch
6:
Pr = min, |1/ 2|2
7:
for 1,, B/b (parallelly) do
8:
0: 110/112
e1 gl
adaptive clipping
9:
gt = Σ, jE(1 B/b) P.
10:
1) 9t
update the model parameters
4 CONVERGENCE ANALYSIS
In this section, we provide the convergence-to-saaieee tottsttttttioary--oittt analysis for SGD micro-batch clipping following (Bu et al. 2024 and show that it achieves asymptotically faster convergence at the cost constant bias under certain assumptions.
4.1 ASSUMPTIONS
Our analysis is based on few assumptions, listed below. Among these, Assumptions 4.1 and 4.2 are standard assumptions adopted from previous work (Chen et al. 2020 Bu et al., 2024). Assumption 4.3 is based on prior research (Chen al.) 2020 Bu et al.) 2024), with a slight modification to accommodate Assumption 4.44 a new assumption that we introduce in this work to capture the existence of dragger examples.
Assumption 4.1 (Lower bound of loss [Bu et al. 2024 Ye, E constant L., s.t. L(w) 2 L.
Assumption 4.2 (Smoothness Bu et al. 2024 ) Define g(w) == @C(w) Dmn Then Vw, 3 constant
2 S.I.
((v) - (C(w) + ( T(y -w)] < 1/ r112.
Assumption 4.3 (Gradient distribution (Bu et al. 2024)). With probubility - the per-example gradient g is an i.dd symmetric unbiased estimator of g with bounded variance o |Bu et al. 2024)).
w.p. Eig] g, EEllg 91l < o²
With probability c, the per-example gradient is dragger
w.p. e, g p
3

---

Published as a conference paper at ICLR 2025
Assumption 4.4 is introduced to formalize the observation that certain gradients within the training data may impede the model's s convergence. First, we assume that these "dragger" gradients are orthogonal to the gradients of benign examples, following the intuition that the cosine similarity between data from different domains should be low. The intuition is both motivated and supported by empirical evidence presented in Section [.I] Second, gradient is both lower- and upper-bounded by the L2 norm of the expected benign gradient scaled by constant factor. The lower-bound captures the intuition that dragger gradients should be large enough to slow down the convergence. Empirical evidence for the intuition is presented in Section 5.3 The upper bound captures the fact that these detrimental gradients are not the result of adversarial manipulation but arise naturally within the vast and diverse training dataset.
assume the I.2 norm of the dragger
Assumption 4.4 (Dragger). The dragger gradients are orthogonal to the benign gradient subspace. Furthermore, we assume that the ratio of the dragger's 's norm the expected benign gradient norm bounded both above and below.
1 gg < ||||| S Cll
4.2 CONVERGENCE RATE OF STANDARD SGD WITH DRAGGER GRADIENTS
We first provide the convergence to stationary point of standard SGD result under the assumptions above.
Theorem 4.1. Under Assumption 4.7
4.2
4.3
(without the symmetry assumption). 4.4
rumning SGD
for T iterations gives, for 1 LVT
1
4L
20
S
T2/1
V
(1
61
(Co L.) +
(1
Proof. In the standard SGD,
7 Eg.
Wr+1 w; where Ô.., here represents "M per-example gradient sampled from the distribution described by Assumption and 4.4
B
By Assumption 4.2
L w.l"
Lit1 Ls S g. (wi-1 wi) +
| |
n9.
gt.i + Σ g1.)) +
Lnp²
Σ
g1,i +
Σ
gr,i|l
B
2Ba
benign
dragger
benign
dragger
The expected improvement at one iteration is
Tay² 2B²
ElL++1 L./we]] - -(1 c) Evenign gr.c]
cng.
Estrager [gr, +
EE|
Σ
9 +
Σ
I1..
benign
dragger
Ln
Lnp
= - 1n
E
Σ
31.111 +
E
I1,211
2B²
2B²
henign
dragger
S 2(11- e)nllgr11 + (1 Lm
2 Lmz
[|gal12 +
(1 B)BBBBB +
|1 12
2
2
$ - L)
2 L
+ (1 - L1720
2 - -(1 c)m +
111
2
2B
The second equation follows from Assumption 4.4 that the benign gradients are perpendicular to the dragger gradients. The fourth inequality is based Assumption 4.4 that the dragger gradient norm
4

---

Published as a conference paper at ICLR 2025
is upper bounded. To make sure the coefficient is negative, require that
c)
2(1 c) - (1 (1) Tmp
2 - - -m1- -im
2 <0=C<
e² ex2²
2((1=))n-( (1-)))(-)))00 (1-
For arithmetic convenience, choose C / over the iterations
Now we do telescoping
Lxp
Lo 1--L. L. Lo ELT = Σ BlC, - -(+1)
(1 - c)n (1 - LIT
(1
c)
E
|1gel13)
4
2B
We apply the same learning rate in (Bu et al.
2024
11 LVT
1-6
(1 (1-
(1 e)o²
Co C, 2
)
|1g.1(²)
2LVT
4LT
2LB
))²
(1 c)o²
E
!gell?)
41. T
2LB
and finally
4h
20²
min min Elg &
(1-
(Co -L.) +
VT
e)2
(1-c)B)
Using Jensen's inequality, we can have
min min E g.ll TIA (1-e) (Lo - L.) + 2r²
1
4L
(1 (1-c)B
In comparison Theorem 9 of (Bu al. 2024). SGD with draggers demonstrates the same asymptotic convergence rate, albeit with a larger constant coefficient. This observation aligns with our intuition that the presence of dragger gradients hinder the model's convergence speed.
4.3 CONVERGENCE RATE OF MICRO-BATCH CLIPPED SGD WITH DRAGGER GRADIENTS
In this subsection, present the convergence analysis for micro-batch clipping. To simplify the algebra, we assume a uniform lower bound for the clipping bound (pt) used in each iteration. Note that this assumption doesn't 't impact the asymptotic behavior. For conciseness, the proofs of the supporting lemmas are provided in the appendix.
Theorem 4.2. Under Assumption 4.1
4.2 4.3 4.4 running micro-batch clipped for,for' T iterations
gives, for n LVT' micro-batch size b and dragger probability 5+1
min El S (vbe Ve(1 - t)) (1+c)
1
Vbe
2L(1+260
1
(Lo L. +
VT
vbc
V1-c
1-c
2T.
Proof Sketch. In the micro-batch clipped SGD with micro-batch size b, the update rule is as follow:
wi+1 = wi
dHb
Σ
9t,i
(2)
B
g
5

---

Published as a conference paper at ICLR 2025
Note that here 9t.; represents 1th micro- batch gradient in the tth iteration. For notation simplicity, omit the subscripts it's clear from context.
By plugging in the update from Equation 2 the smoothness assumption 4.2 and take conditional expectation, the expected improvement at one iteration is
[[ - La Cal & -n9:
9r 19
Laf 2
Re-write ÿ. (1 <)ge + cjh, + A., where A, is a zero-centered random variable whose expected L2 norm bounded below. The proof can be found in Appendix
Lemma 4.3. The expected L2 norm of Ds is upper- hoper-bounded by.
+ c(1 c) ((1 c) gel
14, 2 + (1 cco
B| &
b
To lower bound D, E &. divide the support of A into two half-spaces where denote the positive half as: H4 := {u g 0}. Then using the symmetry assumption 4.3 we have
TT..I we follow (Bu et al. 2024 to use the hyperplane perpendicular to gr to
9t
(1 9:1 11 g A.
91
B(
(11 6) gt + c/e + Arl
(1 e)1l9c|1? + g, A,
(1 1)
9. A.
(
A E H+) + - E
A E H+)
2
[[(1 <)g + cp + A,ll
2
[1(1 c)g
cpe A.l
E
1
(1 <| + +g. A:
(1- 1==e)lg:1'
g. A.
A HH+)
2
e)g. + EHe A.l
[[ c)gr cfe,
A.l ||
1 | A. E H4, | S selmaal
+ 11l S E|| + +egg::))
2
1 B 14, +
e p
|| + |( > e||m||l + ellgell)
Lemma 4.4 tells us that * is always non-negative under Assumption and thus we can only keep the first term. The proof is provided in Appendix B
(1-0 t A + (I-) d
Lemma 4.4. If D. H+ and μr 1 gt. ph 1 A.. then
111-0) tot A,l O.
[11-0) to A.
9:
9.
E
B(/A,,
H4, S selppll (ell:+++llll1])P((.) S e||po|l +elgg:1)
9
2
1
<) ||gh|1
E A
B(
A H+, ||||| || S e <l| + =c|gee1)) (1
2
(1 <) ||g||
<||Meell
+ ||A|||
| + E =
1
(1 - <) e)
1
Ve(1 1) g 2 + c1
14 2 (1 c)o²
(1 (1-
2
||g.l| + 2c
vb
< <l|mu + e gell
1 1-c
--- |1g.|||
1
((1
2(1 + 2cC)
be
vb
e(1 c)
where the second inequality follows from Markov's inequality.
Thus we have
EE CL - La w// < -n
1 1-c
((1
Ilgell
σ E(1 + cc) vb
+ Lnd
2(1 2cC)
be

---

Published as a conference paper at ICLR 2025
Now do telescoping over the iterations
Co - L. 2 Lo ELT
Σ E(C, - L(+1)
1 n 2(1 2cC)
((1 ((1- 1 I1gell
Lii'T
be
vb <(1 + c)
We apply the same learning rate in Bu et al.
2024 11 LVT
(11 -(1- Vi 1-1))999 "lelll
1
Lu L. 2L 2L 260)
vb c(1 +c) )
be
2L
and finally
min t
gllll S min 7 Σ
11:1| S
(vbc v((1 - (1)+c)
vbe 2L 2cC)
(Co L., + 2T!
VT vbc v1-c
1-c
EMPIRICAL EVALUATION
In this section, we'd like to answer the following questions: 1) Does Assumption 4.4 hold in practice? 2) Does the hypothesis on c to explain the sweet spot of micro-batch size hold in practice? 3) Does micro-batch clipping help improve performance beyond speech tasks?
5.1 EMPIRICAL. EVIDENCE FOR ASSUMPTION 4.4
Experiment Setup. To answer question 1). we adopt the experimental setup from Wang et al. (Wang et al. 2024a). Specifically, we fine-tune 600M Conformer XL (Zhang al. 2020) models on the LibriSpeech dataset (Panayotov et al. 2015) and handcrafted canaries (Wang et al. 2024b). The model's ''' encoder is pre-trained using BEST-RQ (Chiu et al.) (022) on the LibriLight dataset (Kahn et al. 2020). The key advantage of this setup is the ability to treat the inserted canaries as surrogate draggers, thereby circumventing the technical challenge of identifying natural draggers in large models.
We compute the cosine similarity between 100 randomly selected pairs of dragger and benign gradients. For calibration purposes, also compute the cosine similarity between 100 pairs of benign gradients.
Evaluation Results. The results summarized in Figure 1 We observe that the magnitude of the cosine similarity between dragger and benign gradients is significantly smaller than the cosine similarity within benign gradients, with negligible standard deviation. This observation supports Assumption 4.4 that dragger gradients are orthogonal to the benign gradient subspace.
5.2 How CHANGES WITH MICRO-BATCH BATCH SIZE?
Experiment Setup. Again we re-use the setup in Wang et al. 2024a). First, we run micro-batch clipping with micro-batch size 1, (per-core batch size), 512 batch size) to substantiate the existence of an optimal micro-batch size that yields maximum performance. Performance is assessed using word error rate (WER) two splits of the LibriSpeech test dataset: test-clean, consisting of relatively clean utterances, and test-other, consisting of noisier utterances.
In the second step, the following proxy estimation is used to monitor the fluctuations in the value of
mean ( gradients])
é =
mean( dragger gradients|])
7

---

Published as a conference paper at ICLR 2025
dragger vs. henign
benign benign
Cosine Similarity
0.1
0.1
0.5
1.5
2
# Iterations
-104
Figure 1: Cosine Similarity between Crafted Dragger and Benign Examples.
To estimate the numerator, we use the mean norm of 100 LibriSpeech samples after trimming the largest 10% to avoid the effect hard-to hard-to-track natural draggers. To estimate the denominator, we use the mean of the gradient norms of 100 inserted canaries. The proxy ê is computed for both non-private training and micro- batch clipping scenarios with micro-batel sizes 1, and 512.
0.1
Baseline
Ours, b=4
WIR
0.0631
0.0398
2,510
3,980
6,310
10,000
15,800 25,100
Iterations
Figure 2: Convergence curve for LibriSpeech experiments. Both x-axis and y-axis log-scale highlight the convergence speed advantage.
Evaluation Results. First, in Table 2, we can observe that when micro-batch size is 4, the model achieves the best performance across all settings, improving test-other WER by 4.8% relatively and test-clean by 4. 4.1% relatively. Furthermore, we note that micro-batch clipping does not enhance performance when the micro-batch size is either on both the test-other and test-clean splits. As per Theorem 4.2 this degradation is attributed to the presence of large constant term when the micro-batch size not optimally selected. Furthermore, we note that micro-batch clipping does not enhance performance when the micro-batch size is either I or 512. Instead, these configurations degrade WER on both the test-other and test-clean splits. As Theorem 4.2 this degradation is attributed to the presence of large constant term when the micro-batch size is not optimally selected. Besides, from Figure we can tell see the convergence
512. Instead, these configurations degrade WER
rate advantage as predicted.
Second, the values of è for different micro-batch sizes are shown Table 3 The results conform with the conjecture that is decreasing with respect the micro-batcl size b. This observation provides strong evidence to support the explanation for the existence of the optimal micro-batch size.
Remark. We'd like note that the difference in proxy è is still not big enough to level out the influence of in the constant term in Theorem 4.2] and this might be attributed to the several factors including the convergence rate bound not strictly tight, the noise in è measurement, and systematic
8

---

Published as a conference paper at ICLR 2025
Table 2: Best WER on LibriSpeech within 20K fine-tuning steps. Mean and standard deviation are calculated across 3 runs. Bold highlights best. The checkpoint picked to optimize test-other split.
Test set
Baseline
Ours, b
Ours, b = 4
Ours, b 512
test-clean
1.93 + 0.04
2.40 = 0.13
1.85 = 0.07
1.94 = 0.00
test-other
3.99 + 0.01
4.86 = 0.24
3.80 + 0.05
4.03 = 0.03
bias between è and c. However, we still believe that the trend of è promising as an evidence for our explanation for sweet-spot micro-batch sizes, and deem getting stricter convergence bound and more accurate relationship between and important future directions to guide the choice of micro-batch sizes.
Table 3: The values of è under different micro-batch sizes.
Micro-batch size b
b =
4
= 512
6.18
5.74
4.42
5.3 IS MICRO-BATCH CLIPPING EFFECTIVE FOR MODELS BEYOND SPEECH?
To address question 3), we conduct an evaluation of micro-batch clipping on vision models and language models. In accordance with the findings of (Wang al., 2024b), we default micro- batch size equivalent the per-core batch size due to its memory efficiency. It is important to note that alternative micro-batch sizes may necessitate careful adjustment to mitigate potential memory overflow issues.
5.3.1 VISION TASKS
Experiment Setup: For vision tasks, we choose apply micro-batch clipping to the DeiT-B model proposed (Touvron et al.) 202T). We train DeiT-B from scratch on the ImageNet dataset (Deng et al. 2009) and achieves parity with the reported Top-1 accuracy in (Touvron et al.) 2021). The mini-batch size used is 4096 and the micro-batch size is 32, the same as the per-core batch size for computational efficiency.
Evaluation Results:
The results are summarized in Table 4 Adding micro-batch clipping im- proves the Top-1 accuracy by 1.5% and Top-5 accuracy by 1.0%.
Table 4: VIT trained w/ or w/o adaptive micro-batch clipping.
Top-1 accuracy (%)
Top-5 accuracy (%)
Baseline Ours
81.8
95.8
83.3
96.8
5.3.2 LANGUAGE TASKS
Experiment Setup: For language tasks, we fine-tune a T5 model (Raffel et al. 2020) on the superGlue dataset (Wang et 2019). The model pre-trained on the C4 dataset (Raffel al.) 2020) for 1M steps. The mini- size is 2048, and the mini-batch size is 16, the same as the per-core batch size. Note that the test set for superGlue private so we report the performance on the validation set. Since we do not use any information of the validation set to improve the micro-batcl clipping part, the improvement shown fair. Also we use accuracy across subsets of superGiuue to be able to compute the weighted average accuracy as a unified metric to compare 2 models.
Evaluation Results: Table summarizes our findings. While micro-batch clipping only marginally improves average accuracy (0.1%), it significantly boosts performance the largest, unbalanced subset, ReCoRD (0.4%), while negatively impacting smaller datasets. This suggests varying gradient distributions across subsets, with micro-batch clipping suppressing gradients from smaller subsets,

---

Published as a conference paper at ICLR 2025
which conforms with the observation that ReCoRD has lower aceuracy when trained with other data mixed (83.9%) than when trained alone (84.5%). Training solely on ReCoRD yields a 0.3% improvement shown in Table reinforcing this hypothesis.
Table 5: T5 trained w/ or w/o adaptive micro-batch clipping.
Accuracy (%)
wt avg
BoolQ
CB
COPA
MultiRC
ReCoRD
RTE
WiC
WSC
|Train|]
9427
250
400
5100
101k
2500
6000
554
Baseline
85.9
93.4
97.8
88.9
93.4
83.9
97.6
89.5
70.3
Ours
86.0
91.8
95.6
88.3
91.6
84.3
95.7
88.5
70.3
These results highlight a crucial lesson: micro-batch clipping, while beneficial when data comes from the same domain but is noisy, can be detrimental with unbalanced multi- data. In such scenarios, micro-batch clipping may hinder certain domains to favor others and should be avoided.
Table T5 trained w/ or w/o adaptive micro-batch clipping.
Accuracy (%)
Baseline
Ours
ReCoRD
84.5
84.8
6 CONCLUSION & DISCUSSION
In this paper, we revisit micro-batch clipping. originally proposed the context of differential privacy, from the lens of data pruning, inspired by recent observations made by (Wang al. 2024a
Our convergence analysis demonstrates that micro- clipping can asymptotically accelerate the convergence rate for smooth loss functions. To elucidate the optimal micro-batch size falling between and the mini-batch size, we introduce the concept of "dragger gradients." Combining this concept with our convergence analysis reveals constant term minimized value between 1 and the mini-batch size.
Our analysis and explanation hinge on two novel hypotheses, which we empirically verify. We also extend the application of micro-batch clipping to vision and language models. We find that when the input data is Lingledommie micro-batch clipping can still enhance the performance of these models.
Limitations. Despite the demonstrated effectiveness of micro-batch clipping across speech, vision and language models, our research reveals limitations that warrant further investigation. As evidenced in Section 5.3.2. micro-batch clipping can adversely affect model performance when the input data originates from multiple distinct domains. This constraint limits the applicability of micro-batch clipping in scenarios where data diversity is prevalent.
Furthermore, this limitation may raise potential fairness concerns, akin to other data pruning methods pointed out by (Vysogorets et al. 2024). The preferential treatment of specific data domains could inadvertently introduce biases and inequities model outcomes. These limitations underscore the need for refined micro-batch clipping approach that can effectively handle multi-domain data.
Future Work. Several intriguing avenues for future research emerge from this paper. Firstly, the memory overhead introduced by micro-batch clipping remains a challenge. While data sharding with a micro-batch size equal to the per-core batch size (Wang et al. 2024a b) can mitigate this issue, the growing size of modern models necessitates the development of alternative solutions to ensure the computational efficiency of micro-batch clipping training scenarios with model sharding alone.
Secondly, from theoretical perspective, more rigorous relationship between micro-batch size b and c is needed, as could guide the choice of the appropriate micro-bate batch size to optimize performance.
Finally, eager to explore a broader range of gradient- based data pruning methods. This includes investigating different variations of clipping, as well as other innovative gradient manipulation techniques based on diverse heuristics.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Vighnesh Birodkar, Hossein Mobahi, and Samy Bengio. Semantic redundancies in image- classification datasets: The 10% you don't 't need. arXiv preprint arXiv:190.114009 2019.
Antoine Bordes, Seyda Ertekin, Jason Weston, Léon Botton, and Nello Cristianini. Fast kernel classifiers with online and active learning. Journal of machine learning research, 6(9). 2005.
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. Advances in Neural Information Processing Systems, 36, 2024.
Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspective. Advances in Neural Information Processing Systems, 33:13773-13782, 2020.
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-pinn paadooo-rooeetionn quantizer for speech recognition. In International Conference on Machine Learning, pp. 3915-39952 33244. PMLR, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Zeyad Ali Sami Emam, Hong-Min Chu, Ping- Ping-Ych Chiang, Wojeiech Czaja, Richard Leapman, Micah Goldblum, and Tom Goldstein. Active learning the imagenet scale. arXiv preprint 1rX.v:2/11./2880 2021.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmentee transformer for speech recognition. arXiv preprint arXiv:2005.088 08000,00000000000 2020.
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Exgeny Kharitonov, Qiantong Xu, Pierre-Emmantnn Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Ligri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D Manning. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. arXiv preprint arXiv: 107.02333, 2021.
H Brendan McMahan, Galen Andrew, Ulfar Brlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, and Peter Kairouz. A general approach to adding differential privacy to iterative training procedures. arXiv:1812.06210. 2018.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6960.
PMLR, 2020.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2075 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2206-5210 IEEE, 2015.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596-20607, 2021.
Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta. How dp-fy ml: A practical guide to machine learning with differential privacy. Journal of Artificial Intelligence Research, 2023.
11

---

Published as a conference paper at ICLR 2025
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. arXiv preprint 1..Kv:1708.00000 2017.
Burr Settles, Active learning literature survey. 2009.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19523-19536, 2022.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: 1812.0519, 2018.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sublayrolles, and Hervé
Jégou. Training data-efficient image transformers distillation through attention. International conference on machine learning. pp. 10347-10357. PMLR, 2021.
Artem Vysogorets, Kartik Ahuja, and Julia Kempe. Robust data pruning: Uncovering and overcoming implicit bias. arXiv preprint arXiv:2404.0557, 2024.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for general- language understanding systems. Advances in neural information processing systems, 32, 2019.
Lun Wang, Om Thakkar, and Rajiv Mathews. Unintended memorization in large asr models, and how to mitigate it. In ICASSP 2024-2024 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pp. 4655-4659, IEEE, 2024a.
Lun Wang, Om Thakkar, Zhong Meng, Nicole Rafidi, Rohit Prabhavalkar, and Arun Narayanan, Effi- ciently train asr models that memorize less and perform better with per-core clipping. InterSpeech 2024, 2024b.
Yu Zhang, James Qin, Daniel Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu. Pushing the limits of supervised learning for automatic speech recognition. arXiv preprint arXiv:2010. 2020.
A PROOF FOR LEMMA 4.3
Proof. First, we upund the variance for the per-example gradients following the distribution described in Assumption 4.3
Bllğ 2 = Elgğ |Eğl 1
< (1 () 1 + 0²) +c|lu|l2 - [|(1 c)g + eHul?
e |l + +e
e) (1 e)
Second, after the micro-batch averaging,
Blgg - 2|
[(I Σ [g - E²)
62
Σ E E(Ig Eg 12)
c(1 <) ||وا + e(1
[ (1 E)o²
where the second equation follows from the fact that per-example gradients are independent from each other.
12

---

Published as a conference paper at ICLR 2025
B PROOF FOR LEMMA 4.4
Proof. If g. A, S (1 )) 9 12 then the lemma obviously holds, so we only need to prove for the
case when 9t D (1 <) 1lg
2119 1 11 + 19:11 11 2 2 (g) A.)² ((1 (1-c))l ge 2 + 9. A,)
(1 e) + + || 21 + 2(1 E)ge A.
((1
9 11 gi A.)
2 (1 e)²1 gt 2 + e2 1
2 + |1.112 2(1
c)g,
A.
(1 - 6) 1::11 + g. A.
(1 (1-c) 1)- 2 9. A
20
[[1 e) c/14 A.l
[[11 c)g: + et4 A/
13


Reviews:
[{"review_id": "dzi9buYQfY", "reviewer": "Reviewer_stJ2", "summary": "This paper revisits micro-batch clipping, an optimization technique initially proposed for memory efficiency in differentially private stochastic gradient descent. The paper conceptualizes micro-batch clipping as a form of data pruning, proposing that certain \"dragger\" gradients impede model convergence. Through convergence analysis and empirical evaluation, the paper demonstrates that micro-batch clipping can asymptotically accelerate convergence but introduces a constant bias term. This bias term explains the existence of an optimal micro-batch size. The paper further validates the effectiveness of micro-batch clipping beyond speech models, showing promising results on vision and language tasks. However, limitations are identified when training data originates from multiple distinct domains.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel Conceptualization: The paper innovatively frames micro-batch clipping as a data pruning technique, introducing the concept of \"dragger\" gradients to explain its mechanism. This provides new insights into why and when micro-batch clipping works.\n2. Comprehensive Analysis: The paper conducts both theoretical convergence analysis and extensive empirical evaluation to support its hypotheses. The results demonstrate the effectiveness of micro-batch clipping across different model architectures and datasets.", "weaknesses": "1. Lack of interpretability or robustness. This paper does not discuss the potential impact of micro-batch clipping on the interpretability or robustness of the trained models, which are important considerations in many real-world applications.\n2. Limited scope of experiments: The experiments are primarily conducted on speech recognition models and limited vision and language tasks. The generalizability of the findings to a broader range of models and datasets is unclear. More extensive experimentation across different domains and model architectures would strengthen the conclusions.\n3. Concern of memory overhead: While the paper mentions that data sharding with a micro-batch size equal to the per-core batch size can mitigate memory overhead, this may not be feasible for larger models. The growing size of modern models necessitates alternative solutions to ensure computational efficiency.\n4. Rigorous relationship between micro-batch size and performance: The paper suggests that there exists an optimal micro-batch size that minimizes the constant bias term in the convergence analysis. However, the relationship between the micro-batch size b and the factor c is not rigorously established. A tighter convergence bound and a more accurate relationship between b and c could better guide the choice of micro-batch sizes in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tH4OTNxdkA", "reviewer": "Reviewer_nmLX", "summary": "Micro-batch clipping has recently demonstrated potential in improving model performance in automatic speech recognition (ASR). This paper seeks to uncover the mechanism driving this enhancement. The authors assume that certain data samples (\"draggers\") may impede model convergence. Through theoretical analysis, they find that micro-batch clipping can accelerate convergence by diminishing the impact of these draggers, albeit introducing a constant bias dependent on micro-batch size. This insight further explains the observed phenomenon where improvements occur only at specific micro-batch sizes. Additional empirical experiments are conducted beyond ASR to investigate the effectiveness of micro-batch clipping.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Uncovering the mechanism behind the performance improvements brought by micro-batch clipping is both interesting and valuable.\n- The paper provides a series of theoretical foundation to analyze this mechanism and support its conclusions.\n- The theoretical findings in this paper can explain empirical results, such as the performance gains achieved through micro-batch clipping and why these improvements occur only at specific micro-batch sizes.\n- Extensive experiments are conducted to explore the benefits of micro-batch clipping beyond ASR.", "weaknesses": "- The Appendix of this paper is somewhat lost, and the full proofs of the theorems are thus invisible, requiring further verification.\n- Assumption 4.4 posits that dragger gradients are orthogonal to the benign gradient subspace, which may be a restrictive assumption and not entirely realistic in practical scenarios.\n- In Sec.1, the authors claim that micro-batch clipping can adaptively suppress samples that hinder convergence. However, it remains somewhat unclear in the theorem whether the effectiveness of micro-batch clipping directly stems from suppressing these draggers.\n- It appears that as $\\epsilon$ decreases, approaching zero and indicating a lower likelihood of draggers, the convergence rate bound in Thm 4.2 actually increases, which seems counterintuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yv7IJNaS2U", "reviewer": "Reviewer_G2Xp", "summary": "This paper presents a theoretical analysis of why the micro-batch clipping method, which has been shown to improve performance in automatic speech recognition (ASR) tasks, is only effective for specific micro-batch sizes. The authors first categorize mini-batch samples into benign samples and dragger samples, followed by a convergence analysis. This analysis provides an explanation of the sweet spot for micro-batch size. Finally, the paper demonstrates that micro-batch clipping is also effective in both vision and language tasks.", "questions": "- How are micro-batches grouped within a mini-batch? Is this process conducted randomly?\n\n- Around line 114, the text states that mini-batch gradients are \"summed,\" but isn't it more accurate to describe this as \"averaged\" or \"aggregated,\" given that the mean is typically computed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The topic of theoretically analyzing the batch-size dependency issue of micro-batch clipping in training ASR models, which has shown empirical success, is both meaningful and interesting.\n\n- Despite the mathematical rigor throughout the paper, the overall writing is easy to read and follow; in particular, both the content and proof of Theorem 4.1 are easy to understand.\n\n- In the case of the ASR task, the paper presents promising empirical results that align well with the theoretical analysis.", "weaknesses": "**Methodology**\n- Section 4.3 is significantly underdeveloped and requires more extensive elaboration. Expanding on this section with a detailed explanation of the method would improve clarity and depth.\n  - In line 279, the paper states, “The proof can be found in ??,” but the proof is missing, and there is no supplementary material to refer to.\n  - The proofs for Lemma 4.3 and Lemma 4.4 are also missing. Furthermore, in line 304, \"The proof is provided in Appendix ??.\" mentions that the proof is in the appendix, but no appendix is provided in the paper.\n  -  I suggest that the authors include these proofs in an appendix or supplementary material and ensure that all references to proofs are properly linked.\n\n-  The samples that slow down convergence, termed \"draggers,\" seem to refer not to those with gradients perpendicular to the benign samples, but rather those with gradients in the opposite direction (i.e., cosine similarity around -1); in Figure 1, there appear to be cases where benign samples have negative cosine similarity. Are there no samples with reverse gradient directions?\n\n- As noted in the paper's limitations, determining the exact micro-batch size $b$ that achieves the \"sweet spot\" is a challenging task. This poses a significant empirical limitation, and even within the paper, the optimal micro-batch size changes due to various factors:\n  - The best micro-batch size varies by modality; for example, the authors use a size of 4 for the speech task, 32 for the vision task, and 16 for the language task.\n  - Additionally, memory constraints may impede the selection of the optimal micro-batch size. For the vision task, a micro-batch size of 32 was used per core, but such size may be difficult to adjust due to memory limitations.\n  - I suggest the authors discuss potential strategies or heuristics for selecting micro-batch sizes in practice, considering the observed \nvariability and memory constraints.\n\n- Along with batch size, one of the most critical hyperparameters in training neural networks is the learning rate. Prior research has explored the relationship between batch size and learning rate [1].\n\n- Is micro-batch clipping independent of mini-batch size, as opposed to micro-batch size? Further investigation into this relationship could be beneficial.\n\n- While this paper analyzes micro-batch clipping in the context of SGD, in practice, SGD is rarely used for training neural networks. It is important to determine whether a similar theoretical analysis holds for modern optimizers like Adam.\n\n- In Section 5.1, to fine-tune the model, the LibriSpeech dataset and handcrafted canaries were synthetically used as benign and dragger samples, respectively. However, this setup seems artificial. In a more realistic scenario where only the LibriSpeech dataset is used, how would dragger samples be detected in practice?  Including practical strategies for detecting draggers would add relevance to the findings.\n\n**Experiments**\n- In Tables 2 and 3, experiments were conducted with only batch sizes of 1, 4, and 512, which is insufficient. The authors should have included batch sizes such as 1, 4, 16, 64, 256, and 1024 to provide a more comprehensive evaluation. If these additional results are not feasible, explaining the rationale behind the specific batch sizes chosen would help clarify this decision.\n\n- The analysis related to $|| \\mu || \\le C || g||$ is completely missing in Section 5.2. I suggest the authors include this analysis in the section or provide an explanation if it was intentionally omitted. \n\n- How does micro-batch clipping perform when training from scratch instead of fine-tuning? This could offer valuable insights into its effectiveness in different training scenarios.\n\n- When conducting the language task experiments, reporting validation performance is inappropriate. It would have been better to test on a separate dataset. Additionally, the paper states that validation set information was not used at all, so how was hyperparameter optimization carried out? Clarifying this would improve transparency.\n\n- The experimental results shown in Table 5 reveal little to no improvement in performance. It would be helpful to discuss possible reasons for this outcome and consider further refinements or additional evaluation metrics.\n\n**Typo**\n- The term \"nominator\" in line 395 should be corrected to \"numerator.\"\n\n- The equation $ C = \\sqrt{\\frac{2(1-\\epsilon)\\eta - (1 - \\epsilon^2) L \\eta^2}{2 \\epsilon^2 L \\eta^2}}$ in line 219 should be updated to $ C = \\sqrt{\\frac{2(1-\\epsilon)\\eta - (1 - \\epsilon)^2 L \\eta^2}{2 \\epsilon^2 L \\eta^2}}. $\n\n- Additionally, the expression $ \\frac{2(1-\\epsilon)\\sigma^2}{LB} $ in line 232 should be updated to $ \\frac{(1-\\epsilon)\\sigma^2}{2LB}. $\n\n[1] Huo et al. \"Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling.\" AAAI 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)