Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation

## Abstract

Abstract Micro-batch clipping, a gradient clipping method, has recently shown potential in enhancing auto-speech recognition (ASR) model performance.
However, the underlying mechanism behind this improvement remains mysterious, particularly the observation that only certain micro-batch sizes are beneficial.
In this paper, we make the first attempt to explain this phenomenon.
Inspired by recent data pruning research, we assume that specific training samples may impede model convergence during certain training phases.
Under this assumption, the convergence analysis shows that micro-batch clipping can improve the convergence rate asymptotically at the cost of an additional constant bias that does not diminish with more training iterations.
The bias is dependent on a few factors and can be minimized at specific micro-batch size, thereby elucidating the existence of the sweet-spot micro-batch size observed previously.
We also verify the effectiveness of micro-batch clipping beyond speech models on vision and language models, and show promising performance gains in these domains.
An exploration of potential limitations shows that micro-batch clipping is less effective when training data originates from multiple distinct domains.

## 1 Introduction

*Micro-batch clipping* [, ] was initially introduced as a memory optimization technique for differentially private stochastic gradient descent (DP-SGD) [].
The method groups per-example gradients within a mini-batch into smaller micro-batches, clips the average gradient of each micro-batch, and updates the model with the mean of the clipped micro-batch gradients to avoid materializing all per-example gradients in memory.
Recent studies [, ] have surprisingly revealed that micro-batch clipping can enhance the performance of automatic speech recognition (ASR) models when used outside DP-SGD.
This finding has sparked interest in exploring micro-batch clipping as a standalone optimization technique, offering potential benefits in model performance.
However, the underlying mechanism driving this improvement remains poorly understood.

In this work, we aim to elucidate the behavior of micro-batch clipping through a combination of theoretical analysis and empirical evaluation.
Specifically, we conceptualize micro-batch clipping as a specialized form of data pruning [].
Unlike traditional data pruning techniques, which deterministically exclude redundant data, micro-batch clipping adaptively suppresses samples that hinder convergence, referred to as “*draggers*”, recognizing that a data sample’s helpfulness can change throughout training.
Guided by this intuition, we introduce Assumption 4.4 to capture certain properties of the draggers’ gradients, which are later empirically verified in Section 5.1.
Based on the assumption, we analyze the convergence-to-stationary-points rate for both standard SGD and micro-batch clipping on smooth loss manifolds and summarize the results in in Table 1.
We can observe that the introduction of draggers slows down the convergence rate for SGD by a constant factor, while the usage of micro-batch clipping *asymptotically accelerates the convergence rate*, at the cost of an additional constant bias term.
The bias term provides an explanation for the phenomenon observed in prior works [, ] where performance improvements are only seen with specific micro-batch sizes.
Concretely, because this term does not vanish with increasing iterations, the performance gain is contingent upon the magnitude of this term being sufficiently small.
According to Theorem 4.2, only certain micro-batch sizes minimize the bias term, thus explaining this effect.

**Table 1: Convergence Rate**
|  | Dragger | Clipping | Convergence Rate |
| --- | --- | --- | --- |
| Thm 9 [] | ✗ | ✗ | $\frac{1}{T^{1/4}}\sqrt{2L(\mathcal{L}_{0}-\mathcal{L}_{*})+\frac{\sigma_{b}^{2<br>}}{B}}$ |
| Thm 4.1 | ✓ | ✗ | $\frac{1}{T^{1/4}}\sqrt{\frac{4L}{(1-\epsilon)^{2}}(\mathcal{L}_{0}-\mathcal{L}<br>_{*})+\frac{2\sigma^{2}}{(1-\epsilon)B}}$ |
| Thm 4.2 | ✓ | ✓ | $\frac{\sigma}{(\sqrt{b}\epsilon-\sqrt{\epsilon(1-\epsilon)})\cdot(1+c)}+\frac{<br>1}{\sqrt{T}}\cdot\mathcal{O}(\mathcal{L}_{0}-\mathcal{L}_{*}+\frac{1}{2L})$ |

To test the effectiveness of micro-batch clipping outside the speech domain, we apply it to vision and language models and observe promising performance improvements.
Another interesting observation is that micro-batch clipping’s efficacy diminishes when facing multi-domain training data, particularly when data sizes between domains are unbalanced.
This may be attributed to the method hindering the convergence of domains with fewer samples by treating them as draggers.

## 2 Background

### 2.1 Micro-batch Clipping

While originally introduced in [] as a transition from record-level differential privacy to user-level differential privacy in federated learning, micro-batch clipping’s widespread adoption in differential privacy libraries stems primarily from its memory efficiency [, , ].
By requiring fewer gradients to be materialized, it reduces memory consumption, especially for large models, at the expense of a lower signal-to-noise ratio under the same differential privacy budget.
Notably, per-core clipping (where the micro-batch size equals the per-core batch size) achieves complete memory parity with non-private training when data parallelism is employed.

The potential of micro-batch clipping as an optimization technique to improve model performance emerged from observations in [, ], where it’s shown to reduce word error rate (WER) when training Conformer-based ASR models [].
However, whether this benefit generalizes beyond specific model architectures and datasets remains an open question that this work seeks to address.

### 2.2 Data Pruning & Active Learning

A growing body of research [, , ] highlights the detrimental effect of unhelpful or even harmful training data on model performance, particularly in large-scale datasets.
Sorscher *et al.* [] propose a metric-based approach to prune such data and alleviate inefficient power law scaling in theory.
Although the approach can effectively reduce the size of the data to store, the method overlooks the dynamic nature of data utility, which can evolve throughout the training process.

Active learning [, , , , , , ] offers an alternative approach to address this challenge by actively selecting the most beneficial training samples.
Our work shares similarities with active learning in that data importance is determined dynamically during training based on gradient information.
However, our approach is more fine-grained, implicitly adjusting data influence through adaptive gradient clipping rather than explicit sample selection.

## 3 Methodology: Adaptive Micro-batch Clipping

This section formally defines micro-batch clipping, focusing on the adaptive variant proposed in [], where the minimal L2 norm of all average micro-batch gradients is used as the clipping bound, thus ensuring that all micro-batch gradients are clipped.
We choose this variant because it avoids introducing clipping bound as an extra hyper-parameter, which requires further tuning, and [] shows that it performs comparably to clipping with manually selected clipping bound.
For brevity, subsequent references to "micro-batch clipping" will refer to this adaptive variant.

Let $\mathcal{D}$ be the training set and we want to run stochastic gradient descent with adaptive micro-batch clipping.
The mini-batch size is $B$ and learning rate is $\eta$.
In each iteration, a mini-batch of training examples are picked and sharded into micro-batches of size $b$.
The average gradient on each micro-batch will be calculated and clipped using the minimum L2 norm within all the micro-batch gradients.
Then the clipped micro-batch gradients are summed to get the mini-batch gradient which is then used to update the model (or passed to the optimizer if adaptive optimizers are used.).
The process is formalized in Algorithm 1.

Figure: Algorithm 1 Pseudocode for SGD with adaptive micro-batch clipping. $\bm{g}_{t}$ represents mini-batch gradients in the $t^{th}$ iteration. $\hat{\bm{g}}_{t}^{j}$ represents the $j^{th}$ micro-batch’s gradient in the $t^{th}$ iteration.

## 4 Convergence Analysis

In this section, we provide the convergence-to-stationary-points analysis for SGD micro-batch clipping following [] and show that it achieves asymptotically faster convergence at the cost of a constant bias under certain assumptions.

### 4.1 Assumptions

Our analysis is based on a few assumptions, as listed below.
Among these, Assumptions 4.1 and 1 are standard assumptions adopted from previous work [, ].
Assumption 4.3 is based on prior research [, ], with a slight modification to accommodate Assumption 4.4, a new assumption that we introduce in this work to capture the existence of dragger examples.

###### Assumption 4.1 (Lower bound of loss [ 4 ] ) .

$\forall\bm{w},\exists$ constant $\mathcal{L}_{*},$ s.t. $\mathcal{L}(\bm{w})\geq\mathcal{L}_{*}$.

###### Assumption 4.2 (Smoothness [ 4 ] ) .

Define $\bm{g}(\bm{w}):=\frac{\partial\mathcal{L}(\bm{w})}{\partial\bm{w}}$. Then $\forall\bm{w},\textbf{v},\exists$ constant $L\geq 0$ s.t.

$$ $\mathcal{L}(\textbf{v})-[\mathcal{L}(\bm{w})+\bm{g}(\bm{w})^{\top}(\textbf{v}- \bm{w})]\leq\frac{L}{2}\|\bm{w}-\textbf{v}\|^{2}.$ (1) $$

###### Assumption 4.3 (Gradient distribution [ 4 ] ) .

With probability $1-\epsilon$, the per-example gradient $\tilde{\bm{g}}$ is an i.i.d. symmetric unbiased estimator of $\bm{g}$ with bounded variance $\sigma^{2}$[].

$$ $w.p.~{}1-\epsilon,\mathbb{E}[\tilde{\bm{g}}]=\bm{g},\mathbb{E}[\|\tilde{\bm{g} }-\bm{g}\|^{2}]\leq\sigma^{2}$ $$

With probability $\epsilon$, the per-example gradient is a dragger

$$ $w.p.~{}\epsilon,\tilde{\bm{g}}=\bm{\mu}$ $$

Assumption 4.4 is introduced to formalize the observation that certain gradients within the training data may impede the model’s convergence.
First, we assume that these “dragger” gradients are orthogonal to the gradients of benign examples, following the intuition that the cosine similarity between data from different domains should be low.
The intuition is both motivated and supported by empirical evidence presented in Section 5.1.
Second, we assume the L2 norm of the dragger gradient is both lower- and upper-bounded by the L2 norm of the expected benign gradient scaled by a constant factor.
The lower-bound captures the intuition that dragger gradients should be large enough to slow down the convergence.
Empirical evidence for the intuition is presented in Section 5.3.
The upper bound captures the fact that these detrimental gradients are not the result of adversarial manipulation but arise naturally within the vast and diverse training dataset.

###### Assumption 4.4 (Dragger) .

The dragger gradients are orthogonal to the benign gradient subspace.
Furthermore, we assume that the ratio of the dragger’s norm to the expected benign gradient norm is bounded both above and below.

$$ $\bm{\mu}\perp\tilde{\bm{g}},c\|\bm{g}\|\leq\|\bm{\mu}\|\leq C\|\bm{g}\|$ $$

###### Assumption 4.1 (Lower bound of loss [ 4 ] ) .

###### Assumption 4.2 (Smoothness [ 4 ] ) .

###### Assumption 4.3 (Gradient distribution [ 4 ] ) .

###### Assumption 4.4 (Dragger) .

### 4.2 Convergence Rate of Standard SGD with Dragger Gradients

We first provide the convergence to stationary point of standard SGD result under the assumptions above.

###### Theorem 4.1 .

Under Assumption 4.1, 1, 4.3 (without the symmetry assumption), 4.4, running SGD for $T$ iterations gives, for $\eta=\frac{1}{L\sqrt{T}}$,

$$ $\min_{t}\mathbb{E}{\|\bm{g}_{t}\|}\leq\frac{1}{T^{1/4}}\sqrt{\frac{4L}{(1- \epsilon)^{2}}(\mathcal{L}_{0}-\mathcal{L}_{*})+\frac{2\sigma^{2}}{(1-\epsilon )B}}$ $$

###### Proof.

In the standard SGD,

$$ $\bm{w}_{t+1}=\bm{w}_{t}-\frac{\eta}{B}\sum\tilde{\bm{g}}_{t,i}$ $$

where $\tilde{\bm{g}}_{t,i}$ here represents $i^{th}$per-example gradient sampled from the distribution described by Assumption 4.3 and 4.4.

By Assumption 1,

$$ $\begin{split}\mathcal{L}_{t+1}-\mathcal{L}_{t}&\leq\bm{g}_{t}^{\top}(\bm{w}_{t +1}-\bm{w}_{t})+\frac{L}{2}\|\bm{w}_{t+1}-\bm{w}_{t}\|^{2}\\ &=-\frac{\eta\bm{g}_{t}^{\top}}{B}(\sum_{\text{benign}}\tilde{\bm{g}}_{t,i}+ \sum_{\text{dragger}}\tilde{\bm{g}}_{t,i})+\frac{L\eta^{2}}{2B^{2}}\|\sum_{ \text{benign}}\tilde{\bm{g}}_{t,i}+\sum_{\text{dragger}}\tilde{\bm{g}}_{t,i}\| ^{2}\end{split}$ $$

The expected improvement at one iteration is

$$ $\begin{split}\mathbb{E}[\mathcal{L}_{t+1}-\mathcal{L}_{t}|\bm{w}_{t}]&\leq-(1- \epsilon)\eta\bm{g}_{t}^{\top}\mathbb{E}_{\text{benign}}[\tilde{\bm{g}}_{t,i}] -\epsilon\eta\bm{g}_{t}^{\top}\mathbb{E}_{\text{dragger}}[\tilde{\bm{g}}_{t,i} ]+\frac{L\eta^{2}}{2B^{2}}\mathbb{E}\|\sum_{\text{benign}}\tilde{\bm{g}}_{t,i} +\sum_{\text{dragger}}\tilde{\bm{g}}_{t,i}\|^{2}\\ &=-(1-\epsilon)\eta\|\bm{g}_{t}\|^{2}+\frac{L\eta^{2}}{2B^{2}}\mathbb{E}\|\sum _{\text{benign}}\tilde{\bm{g}}_{t,i}\|^{2}+\frac{L\eta^{2}}{2B^{2}}\mathbb{E} \|\sum_{\text{dragger}}\tilde{\bm{g}}_{t,i}\|^{2}\\ &\leq-(1-\epsilon)\eta\|\bm{g}_{t}\|^{2}+\frac{(1-\epsilon)^{2}L\eta^{2}}{2}( \|\bm{g}_{t}\|^{2}+\frac{\sigma^{2}}{(1-\epsilon)B})+\frac{\epsilon^{2}L\eta^{ 2}}{2}\|\bm{\mu}_{t}\|^{2}\\ &\leq\bigl{(}\frac{(1-\epsilon)^{2}L\eta^{2}}{2}-(1-\epsilon)\eta+\frac{ \epsilon^{2}L\eta^{2}C^{2}}{2}\bigr{)}\|\bm{g}_{t}\|^{2}+\frac{(1-\epsilon)L \eta^{2}\sigma^{2}}{2B}\end{split}$ $$

The second equation follows from Assumption 4.4 that the benign gradients are perpendicular to the dragger gradients.
The fourth inequality is based on Assumption 4.4 that the dragger gradient norm is upper bounded.
To make sure the coefficient is negative, we require that

$$ $\begin{split}\frac{(1-\epsilon)^{2}L\eta^{2}}{2}-(1-\epsilon)\eta+\frac{ \epsilon^{2}L\eta^{2}C^{2}}{2}<0\Rightarrow C<\sqrt{\frac{2(1-\epsilon)\eta-(1 -\epsilon^{2})L\eta^{2}}{\epsilon^{2}L\eta^{2}}}\end{split}$ $$

For arithmetic convenience, we choose $C=\sqrt{\frac{2(1-\epsilon)\eta-(1-\epsilon^{2})L\eta^{2}}{2\epsilon^{2}L\eta^
{2}}}$.
Now we do a telescoping sum over the iterations

$$ $\begin{split}\mathcal{L}_{0}-\mathcal{L}_{*}&\geq\mathcal{L}_{0}-\mathbb{E} \mathcal{L}_{T}=\sum_{t}\mathbb{E}(\mathcal{L}_{t}-\mathcal{L}_{t+1})\\ &\geq\bigl{(}\frac{(1-\epsilon)\eta}{2}-\frac{(1-\epsilon)^{2}L\eta^{2}}{4} \bigr{)}\mathbb{E}(\sum_{t}\|\bm{g}_{t}\|^{2})-\frac{(1-\epsilon)L\eta^{2} \sigma^{2}T}{2B}\end{split}$ $$

We apply the same learning rate as in [] $\eta=\frac{1}{L\sqrt{T}}$.

$$ $\begin{split}\mathcal{L}_{0}-\mathcal{L}_{*}&\geq(\frac{1-\epsilon}{2L\sqrt{T} }-\frac{(1-\epsilon)^{2}}{4LT})\mathbb{E}(\sum_{t}\|\bm{g}_{t}\|^{2})-\frac{2( 1-\epsilon)\sigma^{2}}{LB}\\ &\geq\frac{(1-\epsilon)^{2}}{4L\sqrt{T}}\mathbb{E}(\sum_{t}\|\bm{g}_{t}\|^{2}) -\frac{2(1-\epsilon)\sigma^{2}}{LB}\end{split}$ $$

and finally

$$ $\min_{t}\mathbb{E}\|\bm{g}_{t}\|^{2}\leq\frac{1}{\sqrt{T}}\bigl{(}\frac{4L}{(1 -\epsilon)^{2}}(\mathcal{L}_{0}-\mathcal{L}_{*})+\frac{2\sigma^{2}}{(1- \epsilon)B}\bigr{)}$ $$

Using Jensen’s inequality, we can have

$$ $\begin{split}\min_{t}\mathbb{E}\|\bm{g}_{t}\|&\leq\frac{1}{T^{1/4}}\sqrt{\frac {4L}{(1-\epsilon)^{2}}(\mathcal{L}_{0}-\mathcal{L}_{*})+\frac{2\sigma^{2}}{(1- \epsilon)B}}\end{split}$ $$

∎

In comparison to Theorem 9 of [], SGD with draggers demonstrates the same asymptotic convergence rate, albeit with a larger constant coefficient.
This observation aligns with our intuition that the presence of dragger gradients can hinder the model’s convergence speed.

###### Theorem 4.1 .

###### Proof.

### 4.3 Convergence Rate of Micro-batch Clipped SGD with Dragger Gradients

In this subsection, we present the convergence analysis for micro-batch clipping.
To simplify the algebra, we assume a uniform lower bound for the clipping bound ($\rho_{t}$) used in each iteration.
Note that this assumption doesn’t impact the asymptotic behavior.
For conciseness, the proofs of the supporting lemmas are provided in the appendix.

###### Theorem 4.2 .

Under Assumption 4.1, 1, 4.3, 4.4, running micro-batch clipped SGD for $T$ iterations gives, for $\eta=\frac{1}{L\sqrt{T}}$, micro-batch size $b$ and dragger probability $\epsilon\geq\frac{1}{b+1}$,

$$ $\begin{split}\min_{t}\mathbb{E}{\|\bm{g}_{t}\|}&\leq\frac{\sigma}{(\sqrt{b} \epsilon-\sqrt{\epsilon(1-\epsilon)})\cdot(1+c)}\\ &\quad\quad\quad\quad\quad\quad\quad\quad+\frac{1}{\sqrt{T}}\cdot\frac{\sqrt{b \epsilon}}{\sqrt{b\epsilon}-\sqrt{1-\epsilon}}\cdot\frac{2L(1+2\epsilon C)}{1- \epsilon}\cdot(\mathcal{L}_{0}-\mathcal{L}_{*}+\frac{1}{2L})\end{split}$ $$

###### Proof Sketch.

In the micro-batch clipped SGD with micro-batch size $b$, the update rule is as follow:

$$ $\bm{w}_{t+1}=\bm{w}_{t}-\frac{\eta b}{B}\sum_{i}\frac{\hat{\bm{g}}_{t,i}}{\| \hat{\bm{g}}_{t,i}\|}$ (2) $$

Note that here $\hat{\bm{g}}_{t,i}$ represents $i^{th}$ micro-batch gradient in the $t^{th}$ iteration.
For notation simplicity, we omit the subscripts if it’s clear from context.

By plugging in the update from Equation 2 in the smoothness assumption 1 and take conditional expectation, the expected improvement at one iteration is

$$ $\begin{split}\mathbb{E}[\mathcal{L}_{t+1}-\mathcal{L}_{t}|\bm{w}_{t}]&\leq- \eta\bm{g}_{t}^{\top}\mathbb{E}\frac{\hat{\bm{g}}_{t}}{\|\hat{\bm{g}}_{t}\|}+ \frac{L\eta^{2}}{2}\end{split}$ $$

Re-write $\hat{\bm{g}}_{t}=(1-\epsilon)\bm{g}_{t}+\epsilon\bm{\mu}_{t}+\Delta_{t}$, where $\Delta_{t}$ is a zero-centered random variable whose expected L2 norm is bounded as below.
The proof can be found in 7.

###### Lemma 4.3 .

The expected L2 norm of $\Delta_{t}$ is upper-bounded by.

$$ $\mathbb{E}\|\Delta_{t}\|\leq\sqrt{\frac{\epsilon(1-\epsilon)\|\bm{g}_{t}\|^{2} +\epsilon(1-\epsilon)\|\bm{\mu}_{t}\|^{2}+(1-\epsilon)\sigma^{2}}{b}}$ $$

.

To lower bound $\bm{g}_{t}^{\top}\mathbb{E}\frac{\hat{\bm{g}}_{t}}{\|\hat{\bm{g}}_{t}\|}$, we follow [] to use the hyperplane perpendicular to $\bm{g}_{t}$ to divide the support of $\Delta_{t}$ into two half-spaces where denote the positive half as: $H_{+}:=\{v:\bm{g}^{\top}v>0\}$.
Then using the symmetry assumption 4.3, we have

$$ $\begin{split}\bm{g}_{t}^{\top}&\mathbb{E}\frac{\hat{\bm{g}}_{t}}{\|\hat{\bm{g} }_{t}\|}=\mathbb{E}\bigl{(}\frac{(1-\epsilon)\|\bm{g}_{t}\|^{2}+\bm{g}_{t}^{ \top}\Delta_{t}}{\|(1-\epsilon)\bm{g}_{t}+\epsilon\bm{\mu}_{t}+\Delta_{t}\|} \bigr{)}\\ &=\frac{1}{2}\mathbb{E}\bigl{(}\frac{(1-\epsilon)\|\bm{g}_{t}\|^{2}+\bm{g}_{t} ^{\top}\Delta_{t}}{\|(1-\epsilon)\bm{g}_{t}+\epsilon\bm{\mu}_{t}+\Delta_{t}\|} \big{|}\Delta_{t}\in H_{+}\bigr{)}+\frac{1}{2}\mathbb{E}\bigl{(}\frac{(1- \epsilon)\|\bm{g}_{t}\|^{2}-\bm{g}_{t}^{\top}\Delta_{t}}{\|(1-\epsilon)\bm{g}_ {t}+\epsilon\bm{\mu}_{t}-\Delta_{t}\|}\big{|}\Delta_{t}\in H_{+}\bigr{)}\\ &=\frac{1}{2}\mathbb{E}\bigl{(}\underbrace{\frac{(1-\epsilon)\|\bm{g}_{t}\|^{2 }+\bm{g}_{t}^{\top}\Delta_{t}}{\|(1-\epsilon)\bm{g}_{t}+\epsilon\bm{\mu}_{t}+ \Delta_{t}\|}+\frac{(1-\epsilon)\|\bm{g}_{t}\|^{2}-\bm{g}_{t}^{\top}\Delta_{t} }{\|(1-\epsilon)\bm{g}_{t}+\epsilon\bm{\mu}_{t}-\Delta_{t}\|}}_{\star}\big{|} \Delta_{t}\in H_{+}\bigr{)}\\ &=\frac{1}{2}\mathbb{E}\bigl{(}\star\big{|}\Delta_{t}\in H_{+},\|\Delta_{t}\| \leq\epsilon\|\bm{\mu}_{t}\|+\epsilon\|\bm{g}_{t}\|\bigr{)}\mathbb{P}(\|\Delta \|\leq\epsilon\|\bm{\mu}_{t}\|+\epsilon\|\bm{g}_{t}\|)\\ &\quad\quad\quad\quad\quad\quad\quad+\frac{1}{2}\mathbb{E}\bigl{(}\star\big{|} \Delta_{t}\in H_{+},\|\Delta_{t}\|>\epsilon\|\bm{\mu}_{t}\|+\|\bm{g}_{t}\| \bigr{)}\mathbb{P}(\|\Delta_{t}\|>\epsilon\|\bm{\mu}_{t}\|+\epsilon\|\bm{g}_{t }\|)\end{split}$ $$

Lemma 4.4 tells us that $\star$ is always non-negative under Assumption 4.4, and thus we can only keep the first term.
The proof is provided in Appendix 8.

###### Lemma 4.4 .

If $\Delta_{t}\in H_{+}$ and $\bm{\mu}_{t}\perp\bm{g}_{t},\bm{\mu}_{t}\perp\Delta_{t}$, then $\frac{(1-\epsilon)\|\bm{g}_{t}\|^{2}+\bm{g}_{t}^{\top}\Delta_{t}}{\|(1-
\epsilon)\bm{g}_{t}+\epsilon\bm{\mu}_{t}+\Delta_{t}\|}+\frac{(1-\epsilon)\|\bm
{g}_{t}\|^{2}-\bm{g}_{t}^{\top}\Delta_{t}}{\|(1-\epsilon)\bm{g}_{t}+\epsilon
\bm{\mu}_{t}-\Delta_{t}\|}\geq 0$.

$$ $\begin{split}\bm{g}_{t}^{\top}&\mathbb{E}\frac{\hat{\bm{g}}_{t}}{\|\hat{\bm{g} }_{t}\|}\geq\frac{1}{2}\mathbb{E}\bigl{(}\star\big{|}\Delta_{t}\in H_{+},\| \Delta_{t}\|\leq\epsilon\|\bm{\mu}_{t}\|+\epsilon\|\bm{g}_{t}\|\bigr{)}\mathbb {P}(\|\Delta_{t}\|\leq\epsilon\|\bm{\mu}_{t}\|+\epsilon\|\bm{g}_{t}\|)\\ &\geq\frac{1}{2}\mathbb{E}\bigl{(}\frac{(1-\epsilon)\|\bm{g}_{t}\|^{2}}{(1- \epsilon)\|\bm{g}_{t}\|+\epsilon\|\bm{\mu}_{t}\|+\|\Delta_{t}\|}\big{|}\Delta_ {t}\in H_{+},\|\Delta_{t}\|\leq\epsilon\|\bm{\mu}_{t}\|+\epsilon\|\bm{g}_{t}\| \bigr{)}(1-\frac{\mathbb{E}\|\Delta_{t}\|}{\epsilon\|\bm{\mu}_{t}\|+\epsilon\| \bm{g}_{t}\|})\\ &\geq\frac{1}{2}\cdot\frac{(1-\epsilon)\|\bm{g}_{t}\|^{2}}{\|\bm{g}_{t}\|+2 \epsilon\|\bm{\mu}_{t}\|}\cdot(1-\frac{1}{\sqrt{b}}\cdot\frac{\sqrt{\epsilon(1 -\epsilon)\|\bm{g}_{t}\|^{2}+\epsilon(1-\epsilon)\|\bm{\mu}_{t}\|^{2}+(1- \epsilon)\sigma^{2}}}{\epsilon\|\bm{\mu}_{t}\|+\epsilon\|\bm{g}_{t}\|})\\ &\geq\frac{1-\epsilon}{2(1+2\epsilon C)}\cdot\bigl{(}(1-\sqrt{\frac{1-\epsilon }{b\epsilon}})\|\bm{g}_{t}\|-\frac{1}{\sqrt{b}}\cdot\frac{\sigma}{\epsilon(1+c )}\bigr{)}\end{split}$ $$

, where the second inequality follows from Markov’s inequality.

Thus we have

$$ $\begin{split}\mathbb{E}[\mathcal{L}_{t+1}-\mathcal{L}_{t}|\bm{w}_{t}]&\leq- \eta\cdot\frac{1-\epsilon}{2(1+2\epsilon C)}\cdot\bigl{(}(1-\sqrt{\frac{1- \epsilon}{b\epsilon}})\|\bm{g}_{t}\|-\frac{1}{\sqrt{b}}\cdot\frac{\sigma}{ \epsilon(1+c)}\bigr{)}+\frac{L\eta^{2}}{2}\end{split}$ $$

Now we do a telescoping sum over the iterations

$$ $\begin{split}\mathcal{L}_{0}-\mathcal{L}_{*}&\geq\mathcal{L}_{0}-\mathbb{E} \mathcal{L}_{T}=\sum_{t}\mathbb{E}(\mathcal{L}_{t}-\mathcal{L}_{t+1})\\ &\geq\eta\cdot\frac{1-\epsilon}{2(1+2\epsilon C)}\cdot\bigl{(}(1-\sqrt{\frac{1 -\epsilon}{b\epsilon}})\|\bm{g}_{t}\|-\frac{1}{\sqrt{b}}\cdot\frac{\sigma}{ \epsilon(1+c)}\bigr{)}-\frac{L\eta^{2}T}{2}\end{split}$ $$

We apply the same learning rate as in [] $\eta=\frac{1}{L\sqrt{T}}$.

$$ $\begin{split}\mathcal{L}_{0}-\mathcal{L}_{*}&\geq\frac{1-\epsilon}{2L\sqrt{T}( 1+2\epsilon C)}\cdot\bigl{(}(1-\sqrt{\frac{1-\epsilon}{b\epsilon}})\|\bm{g}_{t }\|-\frac{1}{\sqrt{b}}\cdot\frac{\sigma}{\epsilon(1+c)}\bigr{)}-\frac{1}{2L} \end{split}$ $$

and finally

$$ $\begin{split}\min_{t}\mathbb{E}\|\bm{g}_{t}\|\leq\min_{t}\frac{1}{T}\mathbb{E} \sum_{t}\|\bm{g}_{t}\|\leq&\frac{\sigma}{(\sqrt{b}\epsilon-\sqrt{\epsilon(1- \epsilon)})\cdot(1+c)}\\ &+\frac{1}{\sqrt{T}}\cdot\frac{\sqrt{b\epsilon}}{\sqrt{b\epsilon}-\sqrt{1- \epsilon}}\cdot\frac{2L(1+2\epsilon C)}{1-\epsilon}\cdot(\mathcal{L}_{0}- \mathcal{L}_{*}+\frac{1}{2L})\end{split}$ $$

∎

###### Theorem 4.2 .

###### Proof Sketch.

###### Lemma 4.3 .

###### Lemma 4.4 .

## 5 Empirical Evaluation

In this section, we’d like to answer the following 3 questions: 1) Does Assumption 4.4 hold in practice? 2) Does the hypothesis on $c$ to explain the sweet spot of micro-batch size hold in practice? 3) Does micro-batch clipping help improve performance beyond speech tasks?

### 5.1 Empirical Evidence for Assumption 4.4

##### Experiment Setup.

To answer question 1), we adopt the experimental setup from Wang *et al.* [].
Specifically, we fine-tune 600M Conformer XL [] models on the LibriSpeech dataset [] and handcrafted canaries [].
The model’s encoder is pre-trained using BEST-RQ [] on the LibriLight dataset [].
The key advantage of this setup is the ability to treat the inserted canaries as surrogate draggers, thereby circumventing the technical challenge of identifying natural draggers in large models.

We compute the cosine similarity between 100 randomly selected pairs of dragger and benign gradients.
For calibration purposes, we also compute the cosine similarity between 100 pairs of benign gradients.

Figure: Figure 1: Cosine Similarity between Crafted Dragger and Benign Examples.

##### Evaluation Results.

The results are summarized in Figure 1.
We observe that the magnitude of the cosine similarity between dragger and benign gradients is significantly smaller than the cosine similarity within benign gradients, with a negligible standard deviation.
This observation supports Assumption 4.4 that dragger gradients are orthogonal to the benign gradient subspace.

### 5.2 How c 𝑐 c italic_c Changes with Micro-batch Size?

##### Experiment Setup.

Again we re-use the setup in [].
First, we run micro-batch clipping with micro-batch size $1$, $4$ (per-core batch size), $512$ (mini-batch size) to substantiate the existence of an optimal micro-batch size that yields maximum performance.
Performance is assessed using word error rate (WER) on two splits of the LibriSpeech test dataset: test-clean, consisting of relatively clean utterances, and test-other, consisting of noisier utterances.

In the second step, the following proxy estimation is used to monitor the fluctuations in the value of $c$.

$$ $\hat{c}:=\frac{\text{mean}(\|\text{benign gradients}\|)}{\text{mean}(\|\text{ dragger gradients}\|)}$ $$

To estimate the nominator, we use the mean norm of 100 LibriSpeech samples after trimming the largest 10% to avoid the effect of hard-to-track natural draggers.
To estimate the denominator, we use the mean of the gradient norms of 100 inserted canaries.
The proxy $\hat{c}$ is computed for both non-private training and micro-batch clipping scenarios with micro-batch sizes $1$, $4$ and $512$.

Figure: Figure 2: Convergence curve for LibriSpeech experiments. Both x-axis and y-axis are log-scale to highlight the convergence speed advantage.

##### Evaluation Results.

First, in Table 2, we can observe that when micro-batch size is 4, the model achieves the best performance across all settings, improving test-other WER by 4.8% relatively and test-clean by 4.1% relatively.
Furthermore, we note that micro-batch clipping does not enhance performance when the micro-batch size is either 1 or 512. Instead, these configurations degrade WER on both the test-other and test-clean splits.
As per Theorem 4.2, this degradation is attributed to the presence of a large constant term when the micro-batch size is not optimally selected.
Furthermore, we note that micro-batch clipping does not enhance performance when the micro-batch size is either 1 or 512. Instead, these configurations degrade WER on both the test-other and test-clean splits. As per Theorem 4.2, this degradation is attributed to the presence of a large constant term when the micro-batch size is not optimally selected.
Besides, from Figure 2, we can tell see the convergence rate advantage as predicted.

**Table 2: Best WER on LibriSpeech within 20K fine-tuning steps. Mean and standard deviation are calculated across 3 runs. Bold highlights best. The checkpoint is picked to optimize test-other split.**
| Test set | Baseline | Ours, $b=1$ | Ours, $b=4$ | Ours, $b=512$ |
| --- | --- | --- | --- | --- |
| test-clean | 1.93 $\pm$ 0.04 | 2.40 $\pm$ 0.13 | 1.85 $\pm$ 0.07 | 1.94 $\pm$ 0.00 |
| test-other | 3.99 $\pm$ 0.01 | 4.86 $\pm$ 0.24 | 3.80 $\pm$ 0.05 | 4.03 $\pm$ 0.03 |

Second, the values of $\hat{c}$ for different micro-batch sizes are shown in Table 3.
The results conform with the conjecture that $\hat{c}$ is decreasing with respect to the micro-batch size $b$.
This observation provides a strong evidence to support the explanation for the existence of the optimal micro-batch size.

*Remark.* We’d like to note that the difference in proxy $\hat{c}$ is still not big enough to level out the influence of $b$ in the constant term in Theorem 4.2, and this might be attributed to the several factors including the convergence rate bound not strictly tight, the noise in $\hat{c}$ measurement, and systematic bias between $\hat{c}$ and $c$.
However, we still believe that the trend of $\hat{c}$ is promising as an evidence for our explanation for sweet-spot micro-batch sizes, and we deem getting a stricter convergence bound and more accurate relationship between $b$ and $c$ important future directions to guide the choice of micro-batch sizes.

**Table 3: The values of $\hat{c}$ under different micro-batch sizes.**
| Micro-batch size | $b=1$ | $b=4$ | $b=512$ |
| --- | --- | --- | --- |
| $\hat{c}$ | 6.18 | 5.74 | 4.42 |

### 5.3 Is Micro-batch Clipping Effective for Models Beyond Speech?

To address question 3), we conduct an evaluation of micro-batch clipping on vision models and language models.
In accordance with the findings of [], we default to a micro-batch size equivalent to the per-core batch size due to its memory efficiency.
It is important to note that alternative micro-batch sizes may necessitate careful adjustment to mitigate potential memory overflow issues.

#### 5.3.1 Vision Tasks

##### Experiment Setup:

For vision tasks, we choose apply micro-batch clipping to the DeiT-B model proposed in [].
We train DeiT-B from scratch on the ImageNet dataset [] and achieves parity with the reported Top-1 accuracy in [].
The mini-batch size used is 4096 and the micro-batch size is 32, the same as the per-core batch size for computational efficiency.

##### Evaluation Results:

The results are summarized in Table 4.
Adding micro-batch clipping improves the Top-1 accuracy by 1.5% and Top-5 accuracy by 1.0%.

**Table 4: ViT trained w/ or w/o adaptive micro-batch clipping.**
|  | Top-1 accuracy (%) | Top-5 accuracy (%) |
| --- | --- | --- |
| Baseline | 81.8 | 95.8 |
| Ours | 83.3 | 96.8 |

#### 5.3.2 Language Tasks

##### Experiment Setup:

For language tasks, we fine-tune a T5 model [] on the superGlue dataset [].
The model is pre-trained on the C4 dataset [] for 1M steps.
The mini-batch size is 2048, and the mini-batch size is 16, the same as the per-core batch size.
Note that the test set for superGlue is private so we report the performance on the validation set.
Since we do not use any information of the validation set to improve the micro-batch clipping part, the improvement shown is fair.
Also we use accuracy across all subsets of superGlue to be able to compute the weighted average accuracy as a unified metric to compare 2 models.

##### Evaluation Results:

Table 5 summarizes our findings. While micro-batch clipping only marginally improves average accuracy (0.1%), it significantly boosts performance on the largest, unbalanced subset, ReCoRD (0.4%), while negatively impacting smaller datasets.
This suggests varying gradient distributions across subsets, with micro-batch clipping suppressing gradients from smaller subsets, which conforms with the observation that ReCoRD has lower accuracy when trained with other data mixed (83.9%) than when trained alone (84.5%).
Training solely on ReCoRD yields a 0.3% improvement as shown in Table 6, reinforcing this hypothesis.

**Table 5: T5 trained w/ or w/o adaptive micro-batch clipping.**
| Accuracy (%) | wt avg | BoolQ | CB | COPA | MultiRC | ReCoRD | RTE | WiC | WSC |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\|$Train$\|$ [] | - | 9427 | 250 | 400 | 5100 | 101k | 2500 | 6000 | 554 |
| Baseline | 85.9 | 93.4 | 97.8 | 88.9 | 93.4 | 83.9 | 97.6 | 89.5 | 70.3 |
| Ours | 86.0 | 91.8 | 95.6 | 88.3 | 91.6 | 84.3 | 95.7 | 88.5 | 70.3 |

These results highlight a crucial lesson: micro-batch clipping, while beneficial when data comes from the same domain but is noisy, can be detrimental with unbalanced multi-domain data.
In such scenarios, micro-batch clipping may hinder certain domains to favor others and should therefore be avoided.

**Table 6: T5 trained w/ or w/o adaptive micro-batch clipping.**
| Accuracy (%) | Baseline | Ours |
| --- | --- | --- |
| ReCoRD | 84.5 | 84.8 |

## 6 Conclusion & Discussion

In this paper, we revisit micro-batch clipping, originally proposed in the context of differential privacy, from the lens of data pruning, inspired by recent observations made by [, ].

Our convergence analysis demonstrates that micro-batch clipping can asymptotically accelerate the convergence rate for smooth loss functions. To elucidate the optimal micro-batch size falling between 1 and the mini-batch size, we introduce the concept of "dragger gradients." Combining this concept with our convergence analysis reveals a constant term minimized at a value between 1 and the mini-batch size.

Our analysis and explanation hinge on two novel hypotheses, which we empirically verify. Furthermore, we extend the application of micro-batch clipping to vision and language models. We find that when the input data is single-domain, micro-batch clipping can still enhance the performance of these models.

##### Limitations.

Despite the demonstrated effectiveness of micro-batch clipping across speech, vision and language models, our research reveals limitations that warrant further investigation.
As evidenced in Section 5.3.2, micro-batch clipping can adversely affect model performance when the input data originates from multiple distinct domains.
This constraint limits the applicability of micro-batch clipping in scenarios where data diversity is prevalent.

Furthermore, this limitation may raise potential fairness concerns, akin to other data pruning methods pointed out by [].
The preferential treatment of specific data domains could inadvertently introduce biases and inequities in model outcomes.
These limitations underscore the need for a refined micro-batch clipping approach that can effectively handle multi-domain data.

##### Future Work.

Several intriguing avenues for future research emerge from this paper.
Firstly, the memory overhead introduced by micro-batch clipping remains a challenge.
While data sharding with a micro-batch size equal to the per-core batch size [, ] can mitigate this issue, the growing size of modern models necessitates the development of alternative solutions to ensure the computational efficiency of micro-batch clipping in training scenarios with model sharding alone.

Secondly, from a theoretical perspective, a more rigorous relationship between micro-batch size $b$ and $c$ is needed, as it could guide the choice of the appropriate micro-batch size to optimize performance.

Finally, we are eager to explore a broader range of gradient-based data pruning methods.
This includes investigating different variations of clipping, as well as other innovative gradient manipulation techniques based on diverse heuristics.

## 7 Proof for Lemma 4.3

###### Proof.

First, we upper-bound the variance for the per-example gradients following the distribution described in Assumption 4.3.

$$ $\begin{split}\mathbb{E}\|\tilde{\bm{g}}-\mathbb{E}[\tilde{\bm{g}}]\|^{2}&= \mathbb{E}\|\tilde{\bm{g}}\|^{2}-\|\mathbb{E}\tilde{\bm{g}}\|^{2}\\ &\leq(1-\epsilon)\bigl{(}\|\bm{g}\|^{2}+\sigma^{2}\bigr{)}+\epsilon\|\bm{\mu} \|^{2}-\|(1-\epsilon)\bm{g}+\epsilon\bm{\mu}\|^{2}\\ &=\epsilon(1-\epsilon)\|\bm{g}\|^{2}+\epsilon(1-\epsilon)\|\bm{\mu}\|^{2}+(1- \epsilon)\sigma^{2}\end{split}$ $$

Second, after the micro-batch averaging,

$$ $\begin{split}\mathbb{E}\|\hat{\bm{g}}-\mathbb{E}[\hat{\bm{g}}]\|^{2}&=\frac{1} {b^{2}}\mathbb{E}\bigl{(}\|\sum(\tilde{\bm{g}}-\mathbb{E}\tilde{\bm{g}})\|^{2} \bigr{)}\\ &=\frac{1}{b^{2}}\sum\mathbb{E}\bigl{(}\|\tilde{\bm{g}}-\mathbb{E}\tilde{\bm{g }}\|^{2}\bigr{)}\\ &=\frac{\epsilon(1-\epsilon)\|\bm{g}\|^{2}+\epsilon(1-\epsilon)\|\bm{\mu}\|^{2 }+(1-\epsilon)\sigma^{2}}{b}\end{split}$ $$

, where the second equation follows from the fact that per-example gradients are independent from each other.
∎

###### Proof.

## 8 Proof for Lemma 4.4

###### Proof.

If $\bm{g}_{t}^{\top}\Delta_{t}\leq(1-\epsilon)\|\bm{g}_{t}\|^{2}$, then the lemma obviously holds, so we only need to prove for the case when $\bm{g}_{t}^{\top}\Delta_{t}>(1-\epsilon)\|\bm{g}_{t}\|^{2}$.

$$ $\begin{split}&\epsilon^{2}\|\bm{g}_{t}\|^{2}\|\bm{\mu}_{t}\|+\|\bm{g}_{t}\|^{2 }\|\Delta\|^{2}\geq(\bm{g}_{t}^{\top}\Delta_{t})^{2}\\ \Rightarrow&\frac{((1-\epsilon)\|\bm{g}_{t}\|^{2}+\bm{g}_{t}^{\top}\Delta_{t}) ^{2}}{(1-\epsilon)^{2}\|\bm{g}_{t}\|^{2}+\epsilon^{2}\|\bm{\mu}\|^{2}+\|\Delta _{t}\|^{2}+2(1-\epsilon)\bm{g}_{t}^{\top}\Delta_{t}}\\ &\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\geq\frac{((1-\epsilon)\|\bm{g}_{t}\|^{2}-\bm{g}_{t}^{\top}\Delta_{t})^{2 }}{(1-\epsilon)^{2}\|\bm{g}_{t}\|^{2}+\epsilon^{2}\|\bm{\mu}\|^{2}+\|\Delta_{t }\|^{2}-2(1-\epsilon)\bm{g}_{t}^{\top}\Delta_{t}}\\ \Leftrightarrow&\frac{(1-\epsilon)\|\bm{g}_{t}\|^{2}+\bm{g}_{t}^{\top}\Delta_{ t}}{\|(1-\epsilon)\bm{g}_{t}+\epsilon\bm{\mu}_{t}+\Delta_{t}\|}+\frac{(1- \epsilon)\|\bm{g}_{t}\|^{2}-\bm{g}_{t}^{\top}\Delta_{t}}{\|(1-\epsilon)\bm{g}_ {t}+\epsilon\bm{\mu}_{t}-\Delta_{t}\|}\geq 0\end{split}$ $$

∎

###### Proof.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: 1. Novel Conceptualization: The paper innovatively frames micro-batch clipping as a data pruning technique, introducing the concept of "dragger" gradients to explain its mechanism. This provides new insights into why and when micro-batch clipping works.
2. Comprehensive Analysis: The paper conducts both theoretical convergence analysis and extensive empirical evaluation to support its hypotheses. The results demonstrate the effectiveness of micro-batch clipping across different model architectures and datasets.
summary: This paper revisits micro-batch clipping, an optimization technique initially proposed for memory efficiency in differentially private stochastic gradient descent. The paper conceptualizes micro-batch clipping as a form of data pruning, proposing that certain "dragger" gradients impede model convergence. Through convergence analysis and empirical evaluation, the paper demonstrates that micro-batch clipping can asymptotically accelerate convergence but introduces a constant bias term. This bias term explains the existence of an optimal micro-batch size. The paper further validates the effectiveness of micro-batch clipping beyond speech models, showing promising results on vision and language tasks. However, limitations are identified when training data originates from multiple distinct domains.
weaknesses: 1. Lack of interpretability or robustness. This paper does not discuss the potential impact of micro-batch clipping on the interpretability or robustness of the trained models, which are important considerations in many real-world applications.
2. Limited scope of experiments: The experiments are primarily conducted on speech recognition models and limited vision and language tasks. The generalizability of the findings to a broader range of models and datasets is unclear. More extensive experimentation across different domains and model architectures would strengthen the conclusions.
3. Concern of memory overhead: While the paper mentions that data sharding with a micro-batch size equal to the per-core batch size can mitigate memory overhead, this may not be feasible for larger models. The growing size of modern models necessitates alternative solutions to ensure computational efficiency.
4. Rigorous relationship between micro-batch size and performance: The paper suggests that there exists an optimal micro-batch size that minimizes the constant bias term in the convergence analysis. However, the relationship between the micro-batch size b and the factor c is not rigorously established. A tighter convergence bound and a more accurate relationship between b and c could better guide the choice of micro-batch sizes in practice.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weakness section.
soundness: 3
strengths: - Uncovering the mechanism behind the performance improvements brought by micro-batch clipping is both interesting and valuable.
- The paper provides a series of theoretical foundation to analyze this mechanism and support its conclusions.
- The theoretical findings in this paper can explain empirical results, such as the performance gains achieved through micro-batch clipping and why these improvements occur only at specific micro-batch sizes.
- Extensive experiments are conducted to explore the benefits of micro-batch clipping beyond ASR.
summary: Micro-batch clipping has recently demonstrated potential in improving model performance in automatic speech recognition (ASR). This paper seeks to uncover the mechanism driving this enhancement. The authors assume that certain data samples ("draggers") may impede model convergence. Through theoretical analysis, they find that micro-batch clipping can accelerate convergence by diminishing the impact of these draggers, albeit introducing a constant bias dependent on micro-batch size. This insight further explains the observed phenomenon where improvements occur only at specific micro-batch sizes. Additional empirical experiments are conducted beyond ASR to investigate the effectiveness of micro-batch clipping.
weaknesses: - The Appendix of this paper is somewhat lost, and the full proofs of the theorems are thus invisible, requiring further verification.
- Assumption 4.4 posits that dragger gradients are orthogonal to the benign gradient subspace, which may be a restrictive assumption and not entirely realistic in practical scenarios.
- In Sec.1, the authors claim that micro-batch clipping can adaptively suppress samples that hinder convergence. However, it remains somewhat unclear in the theorem whether the effectiveness of micro-batch clipping directly stems from suppressing these draggers.
- It appears that as $\epsilon$ decreases, approaching zero and indicating a lower likelihood of draggers, the convergence rate bound in Thm 4.2 actually increases, which seems counterintuitive.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - How are micro-batches grouped within a mini-batch? Is this process conducted randomly?

- Around line 114, the text states that mini-batch gradients are "summed," but isn't it more accurate to describe this as "averaged" or "aggregated," given that the mean is typically computed?
soundness: 3
strengths: - The topic of theoretically analyzing the batch-size dependency issue of micro-batch clipping in training ASR models, which has shown empirical success, is both meaningful and interesting.

- Despite the mathematical rigor throughout the paper, the overall writing is easy to read and follow; in particular, both the content and proof of Theorem 4.1 are easy to understand.

- In the case of the ASR task, the paper presents promising empirical results that align well with the theoretical analysis.
summary: This paper presents a theoretical analysis of why the micro-batch clipping method, which has been shown to improve performance in automatic speech recognition (ASR) tasks, is only effective for specific micro-batch sizes. The authors first categorize mini-batch samples into benign samples and dragger samples, followed by a convergence analysis. This analysis provides an explanation of the sweet spot for micro-batch size. Finally, the paper demonstrates that micro-batch clipping is also effective in both vision and language tasks.
weaknesses: **Methodology**
- Section 4.3 is significantly underdeveloped and requires more extensive elaboration. Expanding on this section with a detailed explanation of the method would improve clarity and depth.
  - In line 279, the paper states, “The proof can be found in ??,” but the proof is missing, and there is no supplementary material to refer to.
  - The proofs for Lemma 4.3 and Lemma 4.4 are also missing. Furthermore, in line 304, "The proof is provided in Appendix ??." mentions that the proof is in the appendix, but no appendix is provided in the paper.
  -  I suggest that the authors include these proofs in an appendix or supplementary material and ensure that all references to proofs are properly linked.

-  The samples that slow down convergence, termed "draggers," seem to refer not to those with gradients perpendicular to the benign samples, but rather those with gradients in the opposite direction (i.e., cosine similarity around -1); in Figure 1, there appear to be cases where benign samples have negative cosine similarity. Are there no samples with reverse gradient directions?

- As noted in the paper's limitations, determining the exact micro-batch size $b$ that achieves the "sweet spot" is a challenging task. This poses a significant empirical limitation, and even within the paper, the optimal micro-batch size changes due to various factors:
  - The best micro-batch size varies by modality; for example, the authors use a size of 4 for the speech task, 32 for the vision task, and 16 for the language task.
  - Additionally, memory constraints may impede the selection of the optimal micro-batch size. For the vision task, a micro-batch size of 32 was used per core, but such size may be difficult to adjust due to memory limitations.
  - I suggest the authors discuss potential strategies or heuristics for selecting micro-batch sizes in practice, considering the observed 
variability and memory constraints.

- Along with batch size, one of the most critical hyperparameters in training neural networks is the learning rate. Prior research has explored the relationship between batch size and learning rate [1].

- Is micro-batch clipping independent of mini-batch size, as opposed to micro-batch size? Further investigation into this relationship could be beneficial.

- While this paper analyzes micro-batch clipping in the context of SGD, in practice, SGD is rarely used for training neural networks. It is important to determine whether a similar theoretical analysis holds for modern optimizers like Adam.

- In Section 5.1, to fine-tune the model, the LibriSpeech dataset and handcrafted canaries were synthetically used as benign and dragger samples, respectively. However, this setup seems artificial. In a more realistic scenario where only the LibriSpeech dataset is used, how would dragger samples be detected in practice?  Including practical strategies for detecting draggers would add relevance to the findings.

**Experiments**
- In Tables 2 and 3, experiments were conducted with only batch sizes of 1, 4, and 512, which is insufficient. The authors should have included batch sizes such as 1, 4, 16, 64, 256, and 1024 to provide a more comprehensive evaluation. If these additional results are not feasible, explaining the rationale behind the specific batch sizes chosen would help clarify this decision.

- The analysis related to $|| \mu || \le C || g||$ is completely missing in Section 5.2. I suggest the authors include this analysis in the section or provide an explanation if it was intentionally omitted. 

- How does micro-batch clipping perform when training from scratch instead of fine-tuning? This could offer valuable insights into its effectiveness in different training scenarios.

- When conducting the language task experiments, reporting validation performance is inappropriate. It would have been better to test on a separate dataset. Additionally, the paper states that validation set information was not used at all, so how was hyperparameter optimization carried out? Clarifying this would improve transparency.

- The experimental results shown in Table 5 reveal little to no improvement in performance. It would be helpful to discuss possible reasons for this outcome and consider further refinements or additional evaluation metrics.

**Typo**
- The term "nominator" in line 395 should be corrected to "numerator."

- The equation $ C = \sqrt{\frac{2(1-\epsilon)\eta - (1 - \epsilon^2) L \eta^2}{2 \epsilon^2 L \eta^2}}$ in line 219 should be updated to $ C = \sqrt{\frac{2(1-\epsilon)\eta - (1 - \epsilon)^2 L \eta^2}{2 \epsilon^2 L \eta^2}}. $

- Additionally, the expression $ \frac{2(1-\epsilon)\sigma^2}{LB} $ in line 232 should be updated to $ \frac{(1-\epsilon)\sigma^2}{2LB}. $

[1] Huo et al. "Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling." AAAI 2021.

Meta-review: This paper revisits micro-batch clipping, an optimization technique initially introduced for improving memory efficiency in differentially private stochastic gradient descent. The authors conceptualize micro-batch clipping as a form of data pruning, proposing that certain "dragger" gradients hinder model convergence. Through both convergence analysis and empirical evaluation, the paper demonstrates that micro-batch clipping can asymptotically accelerate convergence, although it introduces a constant bias term. This bias term helps explain the existence of an optimal micro-batch size. 

Additionally, the paper validates the effectiveness of micro-batch clipping beyond speech models, showing promising results on vision and language tasks. However, the authors identify limitations when training data comes from multiple distinct domains. The paper offers an innovative perspective by framing micro-batch clipping as a data pruning technique and introducing the concept of "dragger" gradients to explain the underlying mechanism. This approach provides valuable insights into why and when micro-batch clipping is effective.

However, Assumption 4.4, which posits that dragger gradients are orthogonal to the benign gradient subspace, may be a restrictive assumption that doesn't fully reflect practical scenarios. Furthermore, it appears that as the dragger gradient parameter decreases, approaching zero and suggesting a lower likelihood of draggers, the convergence rate bound in Theorem 4.2 actually increases. This result seems counterintuitive, raising questions about the assumptions and implications of the analysis.

Decision: Accept (Poster)