Zipformer: A faster and better encoder for automatic speech recognition
Authors: Zengwei Yao
Liyong Guo
Xiaoyu Yang
Wei Kang
Fangjun Kuang
Yifan Yang
Zengrui Jin
Long Lin
Daniel Povey
Venue: ICLR 2024 oral (2024)
Keywords: Zipformer
ScaledAdam
automatic speech recognition
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: The Conformer has become the most popular encoder model for automatic speech recognition (ASR).  It adds convolution modules to a transformer to learn both local and global dependencies. In this work we describe a faster, more memory-efficient, and better-performing transformer, called Zipformer.  Modeling changes include: 1) a U-Net-like encoder structure where middle stacks operate at lower frame rates; 2) reorganized block structure with more modules, within which we re-use attention weights for efficiency; 3) a modified form of LayerNorm called BiasNorm allows us to retain some length information; 4)  new activation functions SwooshR and SwooshL work better than Swish.  We also propose a new optimizer, called ScaledAdam, which scales the update by each tensor's current scale to keep the relative change about the same, and also explictly learns the parameter scale. It achieves faster converge and better performance than Adam. Extensive experiments on LibriSpeech, Aishell-1, and WenetSpeech datasets demonstrate the effectiveness of our proposed Zipformer over other state-of-the-art ASR models. Our code is publicly available at https://github.com/k2-fsa/icefall.

Published as a conference paper at ICLR 2024
ZIPFORMER: A FASTER AND BETTER ENCODER FOR AUTOMATIC SPEECH RECOGNITION
Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang, Zengrui Jin, Long Lin, Daniel Povey Xiaomi Corp., Beijing, China dpoveyêx aomi com
ABSTRACT
The Conformer has become the most popular encoder model for automatic speech recognition (ASR). It adds convolution modules to a Transformer to learn both local and global dependencies. this work we describe a faster, more memory- efficient, and better-performing Transformer, called Zipformer. Modeling changes include: 1) a U-Net-like encoder structure where middle stacks operate at lower frame rates; 2) reorganized block structure with more modules, within which we re-use attention weights for efficiency; 3) a modified form of LayerNorm called BiasNorm allows to retain some length information; 4) new activation functions SwooshR and Swooshl. work better than Swish. We also propose new optimizer, called Adam, which scales the update by each tensor's current scale to keep the relative change about the same, and also explictly learns the parameter scale. It achieves faster convergence and better performance than Adam. Extensive ex- periments on LibriSpeech, Aishell-1, and WenetSpeech datasets demonstrate the effectiveness of our proposed Zipformer over other state-of offte--o--hhhrrrr ASR models. Our code is publicly available at https://ithub..om/// faaaiccflll.
INTRODUCTION
End-to-end models have achieved remarkable success in automatic speech recognition (ASR). An effective encoder architecture that performs temporal modeling on the speech sequence plays vital role in end-to-end ASR models. most prominent example Conformer (Gulati al.) 2020). which combines the advantages of the convolutional neural network (CNN) models (Zhang et al.
2017 Li et al. 2019 Kriman et al. 2020) and Transformer models (Dong et al. 2018 Karita et al, 2019 Zhang 2020b). By integrating CNN into Transformer (Vaswani et al. 2017). Conformer
is able to extract both local and global dependencies on speech sequences, and achieves ofate-of-the- art performance in ASR.
In this work, we propose a faster, more memory-efficient, and better better-perfonee performing Transformer as ASR encoder, called Zipformer. First, unlike Conformer that operates the sequence constant frame rate, Zipformer adopts a U-Net-like (Ronneberger et al.) (015)) structure, which consists of multiple stacks downsamping the sequence various lower frame rates. Second, we re-design the block structure, which is equipped with more modules like two Conformer blocks, and reuses the attention weights for efficiency. We propose BiasNorm as simpler replacement of LayerNorm, which allows for retaining length information in normalization. We also replace Swish with our new activation functions SwooshR and SwooshL achieve better results. In addition, we devise version of Adam, called ScaledAdam, which scales the update by the cur- rent parameter scale and also explicitly learns the parameter scale. Compared to Adam, ScaledAdam enables faster convergence and better performance.
srrameter-saale-ivarii
Extensive experiments are conducted on LibriSpeech, Aishell-1, and WenetSpeech datasets, and results demonstrate the effectiveness of the proposed modeling and optimization--relteed innovations. Zipformer achieves state-of of-te-----e-r results on all three datasets. is worth mentioning that Zipformer the first model ever achieve results comparable those reported in the Conformer paper the LibriSpeech dataset (these results have proved difficult for others to reproduce). In terms of efficiency, Zipformer converges faster during training and speeds up the inference by more than

---

Published as a conference paper at ICLR 2024
50% compared to previous studies while requiring less GPU memory. We perform detailed ablation studies to investigate the contribution of individual components.
2 RELATED WORK
Model architecture. Deep convolution architectures have been applied to end-to-end ASR Zhang et al. 2017: Li et al. 2019). Follow-up works explore improvements by using depthwise separable convolutions (Howard et al. 2017) for efficiency (Kriman et al. (020), and incorporating squeeze- and-excitation module (Hu et al. 2018 to capture longer context (Han et al. 2020). Inspired by
the success of Transformer (Vaswani et al. 2017) in natural language processing (NLP) field, some works adapt Transformer to speech applications (Dong et al. 2018 Karita et al. 2019 Zhang et al. 2020b, Wang al. 2020 Zhang al. 2020a). Compared to CNN, the remarkable benefit of Transformer is that it can learn global dependencies based on self-attention, which is essential for speech processing task. By integrating convolution into Transformer, Conformer (Gulati al. 2020) gains powerful capability of modeling both local and global contextual information, and outperforms all previous ASR models.
Recent works explore architecture changes on Conformer to further reduce the computational cost and improve the recognition performance. Squeezeformer (Kim et al.] [022] adopts a temporal U-Net structure in which the middle modules operate at half frame rates, and also redesigns the block structure to make it similar to the standard Transformer block (Vaswani et al., (2017). Branch- former (Peng al., 2022) incorporates parallel branches to model various ranged context, in which one branch captures the local context with convolutional gating multi-layer perceptron (MLP), while the other branch learns long-range dependencies with self-attention. E- E-Branchformer (Kim al.,
2023) further improves Branchformer by enhancing the branch merging mechanism by convolution- based module.
Zipformer shares similar ideas about temporal downsampling as the previous work Squeezeformer. However, compared to the fixed downsampling ratio in Squeezeformer, Zipformer operates differ- ent downsampling ratios at different encoder stacks and uses much more aggressive downsampling ratios in the middle encoder stacks. In addition to the modeling differences, our work also focuses on optimization- related changes including new optimizer ScaledAdam, which are shown to improve convergence in the experiments.
End-to-end framework. Connectionist temporal classification (CTC) (rrvvss et al., 2006) is one of the earliest frameworks for end-to-end ASR, but its performance limited by the frame inddpee- dent assumption. To this end, hybrid architecture that integrates attention- based encoder-deocder (AED) (Chan et al., 2015) in CTC (Watanabe et al. 2017) (CTC/AED) is proposed to improve the performance. Neural transducer (Graves 2012), commonly known as RNN-T, addresses the frame independence assumption using a label decoder and a joint network and becomes a popular framework due to its superior performance. Recently, various approaches such pruning (Kuang et al. 2022, Wang et al., 2023 Mahadeokar et al., [202]) or batch-splitting (Kuchaiev et al. 2019) proposed to accelerate the training speed and reduce memory usage of neural transducers.
3 METHOD
3.1
DOWNSAMPLER EED ENCODER STRUCTURE
Figure presents the overall architecture of the proposed Zipformer model. Different from Con- former (Gulati al. 2020) that processes the sequence at fixed frame rate of 25Hz, Zipformer uses a U-Net-like structure learning temporal representation at different resolutions in a more effi- cient way. Specifically, given the acoustic features with frame rate of 100Hz, the convolution-based module called Cony- Embee first reduces the length by a factor of 2, resulting in a 50Hz embedding sequence. The obtained sequence is then fed into cascaded stacks to learn temporal representa- tion at frame rates of 50Hz, 25Hz, 12.5Hz, 6.25Hz, 12.5Hz, and 25Hz, respectively. Except for the first stack, the other stacks all adopt the downsampled structures, processing the sequence at lower frame rates. The frame rate between stacks is consistently 50Hz. Different stacks have different embedding dimensions, and the middle stacks have larger dimensions. The output of each stack is truncated or padded with zeros to match the dimension of the next stack. The final encoder output dimension is set to the maximum of all stacks' dimensions. Specifically, if the last stack output has the largest dimension, it is taken the encoder output; otherwise, it is concatenated from differ-
2

---

Published as a conference paper at ICLR 2024
50Hz
50Hr
3X
2x
2x
Cosy Imbed
Ziploras
bibeds
Daumamppe
Zipbomer
bibocks
Doumanppe
Zquhamer
Hooes
Upan pla
Bana
Zipformr
blookk
Upanael
Bypass
Donnanple
Zipfomer
Uppompp
+
Bypas
Doveammpk
Zipfomner
Blocks
Uppnnple
Paynss
Doumeumpl
2512
2581x
12,58z
6.25 Hi
12,5812
251
Figure 1: Overall architecture of Zipformer.
Zipformer Block
Attration
Mult-holl Attentin
Altestion weights
Lineur
Single-head Attemnien
Nar-iiaa Atterion
Salt- Attenti
Consolaton
Fool-frea Feul-foraard
Food Sel--rrrrrdd
weights
Da
tant
Food
Figure 2: (Left): Zipformer block structure. (Right): Non-Linear Attention module structure.
ent pieces of stack outputs, taking each dimension from the most recent output that has present. Finally, Downsample module converts the sequence 25Hz, resulting in the encoder output.
Conv-Embed. In Cony-Embed we use three 2-D convolutional layers with time frequency strides of 1 x 2, 2 x 2, and 2, and output channels of 8, 32, and 128, respectively. Subsequently, we utilize one ConvNeXt layer (Liu al.) 2022) similar to Nextformer (Jiang et al., (2022). which composed of depth-wiie wise convolution with kernel size of 7 x 7, point wise convolution with 384 output channels, a SwooshL activation function (described in Section (.4), and a point-wise convolution with 128 output channels. Residual connection is applied the ConvNeXt module.
Finally, a linear layer followed by a BiasNorm (described Section 3.3 is used to adjust the feature dimension to match the first stack.
Downsampled stacks. In the downsampled stacks, the pairwise Downsample and Upsample mod- ules perform symmetric scaling down and scaling up in sequence length, respectively, using almost the simplest methods. For example, with a factor of 2, the Downsample module averages every frames with 2 learnable scalar weights (after softmax normalization), and the Upsample module just repeats each frame twice. After downsampling, it employs the stacking Zipformer blocks (described in Section 3.2 for temporal modeling at lower frame rates. Finally, utilizes the Bypass module (described in Section 3.2) combine the stack input and stack output in a learnable way.
3.2 ZIPFORMER BLOCK
Conformer block consists four modules: feed-forward, Multi-Head Self-Attention (MHSA), con- volution, and feed-forward. MHSA learns global context by two steps: computing attention weights using the dot-product operation and aggregating different frames with these attention weights. How- ever, MHSA typically accounts for large computational cost, since above two steps both require quadratic complexity with respect to the sequence length. Hence, we decompose MHSA into two individual modules according to above two steps: Multi-Head Attention Weight (MHAW) and Self- Attention (SA). This change allows to perform the attention computation twice more efficiently in each block by using one MHAW module and two SA modules. addition, we propose new module called Non-Linear Attention (NLA) to make full of the computed attention weights to capture the global information.
As illustrated in Figure (Left), Zipformer block is equipped with about twice the depth of the Conformer block (Gulati et al. 2020). The main motivation is to allow the re-use of the attention weights time and memory. Specifically, the block input is first fed into an MHAW module, which calculates the attention weights and shares them with an NLA module and two SA modules. Meanwhile, the block input is also fed into feed-forward module followed by the NLA module.
3

---

Published as a conference paper at ICLR 2024
Then it applies two module groups, each consisting of SA, convolution, and feed-forward. Finally, BiasNorm (described in Section 3.3) is used to normalize the block output. In addition to the regular residual connections using adding operation, each block utilizes two Bypass modules to combine the block input and the module outputs, placed in the middle and end of the block. Note that different from regular Transformer models (Vaswani et al. (017). we don't as LayerNorm (Ba et al. 2016) for each module to periodically prevent activations from becoming either too large or too small, since our proposed ScaledAdam optimizer is able to learn the parameter scales (described in Section 3.5
normalization layer such
Non-Linear Attention. Figure 2 (Right) presents the NLA structure. It also leverages the pre- computed attention weights from MHAW to aggregate the embedding vectors over the time axis, which similar to SA. Specifically, it first projects the input with 3 linear layers to A, B, and C, each being of 3/4 input dimension. The module output is linear( A à attention(tanly where @ denotes the element-wiss multiplication, attention represents matrix- multipiil on the time axis by single head of previously computed attention weights, and the linear layer recovers the dimension to the same as the input.
@ C)),
Bypass. The Bypass module learns channel-wise scalar weights с to combine the module input and module output y: (1 -1-ca c) 3 x + c 3 y. In training, we initially limit the values of c in range of [0.9, 1.0] and then change the minimum 0.2 after 20000 steps. We found that making modules sarraightthhoouge at the beginning (i.e. allowing very little bypass) helps model convergence..
3.3 BIASNORM
Conformer (ulati et al. 2020) utilizes LayerNorm (Ba et al. (2016) to normalize the module acti- vations. Given x with 1) channels, LayerNorm formulated as:
LayerNorm(x) x Exx ①
B.+
(1)
Var[x] +
Specifically. it first computes the mean E[x] and the scaling the vector length to VD. Then it uses the learnable channel-wise scale y and bias B for transformation, which helps to adjust the size of activations and balance the relative contributions of specific modules. However, we observe that the trained Conformer using LayerNorm suffers from two problems: 1) It sometimes sets one channel aims to "defeat" the LayerNorm which fully removes the vector length, functioning as a very large value so that length information could be retained after normalization. 2) Some modules (typically feed-forward or convolution) are "dead" as they have extremely small output values, c.g., 10 We argue that early in training, the un-trained modules are not useful so they are "turned off" by the LayerNorm scale y approaching zero. If the scale y oscillates around zero, the inconsistent sign constantly reverses the gradient directions back propaaanggg to the modules. Because of the inconsistent gradient sign, the modules never learn anything useful, since this is a bad local optimum which is hard to escape because of the dynamics of stochastic gradient descent-like updates.
v Var [X] for normalizing,
large constant value, e.g. 50. We argue that it
To address above problems, propose the BiasNorm which is intended to be simpler replacement of LayerNorm. Specifically, BiasNorm is formulated as:
BiasNorm(x)
RMS b] -ex((7),
X
where b is the learnable channel-wise bias, RMS(x b] is the root-mean-squee value taken over channels, and 7 is sealar. We first remove the operation of mean subtraction since is a waste of time unless it follows a non-linearity. The bias b serves as the large constant value which allows to retain the vector length information after normalization. Since the scale exp(7) is always positive, it avoids the gradient oscillation problem.
3.4 SWOOSHR AND SwooSHL ACTIVATION FUNCTIONS
Conformer (Gulati al. 2020 adopts Swish RRamachandran al. 2017 activation function with the following formula:
Swish(x) x. exp(-x)))-1
(3)
4

---

Published as a conference paper at ICLR 2024
In this work, propose two new activation functions respectively called SwooshR and SwooshL as replacements of Swish:
SwooshR(<) (a: log(1 exp(a: 1)) 0.08& 0.313261687,
(4)
SwooshL.(r) = log(1 + exp(x 4)) - 0.08x - 0.035.
In SwooshR, the offset 0.313261687 is to make it pass through the origin; in SwooshL, the offset 0.035 was tuned, which slightly outperformed the value exactly making the curve pass through the origin. We present the curves of Swish, SwooshR, and Swooshl. in Appendix Section A.2 Swooshl. is roughly right shifted version of SwooshR. Note that the suffix "L"" or "R" represents whether the left or right zero-crossing at or around x = O. Similar to Swish, SwooshR and SwooshL. have lower bounds and are non-mono..iic Compared to Swish, the most striking difference is that SwooshR and SwooshL have non-vanishing slopes for negative inputs, which helps to escape from situations where the input is always negative and prevents the denominator term Adam-type updates from getting dangerously small. When replacing Swish with SwooshR, we observe that the modules with bypass connections, such as feed-forward and ConvNeXt, tend to learn a large negative bias in the preceding linear layer to learn "normally-off" behavior. Therefore, we use Swooshl. for these "normally-off" modules and use SwooshR for convolution modules and the rest Conv-Embedt
3.5
SCALEDĀDAM OPTIMIZER
We propose a ceanenter-scal-invaaiie version of Adam (Kingma & Ba 2014) called ScaledAdam, which enables faster convergence and better performance. ScaledAdam scales each parameter's update proportional to the scale of that parameter, and also explicitly learns the parameter scale. Algorithm in Appendix Section A.1.I presents the pseudo-code of the ScaledAdam.
Let f() be the loss function that aim to minimize, which differentiable w.r.. the learnable parameter 0. At each step 1, Adam computes the parameter gradient g: (o.())))), and updates the first moment m 31 m (1-31) -1)... and the second moment v; f2 Vi- +1 f2) g?
of gradients, where 31, 32 E 1) are coefficients used to compute the moving averages. The parameter update A, step is formulated as:
8 B
m
A = a0
(5)
V +
where or is the learning rate typically specified by an external schedule, V1-
1-8f is the bias-correction term, and 10-8 Whilst Adam is invariant to gradient scale of each parameter, we argue that it still suffers from two limitations: 1) The update A, in Equation 5 does not take into account the parameter scale (denoted as rı-1). Considering the relative parameter change A Adam might cause learning in relative terms slowly for parameters with large scales, too fast for parameters with small scales. 2) It difficult to learn the parameter scale directly, as the direction of growing shrinking the parameter tensor is very specific direction in large-dimensional space. It's particularly difficult to shrink parameter, since each gradient step g: adds noise which tends to grow the parameter norm.
Scaling update. To keep the relative change A A//r-1 over parameters of varying scales about the same, we scale the update A, in Equation 5 by the parameter scale r=-1:
A = -
(6)
3
Vi
We compute the parameter scale r;-1 as the root-mean-square value RMS[0,-1]- Because the ScaledAdam update is less prone to divergence than Adam, Eden that does not have a long warm-up period; we also use absolutely larger learning rate values because the parameter RMS value is normally much less than one.
learning rate schedule called
Learning parameter scale. To explicitly learn the parameter scale, we treat it as regular parameter to be learned, as if we have factored each parameter as = r- and we are doing gradient descent on the parameter scale and the underlying parameter 0'. Let / be the gradient of the parameter scale r, at step we get h4 = ( = gi 6/-1- Since Adam is nearly invariant to changes in
the gradient scale, for simplicity we replace this with h, g: (r 0
ge 0 Following
the Adam algorithm, we maintain the first moment n, = 31 n4-1 + (1 1--3 B1) h, and the second
moment wt B2 W + (1 B2) h2 of the scale gradients h4. The parameter change on 0 caused
5

---

Published as a conference paper at ICLR 2024
by updating parameter scale from rt-1 to ri is A/,- (rr r1-1) also integrate the parameter scale Tt-1 into the update ب'م;
@ 0/- Similar to Equation 6
31
A/r, -17- a2 Tt-1
74
0
1
v Wi
+
=- / 31
32
Bt
① 01-1-
W1
where 77 is scaling factor on learning rate ae, and we found that setting 77 0.1 helps stabilize the training. Now the update A; is replaced with A + A which amounts to adding an extra gradient term in the direction of growing or shrinking each parameter. This also allows simplify the network structure by removing most of normalization layers in our Zipformer Block (described in Section (..2), since the modules now can easily learn to scale the activations in a suitable range. One similar method called weight normalization (Salimans & Kingma 2016) decouples the parameter norm from its direction to speed up the convergence. It replaces each parameter with two parameters, respectively specifying the direction and the magnitude. However, ScaledAdam learns the parameter scales by adding an extra update term A which makes writing the modeling code simpler.
Eden schedule. The proposed Eden learning rate schedule is formulated as:
12 Ggtep (=0
0.25
ee2 + a2
0.25
02
Othase
Q2 epoch
epoch
llinear ( t).
Herein, t is the step index, e is the epoch index, Ostep and Cepoch respectively control the num- ber of steps and number of epochs after which we start significantly decreasing the learning rate, Ainar((start:: twarmup, t) warmup scale increasing linearly from O'start over warrmu steps and then staying constant at 1, Obbss is the maximum value when setting Osstart = 1, O'warmup 0. The reason for making Eden dependent on both the step index / and the epoch index e to keep the amount of parameter change after certain amount of training data (e.g., one hour) approximately constant when we change the batch size, so the schedule parameters should not have to be re-tuned if we change the batch size. Other versions of Eden replace the "epoch" parts of the formula with some suitable measure of the amount of data seen. In this work, we use Obase = 0.045, Ostart = 0.5, and wwarmup 500.
Efficient implementation. To speedup the optimization in ScaledAdam, we group the parameters into batches according to their shape and perform the computation batch by batch. Note that this doesn't affect the outcome. ScaledAdam just requires a little more memory than Adam to cache the gradient moments n4 and we (in Equation 7) for the parameter scales.
4 EXPERIMENTS
4.0.1 EXPERIMENTAL SETUP
Architecture variants. We build our Zipformer variants with three model scales: small (Zipformer- S), medium (Zipformer-M), and large (Zipformer-L). For the 6 encoder stacks, the numbers of at- tention heads are set (4,4.4.8.4.4), the convolution kernel sizes are set (31.31.15.15155.31, In each attention head, the query dimension and value dimension are set to 32 and 12, respectively. For the three seed-forward modules in each Zipformer block, the hidden dimensions in the first one and the last one are 3/4 and 5/4 of that in the middle one. We adjust the layers numbers, the embedding dimensions, and the hidden dimensions the middle feew-forward in each stack obtain different model scales:
Table Configuration of Zipformer at three different scales.
Scale
layer-numbers
mubedding--imennse
feed-forward-
s
(2.2.2.2.222)
(192.256.26.256.2666
{512.768, 766.768.768,66.. 7683366687777786688 768) (512,768, 1536, 1024,768
M
[2.2.3.4332]
L
[2.2.4.5.422]
(192.556.112.78851222255
Datasets. We perform experiments to compare our Zipformer with -f-te---f-thh-ther models on three open-source datasets: 1) LibriSpeech (Panayotov al. 2013) which consists of about 1000

---

Published as a conference paper at ICLR 2024
hours of English audiobook reading, 2) Aishell-1 (Bu al. 2017) which contains 170 hours of Mandarin speech; 3) WenetSpeech (Zhang et al. 2022a) which consists of 10000+ hours of multi- domain Mandarin speech.
Implementation details. We use Lhotse Zelasko et al. 2021) toolkit for speech data preparation. The model inputs 80-dimension Mel filter-bank features extracted on 25ms frames with frame shift of 10ms. Speed perturbation (Ko et al. | (0155 with factors of 0.9, 1.0, and 1.1 is used to augment the training data. Sugmentment (Park al. 2019) is also applied during training. We use mixed precision training for our Zipformer models. We also employ the actívation constraints including Balancer and Whitener ensure training consistency and stability. The details of Balancer and Whitener are presented in Appendix Section[A.3] Pruned transducer (Kuang et al. 2022), memory- efficient version of transducer loss that prunes path with minor posterior is used as the training objective. During decoding. beam search of size 4 with the constraint of emitting at most one symbol per frame is employed (Kang et al.) (023). We don't use external language models for rescoring, since in this work we focus on improving the encoder model. We employ word-error-rate (WER) and character error rate (CER) as evaluation metric for English and Mandarin datasets, respectively. By default, all of models are trained on 32GB NVIDIA Tesla V100 GPUs. For Librispeech dataset, Zipformer-M and Zipformer-L are trained for 50 epochs on 4 GPUs, and Zipformer-S is trained for 50 epochs 2 GPUs For A dataset, our models trained for 56 epochs on GPUs. For WenetSpeech dataset, our models are trained for 14 epochs on 4 GPUs.
4.0.2 COMPARISON WITH STATE-OF-TIIE-ART MODELS
In this section, we compare the proposed Zipformer with other state-of-the-ae models.
LibriSpeech dataset. Table 2 shows the results on LibriSpeech test datasets for Zipformer and other state-of-the-arr models. For Conformer, we also list the WERs reproduced by us and other open-source frameworks. Note that there is performance gap between the open-source reproduced Conformer and the original Conformer. Our Zipformer-S model achieves lower WERs than all vari- ants of Squeezeformer while having much fewer parameters and floating point operations (FLOPs). Our Zipformer-Li outperforms Squeezeformer- Branchformer and our reproduced Conformer- L by large margin while saving over 50% FLOPs. Noticeably, when trained on 8 80G NVIDIA Tesla A100 GPUs for 170 epochs, Zipformer-L achieves WERs of 2.00%/4.38%% with sufficient computing resources (last row), which is the first model to approach Conformer-L the best of our knowledge.
We also compare the speed and memory usage between the proposed Zipfomer and other state-of- the-art models. Figure [ presents the comparison results in terms of averaged inference time and peak memory usage in inference mode for batches of 30-second audios on an NVIDIA Tesla V100 GPU. The batch size set to 30 to ensure all models do not have of memory problems during in- ference. In overall, Zipformer models achieves better trade-off between performance and efficiency than other models. Especially for the large scale, Zipformer-L requires much less computation time and memory than other counterparts.
6.5
Zipformer, pruned transducer
6.5
Zipformer, pruned transducer
Averaged WER (%)
6.0
Branchforrmrr CTC/AED
Avraged WER (%)
6.0
E- Branchhfommer CTC/AED
5.5
Squeezeloemer,
5.5
fqumezeformer, former, CTC Conformer, praned transducer
Conformer, pruned transducer
5.0-
SM
5.0
4.5
4.5
ML
4.0
4.0-
3.5
3.5
200
400
600
800
1000
1200
2
12
16
Avergged inference time (ms)
Peak memory usage (GB)
Figure 3: (Left) Averaged inference time and (Right) peak memory usage vs. WER comparison for different models. The WER is averaged on LibriSpeech test-clean and test-other. Averaged inference time and peak memory usage are reported for the encoders in inference mode for batches of 30-second audios with batch size of 30 on a single NVIDIA Tesla V100 GPU.
Aishell-1 dataset. Table 3 shows the CERs on Aishell-1 1 dataset. Compared to the Conformer model implemented in ESPnet toolkit, our Zipformer-S achieves better performance with fewer parameters. Scaling up the model leads to lower WERs, and Zipformer-M/I. outperform all other models.
7

---

Published as a conference paper at ICLR 2024
Table 2: (ER(%) comparison between different models on LibriSpeech dataset. We also in- clude the number of parameters and FLOPs of encoder for a 30s input audio measured with Deep- Speed (Rasley et
2020
"Trained with 80G NVIDIA Tesla A100 GPUs for 170 epochs.
Model
Type
Purams (M)
GFLOPs
test-clean (%)
test-other (5)
Squeezeformer-X Kim et
2022
CTC
9.0
18.2
3.74
Squeezeformer-S Kimm 2022
CTC
18.6
33.7
3.08
7.47
Squeezeformer SM
(Km cl
4022
CTC
28.2
47.6
2.79
6,89
Squeezeformer-M
ef 2022
CTC
88.4
2.56
Squeezeformer-MM]
CTC
125.1
183.3
2.61
6.05
Squeezeformer-1. Kim al
2722
CTC
236.3
333.7
2.47
5.97
E: BrBanchformer-B 8
Kim al.
2023
CTC/AED
41.1
78.1
2.49
5.61
Branchformer Peng al 2022
CTC/AED
116.2
238.3
2.4
E- E-Branchfiorme--LL KKim
2023
CTC/АED
148.9
284.4
2.14
4.55
Conforme-S Gulati al.
2020
transducer
10.3
2.7
6.3
Confommer-M M Lubaai et
2020
transducer
30.7
2.3
5.0
Conformer-I Gilaat et
2020
transducer
Conformer WeNet (Zhang 2022b
CТCAAED
121.3
2.66
Conformer in ESPact [Myazaka at. 2023
CTC/AED
113.2
2.29
5.13
Conformer-S S
pruned transducer
9.8
29.1
3.75
9.24
Conformer-M
pruned tansducer
77.0
2.96
Conformer-L
pruned transducer
122.5
294.2
2.46
5.55
S
pruned transducer
23.3
40.8
2.42
5.73
Zipfonner-M
pruned transducer
65.6
62.9
2.21
4.79
Zipformer-L Zipformer-L."
pruned transducer
148.4
107.7
2.06
4.63
pruned transducer
148.4
107.7
2.00
4.38
WenetSpeech. Table presents the experimental results WenetSpeech dataset. Again, our Zip- former-M and Zipformer-L Lotteeerrrooooprrrooo all other models on Test_Net and Test_ Meeting test sets. With only one third of the parameters, Zipformer-S yields lower WERs than Conformer models.
Table 3: CER(%) comparison between different models on Aishell-1 dataset.
Model
Params (M)
Type
Dev
Test
Conformer ESPnet (Watanabe al 2018
46.2
CTC/AED
4.5
4.9
Conformer WeNer Yao
46.3
CTC/ADDD
4.61
E- Brancchformer in ESPnet Waranabe et al. 2018
37.9
CTC/AED
4.2
4.5
Branchformer (Peng et al. 2022
45.4
CTC/AED
4.19
4.43
S Zipformer-M Zipformer-L.
pruned transducer
4.67
73.4
pruned transducer
4.13
4.4
157.3
pruned transducer
4.03
4.28
Table 4: CER(%) comparison between different models on WenetSpeech dataset.
Model
Params (M)
Type
Dev
Test Net
Test Meeting
Conformer
2018
CTC/AED CTC/AED
15.90
Conformer WeNet
Yao al.
116.9
8.88
9.70
15.59
Conformer
Monformer-MnF(16e MoE(1
You et al.
2022
425
CTC/AED, MoFi
7.67
8.28
13.96
Conformer Conformer-MoE322e
MoE(32e)
You ef
2022
CTC/AED. McE
7.49
7.99
13.69
Conformer MoF(64e)
You
CTC/AED, MoF
7.19
8.36
13.72
Zipformer S
32.3
pruned transducer
7,96
8.6
13.97
7ipformer-M Zipformer-L
75.9
pruned transducer
7.32
7.61
12.35
160.9
pruned transducer
7.29
7.24
12.06
4.0.3 ABLATION STUDIES
We perform ablation experiments on LibriSpeech dataset investigate the effect of each proposed functional technique. With Zipformer-M as the base model, we make one change each time while keeping the others untouched. T the experimental results.
Encoder structure. We remove the temporal downsampling structure from Zipformer and use Cony-Ember with downsampling rate of 4 like Conformer. The resulting model has 12 Zipformer blocks with constant embedding dimension of 512 and has more parameters than the base model.
8

---

Published as a conference paper at ICLR 2024
Table Ablation studies for Zipformer-M, including encoder structure, block structure, normaliza- tion layer, activation function and optimizer.
Ablation
Params (M)
test-clcan (%)
test-other (5r
Zipformer-M
65.6
2.21
4.79
Encoder structure
No temporal downsampling
94.2
2.23
5.09
Block structure
Double Conformer-style blocks
73.9
2.18
4.95
No NLA
58.7
2.16
4.97
No NI.A, no attention weights sharing Bypass
6009
2,20
5.10
65.5
2.25
4.86
Normalization layer
LayerNorm
65.6
2.29
4.97
Activation function
Only SwooshR
65.5
2.32
5.21
Swish
65.5
2.27
5.37
Optimizer
Adam
65.6
2.38
5.51
Experimental results in Table show that the resulting model without the downsampled structure yields higher WERs on both test set. It indicates that the temporal downsampling structure for efficiency does not cause information loss, but facilitates the modeling capacity with less parameters.
Block structure. As each Zipformer block has roughly twice modules as Conformer block, we replace each Zipformer block in the base model with two Conformer blocks stacked together. This leads 0.16% absolute WER reduction on test-other even with larger model size, suggesting the benefits of Zipformer block structure. Removing either NLA or Bypass leads to performance degradation. we further remove attention weights sharing mechanism after removing NLA, model has slightly more parameters and slower inference speed, but the WERs are not improved. We hypothesize that the two attention weights inside one Zipformer block are quite consistent and sharing them does not harm the model.
Normalization layer. Replacing BiasNorm with LayerNorm in Zipformer leads to WER drops of 0.08% and 081% test-clean and test-other, respectively. It indicates the advantage of the proposed BiasNorm which allows to retain some length information in normalization.
Activation function. When using only SwooshR for all modules in Zipformer, the WER drops by 0.11% and 0.42% on test-clean and test-other, respectively, which validates the effectiveness of particularly using Swooshl. for the "normally-off' modules. Employing Swish leads more performance degradation, which indicates the advantage of SwooshR over Swish.
Optimizer. When using Adam to train Zipformer, have apply BiasNorm for each module in Zipformer block to avoid model divergence, since Adam cannot leam the scale of each parameter adjust the module activations like ScaledAdam, We try different learning rate factors (denoted as Obase) for ScaledAdam (0.025, 0.035, 0.045, 0.055) and Adam (2.5, 5.0, 7.5, 10.0) separately. Following (Gulati al. 2020 the learning rate schedule for Adam a O'base 512 0.5
min(t 0.5 .t 10000 test-clean and test-other at different epochs as well as the learning rates at different steps. We show the best results of ScaledAdam with Obasse 0.045 and Adam with Obase 7.5 in Table ScaledAdam outperforms Adam by 0. 0.7 and 0.72% on test-clean and test-other, respectively. The results indicate that ScaledAdam enables faster convergence and better performance than Adam.
Figure A.2 in Appendix Section AA.12 presents the averaged WERs on
5 CONCLUSION
In this work, present the Zipformer, which serves efficient ASR encoder. It has U- Net-like encoder structure, which downsamples the sequence to various lower frame rates. The re-designed block structure equipped with more modules reuses the computed attention weights for efficiency. It also employs the new normalization method BiasNorm, as well as the new activation functions SwooshR and SwooshL. Mcanwhile, the proposed optimizer ScaledAdam enables faster convergence and better performance. Extensive experiments on LibriSpeech, Aishell-1 and Wenet- Speech datasets have demonstrated the effectiveness of the proposed Zipformer.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.00 2016.
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 20th conference of the oriental chapter of the international coordinating committee speech databases and speech I/O systems and assessment (O-COCOSDA), pp. 1-5, 2017.
William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint ar ArXiv: 0/ 2015.
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer: no-recurrence sequence-to-sen -o-seueece model for speech recognition. In IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5884-5888, 2018.
Alex Graves. Sequence transduction with recurrent neural networks.
arXiv preprint
arXiv:/21.37711 2012.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Anmol Gulati, James Qin, Shung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution Convolution-augmented Transformer for Speech Recognition. In Proc. Interspeech 2020, pp. 5036-5040, 2020.
Wei Han, Zhengdong Zhang. Yu Zhang, Jiahui Yu, Chung- Chiu, James Qin, Anmol Gulati, Ruoming Pang, and Yonghui Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. arXiv preprint arXiv:2005.0319/ 2020.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze- Squeezo-and-cxcitattoo networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Yongjun Jiang, Jian Yu, Wenwen Yang. Bihong Zhang, and Yanfeng Wang. Nextformer: A convnext augmented conformer for end-to-end speech recognition. arXiv preprint arXiv:2206. 14747, 2022.
Wei Kang. Liyong Guo, Fangjun Kuang, Long Lin, Mingshuang Luo, Zengwei Yao, Xiaoyu Yang. Piotr Zelasko, and Daniel Povey. Fast and parallel decoding for transducer. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al. A compar- ative study on transformer vs rnn speech applications. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 449-456, 2019.
Kwangyoun Kim, Felix Wu, Yifan Peng, Jing Pan, Prashant Sridhar, Kyu Han, and Shinji Watan- abe. E-branchformer: Branchformer with enhanced merging for speech recognition. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 84-91 JEEE, 2023.
Schoon Kim. Amir Gholami, Albert Shaw, Nicholas Lee, Karttikeya Mangalam, Jitendra Malik, Michael W Mahoney, and Kurt Keutzer. Squeezeformer: An efficient transformer for automatic speech recognition. Advances in Neural Information Processing Systems, 35:9361-9373, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:141 1e.. 6980, 2014.
10

---

Published as a conference paper at ICLR 2024
Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. Audio augmentation for speech recognition. In Sixteenth annual conference of the international speech communication association, 2015.
Samuel Kriman, Stanislav Beliaev. Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, and Yang Zhang. Quariznet: Deep automatic speech recog- nition with Id time-channel separable convolutions. IEEE International Conference on Acous- lics, Speech and Signal Processing (ICASSP), pp. 6124-6128, 2020.
Fangjun Kuang, Liyong Guo, Wei Kang, Long Lin, Mingshuang Luo, Zengwei Yao, and Daniel Povey. Pruned mn-t for fast, memory-efficient asr training. arXiv preprint arXiv:2206. Nadv:2206.13336. 13236, 2022.
Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, et al. Nemo: toolkit for building ai applications using neural modules. arXiv preprint (eXiv:1909.0900 09577997999999...90 2019.
Jason Li, Vitaly Lavrukhin, Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M Cohen, Huyen Nguyen, and Ravi Teja Gadde. Jasper: An end-to-end convolutional neural acoustic model. arXiv preprint arXiv: 19XXvv:904..33 033288, 2019.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVE conference on computer vision and pultern recognition, pp. 11976-1 1986, 2022.
Jay Mahadeokar, Yuan Shangguan, Due Le, Gil Keren, Hang Su, Thong Le, Ching-Feng Yeh, Chris- tian Fuegen, and Michael L Seltzer. Alignment restricted streaming recurrent neural network transducer. In Proc. SLT. IEEE, 2021.
Koichi Miyazaki, Masato Murata, and Tomoki Koriyama. Structured state space decoder for speech recognition and synthesis. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2210, 2015.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoe Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: 1904.0877, 2019.
Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe. Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding. In International Conference on Machine Learning, pp. 17627-17643. PMLR, 2022.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for activation functions. arXiv preprint 1710.05711.59944 2017.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. -5066, 2020.
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAI, pp. 234-2411 2015.
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterizatton accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
11

---

Published as a conference paper at ICLR 2024
Yongqiang Wang, Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al. Transformer based acous- tic modeling for hybrid speech recognition. In IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 6874-6878, 2020.
Yongqiang Wang, Zhehuai Chen, Chengjian Zheng, Yu Zhang, Wei Han, and Parisa Haghani. Ac- celerating mn-t training and inference using ctc guidance. Proc. ICASSP. IEEE, 2023.
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 1 118)):2200-253,, 2017.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Proceedings of Inter- speech, pp. 2207-2211, 2018.
Zhuoyuan Yao, Di Wu, Xiong Wang. Binbin Zhang, Fan Yu, Chao Yang. Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei. WeNet: Production Oriented Streaming and -on-atrea End-to- End Speech Recognition Toolkit. In Pruc. Interspeech, pp. 4054-4058, 2021.
Zhao You, Shulin Feng, Dan Su, and Dong Yu. 3m: Multi-loss, multi-path and multi-level neu- ral networks for speech recognition. In 2022 13th International Symposium Chinese Spoken Language Processing (ISCSLP), pp. 170-174. IEEE, 2022.
Piotr Zelasko, Daniel Povey, Jan Trmal, Sanjeev Khudanpur, et al. Lhotse: a speech data represen- tation library for the modern deep learning ecosystem. arXiv preprint arXiv:21 10.12561. 2021.
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 6182-6186. IEEE, 2022a.
Binbin Zhang, Di Wu, Zhendong Peng. Xingchen Song, Zhuoyuan Yao, Hang Lv, Lei Xie, Chao Yang, Fuping Pan, and Jianwei Niu. Wenet 2.0: More productive end-to-end speech recognition toolkit. arXiv preprint arX1v:2203.15455, 2022b.
Frank Zhang, Yongqiang Wang, Xiaohui Zhang, Chunxi Liu, Yatharth Saraf, and Geoffrey Zweig. Faster, simpler and more accurate hybrid asr systems using wordpieces. arXiv preprint arXiv:2005.00 2000000009500.. 2020a.
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer coders and rnn-t loss. In IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pp. 7829-7833, 2020b.
Ying Zhang, Mohammad Pezeshki, Philémon Brakel, Saizheng Zhang, Cesar Laurent Yoshua Ben- gio, and Aaron Courville, Towards end-to-end speech recognition with deep convolutional neural networks. arXiv preprint 170.0007. 2017.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1
SCALEDADAM OPTIMIZER
A.1.1 SCALEDADAM ALGORITHM.
Algorithm/ ScaledAdam Algorithm. RMS refers to root-mean-saje -quare function. of refers to gi Gg.- a, controlled by Eden learning rate schedule. Good default settings are 31 = 0.9, [3 0.98, 17 0.1, and 6 = 10
Require: learning rate a;; exponential decay rates for the moment estimates 31, 32 E [0, 1): scaling factor the learning rate for parameter scale yy; objective function f(0) with parameters A; initial parameter 0n-
Initialize step. Initialize first and second moment of parameter gradient. Initialize first and second moments of parameter scale gradient. Initialize parameter scale.
mo f- 0, vo - 0
mo 0, wy
To < RMS(Oe)
while O₂ not converged do t + 1+1
g: < (o.:(0:-1)
D Get parameter gradient. D. Get parameter scale gradient. b Update the parameter scale. Update first moment of parameter gradient. b Update second moment of parameter gradient.
h4 + g: 0:11
Tt-1 t RMS( (0,-)
my 31 m1-1 + (1 B1) g
V. = 82 V + (1 32) gi
-1-
A -a T(-1
Compute parameter change.
(1 1-8
Vir+t
n, + 81 n--1 +
B1) lig
b Update first moment of parameter scale gradient.
Wt Wh W4-1
32) hi
Update second moment of parameter scale gradient.
1-
Arr tr
04
1-31
The
0.-1
b Compute parameter change by updating
parameter scale.
0, < 0,-1 + ! + A
> Update parameter.
end while
A.1.2
COMPARISON BETWEEN SCALEDADAM AND ADAM.
335
Adam, Guaue
Adm, Guaeed
5.0
Adati, Rbew 7.5
Adati, Oue
Adam -00
Scoledhikas... Seaw
475
Sea Sesleddiibmm... 0.005
ScaledAdam 0.045
Sialiddfiin 034
Leaning eH
4.50
Obess
Obew
602
3755
001
3.50
10
20
25
35
50
20000
40300
55000
$0003
100000
120000
Epuch
Stop
Figure A.2: Comparison between ScaledAdam and Adam terms of: (Left) averaged WER on LibriSpeech test-clean and test-other at different epochs; (Right) learning rate at different steps.
13

---

Published as a conference paper at ICLR 2024
A.2
ACTIVATION FUNCTIONS
Swish
SwooshR
2-
SwooshL
0
10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
10.0
X
Figure A.3: The activation functions: Swish, SwooshR, and SwooshL.
A.3 ACTIVATION CONSTRAINTS
Table 6: Ablation studies of activation constraints for Zipformer-M on LibriSpeech dataset. All models are trained for 40 epochs.
Ablation
test-clean (%)
test-other (%)
Zipformer-M
2.21
4.91
No Balancer
2.23
4.97
No Whitener
2.25
5.15
No Balancer, No Whitener
2.25
5.07
To ensure consistency in training and avoid badly trained modules, we propose the Balancer and the Whitener, which regularize the activations in memory-efficient way. In forward pass, they are no-ops; in backward pass, they compute the gradients of the additional losses putting constraints on the activations g', and add that the origin activation gradients g: g g + g'. The placements of Balancer and Whitener may not seem to follow any very clear rules. They are generally applied when encountering specific instances of not-normally- normally-trained models model divergence. We first locate the abnormality to specific module and then add the Balancer or Whitener to fix it.
A.3.1 BALANCER
Two failure modes commonly observed from the channel-wise statistical distribution are: 1) the issue of activation values becoming excessively large small give rise instability during the training process, particularly when employing mixed precision training; 2) a significant number of "dead" neurons, whose outputs consistently remain negative, was observed upon examining the channel-wise statistics prior to the application of the non-linear activation function within the feed- forward modules. Balancer solves these issues by enforcing four constraints on each channel: lower and upper bounds of the mean value of absolute values, denoted as amin and amaxi minimum and maximum proportion of positive values, denoted as Pmin and Pmax respectively. Given the activation x, intuitively we have E[x] x X, where À represents the proportion of positive values. Due to the non-differeeniable nature of positive value counting operation, shifted version of Gaussian error function erf is introduced in order to approximate the mapping between E[x] E (-o0, o0) and À E [0, 1] by cx) 1, the inverse function which can be approximated using f poee-B/VVar (x) = arctanh(22 (2x ()/(VT log 2) without loss of generality. Mmin = f /Vrr (Pmin) and Pmax fpos - VVr (Pmax) can be further derived from the approximation. Following the same Gaussian assumption, the RMS given by 1 o² e- (x==))² abs(x)dx, where μ and o² refer to the mean and variance of the Gaussian distribution. It can be approximated using fans- = / x when μ Thus Tiin Tabs->RMa (Mmin) and Fmax fabs- (0max) be further derived.
VIr
14

---

Published as a conference paper at ICLR 2024
Specifically, the additional loss Lalancer conditioned these constraints is defined as:
LiMs = log(min(max Tmmax) ('minn/ /RMm))))))
/a/ [[(x)
Var x clamp(E|x)) Var x Pmin Pmax
(9)
Lbllancer LRSS + E//Varr
where the statistics RMS[x). E[x], and / /ar[x]] calculated in each channel. Before adding the additional gradient g' = Vtaaanneeer to the original activation gradient g. g' is scaled to g' = g' a a/RMS(gg") |g]. Herein, is used to prevent g' from overwhelming g, and the per-element magnitude gg is used to prevent the model from concentrating its "fixes" to the data distribution in frames with small gradients such as the padding frames. We a 0.04 in this work.
A.3.2 WHITENER
Another failure mode activations is that for the feature covariance, are dominating while others are extremely small. This tends to happen in a model that is about diverge. Whitener encourages a more informative output distribution, by restricting the feature covariance after mean subtraction to have less unequal eigenvalues. Specifically, for output x E ND with N frames of D-dimensional features, we first compute the covariance matrix C' (x E[x))*((x (x-))""xxEEe E[x]), where C E R RDD and E[[] is per-channel mean. The auxiliary loss which measures the whiten metric Lwhiten is defined as:
few eigenvalues
whiteenr E X2DDI// [WPD)
(10)
where A = -A } are the eigenvalues of the covariance matrix C. To keep the original activation gradient g dominant after adding the additional gradient g' VXLChiteners g'' is scaled to g' = g' ( (g), where e² denotes the L2 norm, and a is set to 0.01. The modification g g + g' is done only when the whiten metric Lwhiten is above certain value Wmin prevent the model from learning pathological activation distributions. We usually set Wmin to 10.
A.3.3 ABLATION STUDIES.
We perform ablation experiments on LibriSpeech dataset to validate the effect of Balancer and Whitener. presents the experimental results. All models are trained for 40 epochs. Removing Balancer does not lead to obvious change model performance. However, it would increase the risk of model divergence without the value range constraints especially when employing mixed precision training. Removing Whitener results in 0.04% and 0.24% WER reduction test-clean and test-r respectively. This indicates that restricting the feature covariance to have less unequal eigenvalues in Whitener boost performance.
A.4 EXPERIMENTS ON LIBRISPEECH DATASET
A.4.1 TRAINING CONFIGURATIONS ZIPFORMER MODELS
Before training, the Mel filter-bank features are per-computed and saved disk. In training, use Buieeiiccccceennnnnnann
in Lhotse toolkit (Zelasko et al. 2021) to form the batches, where the batch size is determined dynamically given the constraint of the maximum total speech duration (in seconds). Table 7 presents the training configurations of Zipformer models on LibriSpeech dataset with speed perturbation with factors of 0.9, 1.0, and 1.1.
A.4.2 COMPARISON WITH STATE-OF-THE-MNE MODELS
As an extension Table Table S adds the results on LibriSpeech dataset for Zipformer with CTC and CTC/AED architectures respectively. For the Zipformer CTC/AED model, we use 6- layer Transformer as AED decoder, each layer with attention dimension of 512, attention heads number of 8, and feed-forward hidden dimension of 2048. The Zipformer CTC models trained for 100 epochs while the Zipformer CTC/AED models are trained for 50 epochs. Detailed training configurations provided in Section A.4.1
15

---

Published as a conference paper at ICLR 2024
Table Training configurations of Zipformer models on LibriSpeech dataset.
Type
Params (M)
Max duration (5)
GPUs
Epochs
Training time epoch (m)
Zipformer-S
CTC
22.1
1700
2 3.32 Tesla V100
100
Zipformer-M M
CTC
64.3
1400
4 3,2 Tesla V100
100
60
Zipformer-A.
1200
4.323
7ipformer
CTC/ARD
46.3
1700
2 32G Tesla V100
50
105
Zyplonner-M
CTC/AEDD
90.0
1200
4 32G Tesla V100
50
7ipformer
CTC/ARD
Zipformer-S
pruned transducer
23.3
1500
2 32G Tesla V100
50
Lipformer-M
pruned transducer
65.6
1000
4 32G Tesla. V100
50
Zipformer
pruned transducer
1000
Tesla V100
50
Z.pfomer-..
pruned transducer
148.4
2200
8 800 Tesla 100
200
18
For the CTC systems, Zipformer-M outperforms Squeezeefrrmr-ML on both test sets with only about half the number of parameters, and Zipformer-L also surpasses Squeezeformer-L by 0.27% on test-other with fewer parameters. For CTC/AED systems, Zipformer-M outperforms Conformer models and Branchformer, while Zipformer-I. achieves comparable results with B-Brrnchformer--L. Note that presented in Figure Zipformer-L. is much more efficient than F--rraccformerr1..
Table 8: WER(%) ()) comparison between different models on LibriSpeech dataset. We also in- clude the number of parameters and FLOPs encoder for a 30s input audio measured with Deep- Speed (Raseey et al. 2020). "Trained with 8 80G NVIDIA Tesla A100 GPUs for 170 epochs.
Model
Type
Params
GFLOPs
lest-clean (%)
test-other
Squerzeformer- XS Kim al.
2023
CTC
9.0
18.2
3.74
9.09
Squeezeformer--
Kma 2044
CTC
18.6
33.7
3.08
7.47
Squeezeformer-SM
Kim et 2022
CTC
47.6
2.79
Squeezeformer--M M
Kim
2022
CTC
55.6
88.4
2.56
6.50
Squeezeformer-Me ML Killi et
2022
CTC
125.1
183.3
2.61
6.05
Squeezeformer--L
Kim ct
CTC
236.3
333.7
2.47
E-Branchformerr-BB Kim
2023
CTC/АED
78.1
2.49
Branchformer (Peng
CTС/AED
116.2
238.3
2.4
5.5
E- E-Branchforme-L1 Kilh m1 2023
CTC/AED
148.9
284.4
2.14
4.55
Conformer-S Gulari et al. 2020
transducer
10.3
2.7
6.3
Confommer-N. M Gulaa el al.
transducer
2.3
Conformer- Gulai cl
20201
transducer
118.8
2.1
4.3
Conformer in WeNet iZhung et ul.
20226
CTC/АED
121.3
2.66
6,53
Conformer in ESPnet (Myazzkii el al. 2023
CTCЛAED
113.2
2.29
5.13
SonfornerSS
pruned transducer
29.1
3.75
9.24
Conformer-M
pruned transducer
28.4
77.0
2.96
7.11
Conformer-L
pruned transducer
122.5
294.2
2.46
5.55
Zipformer-S
CTC
22.1
40.8
2.85
6.91
Zipformer-M
CTC
2.51
Zipformer-1.
CTC
147.0
107.7
2.49
5.7
Zipforner-S
CTC/AED
46.3
40.8
2.46
6.04
Zipformer-M
CTC/AED
90.0
62.9
2.22
4.97
Zipformer-L.
CTC//ED
107.7
Zipformer-S
pruned transducer
23.3
40.8
2.42
5.73
Zipformer-M
pruned transducer
65.6
62.9
2.21
4.79
Zipformer-L.
pruned transducer
148.4
107.7
2.06
Zipformer-L
pruned transducer
148.4
107.7
2.00
16


Reviews:
[{"review_id": "ZwS2e1Mq9j", "reviewer": "Reviewer_UyUP", "summary": "This paper proposes a new encoder architecture based on conformer for speech recognition task. Some of the newly introduced modifications are well motivated based on some findings on trained the original conformer models. The experiments show strong performance compared with original conformer model and multiple other conformer variants. Considering the popularity of conformer for speech recognition and other speech tasks, the findings and contribution of this paper could benefit the speech/audio community a lot!", "questions": "1. Curious about whether the authors tried the new activation functions, biasnorm and scaledadam on other tasks? Do they still show superiority?\n2. How could you make the model work in streaming fasion?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A newly designed conformer variant that achieve SOTA performance on speech recognition task.\n2. Experiments are done on different datasets with training data at different scales (hundreds, thousands and tens of thousands).\n3. Experimental results are strong, indicating the effectiveness of model.", "weaknesses": "1. While the motivation for biasnorm and scaledadam are well explained, the motivation for zipformer blcok, especially those downsampling and upsampling modules are not well presented. \n2. The results on aishell1 is not quite convincing compared to other conformer variants. The author could elaborate more on the performance. Could be that this is a small dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTniyKU9qm", "reviewer": "Reviewer_6jVU", "summary": "Context: Automatic speech recognition (ASR), where Conformer currently is the state-of-the-art encoder, which is used by most groups. A new variant of the Conformer is proposed in this paper, along with a lot of changes.\n\n- New ZipFormer model as an alternative to the Conformer.\n- BiasNorm as an alternative to LayerNorm.\n- Bypass module instead of residual connections\n- Downsample module instead of simply average or max pooling or striding\n- Re-use of attention weights inside a ZipFormer block\n- Non-linear attention instead of standard attention, which uses a multiplicative element\n- New activation functions SwooshR and SwooshL as an alternative to Swish.\n- Additional learnable scalar for the scale of weights\n- ScaledAdam optimizer as an alternative to Adam.\n- Eden learning rate scheduling which is both step-based and epoch-based, as an alternative to purely step-based or purely epoch-based LR schedules.", "questions": "The capitalization is a bit inconsistent. Transformer is always lower case (transformer), while Conformer, ZipFormer etc are all capitalized.\n\nIt would make sense to state that (or whether) the code is published (even if you want to put the link only in the final version).\n\nHow long does the training take? What batch sizes are used?\n\n\n> Downsample module averages every 2 frames with 2 learnable scalar weights (after softmax normalization)\n\nI don't exactly understand how this is done. For the whole module, there are two learnable scalar weights only? So it could learn to always take really the average, but also to always take the first and ignore the second, or vice versa? This does not make sense to me. Why does it make sense to learn that? What happens if you just take the average? Or maybe use max-pooling instead? What is the influence of this?\n\n\nIf you downsample at the very end to 25 Hz, why not do that earlier, and downsample right in the beginning to 25 Hz, and then never to 50 Hz again? That would make it quite a bit faster. So how much worse does this get?\n\n\nLayerNorm/BiasNorm: When there is a large value, as even explicitly modelled by the bias in BiasNorm, isn't this a problem, that there is a large number in the denominator, so then all the values become very small? The exp(γ) can compensate that then again, however, I wonder if there are numerical problems.\n\nWhat is the influence of exp(γ) vs just γ (without exp) in BiasNorm?\n\n\nHow does ScaledAdam compare to methods like WeightNorm, where the weights are also normalized?\n\n\n> Since Adam is nearly invariant to changes in the gradient scale, for simplicity we replace this with ht = gt ·(rt−1 ⊙θt′−1) = gt ·θt−1\n\nI'm not sure that this would make the code simpler, to have this special case? But I also wonder, what is the influence of this?\n\n\nFor ScaledAdam, as I understand, adding the learnable parameter scale would actually change the model, as this scale now is also part of the model? Is this added for every parameter, or only some selected ones, e.g. the linear transformations?\n\nWhat is the influence of the Eden learning rate scheduling?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- There are a lot of interesting novelties here in the paper, like the ZipFormer model itself, ScaledAdam, BiasNorm, new activation functions, and more. (Although having so many different novelties is also a weakness, see below.)\n\n- Improvements are very good, i.e. good relative WER improvements, while also having it more efficient.\n\n- Good ASR baselines.", "weaknesses": "- There are maybe too many new things being introduced here, which are all interesting by themselves, but each of them would maybe require more investigation and analysis on their own. E.g. introducing a new optimizer (ScaledAdam) is interesting, but it should be tested on a couple of different models and benchmarks, and this would basically a work on its own. Now we have way too little analysis for each of the introduced methods to really tell how good they are. The ablation study is basically only a single experiment, showing ScaledAdam vs Adam, or LayerNorm vs BiasNorm. A single experiment is not really enough to really see how ScaledAdam vs Adam performs, or BiasNorm vs LayerNorm, etc. E.g. replaying LayerNorm by BiasNorm in a standard Transformer, maybe for language modeling, how would this perform? Many of these introduced methods seem to be quite orthogonal to each other.\n\n- For comparisons, it would have been better to use a more standard transducer, not a pruned transducer. Or maybe even CTC, or CTC/AED. This would have made it easier to compare the results to other results from the literature (Table 2).\n\n- Measuring just FLOPs can sometimes be misleading as indicator for speed, because certain operations might be faster than others, and certain operations can be executed in parallelized, while others can not. It would be interesting to also see the actual measured speed on some given hardware.\n\n- No systematic comparison to other Conformer variants, e.g. E-Branchformer, etc. (I don't just mean to have the number from literature as a comparison, as done in Table 2. I mean that the differences are compared, i.e. discussed and maybe also experimentally compared.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s7cFHk7Thm", "reviewer": "Reviewer_2d3G", "summary": "This paper presents an interesting alternative, zipformer, to the transformer/conformer encoder which are widely used for automatic speech recognition. The author also claims their modified optimizer, ScaledAdam, learns faster and converges to better optimum than the standard adam. The authors have presented that with the proposed zipformer and together with the modified optimizer, the resultant system achieves similar performance compared to conformer, but with better memory and FLOPS efficiency.", "questions": "- In Table 5 and section 4.0.3, it is unclear to me why \"no temporal downsampling\" will result in increased #params (from 65.6M to 94.2M). All the major components in zipformers: attention , feedfoward and convolution use the same parameters for different temporal resolution.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work presents an alternative approach to the standard (or widely used) transformer encoder structure, which is very interesting to read. The authors explain motivations behind some of the proposed modification, together with the ablation studies.", "weaknesses": "From my biased view, the major weakness in this paper lies in the fact that this work actually presents two inter-connected but (arguably) separate works: one is to the novel new encoder structure, including the U-net with middle stacks operate at a lower frame rates, sharing the attention weight with two self attentions, a novel non-linear attention, a BiasNorm and a slightly modified swooshL/swooshR activation function; the other is about the modified Adam optimizer, scaledAdam, which the authors claim that it can explicitly learn the parameter scale, which the widely used adam failed to. Though the author gives some motivations behind these changes and give some ablation studies, it still points to the following unanswered questions: \n\n  - the authors claim that the zipformer is better than the (reproduced) conformer, squeezeformer or other variants of transformer. However, it is unclear to me whether this is because of the model structure or because of the modified optimizer ? For example, in Table 5, the author presents that with all the proposed change (model and optimizer), zipformer-M can achieve 4.79% WER on test-other, however, with the standard adam optimizer, the WER becomes 5.51%. Will the other variants of transformer get better WER with the author proposed optimizer ? \n\n  - On the other hand, I am wondering whether the proposed optimizer, which has many adhoc modifications compared with the much widely used adam optimizer, can be more widely used for other deep learning optimization tasks or does it only work for speech tasks or does it only work for zipformer ? Given the zipformer-L is a pretty small model (only 60M parameters) trained on limited data (<1000hrs) according to today's standard,  will the proposed change still work better than the standard adam when the model and the training data scales up? \n\n\nOther weakness of this paper include: \n\n  - The authors propose changes to many widely used components, which have been well tested over time and over different tasks beyond just speech recognition. For example, the authors claim that the mean extraction in layer norm is not necessary and also remove the variance normalization. While this may seem to work for authors use case, I am intrigued to learn how generalizable this claim can be. \n\n  - Some of the proposed modification seem arbitrary. It would be great if the authors can explain a bit. For example, in Eq (4), for the swooshR and swooshL functions, how the -0.08 is selected ? The authors mentioned that the coefficient 0.035 in Eq (4) is slightly tuned. How general is this parameter ? \n\n- In the ablation study, quite a few modifications results in less than 0.2% absolute WER change. How sensitive of the WERs on librispeech relative to other factors like checkpoint choosing, different training run with different seed ? In my own experiments, these factors can result in up to 0.1% WER changes.\n\n\n- The authors also miss a series of work in the literature that directly apply standard transformer (without much modification) for ASR. For example, \n   * Y. Wang, A. Mohamed, D. Le, C. Liu et al., “Transformer-based Acoustic Modeling for Hybrid Speech Recognition,” in Proc ICASSP, 2020, it reports a WER of 2.26% / 4.85% on test-clean/test-other of librispeech, achieved by CTC + external NNLM; \nThis is followed by a work in: \n  * Zhang, Frank, et al. \"Faster, simpler and more accurate hybrid asr systems using wordpieces.\" arXiv preprint arXiv:2005.09150 (2020).\nwhich achieves a WER of 2.10% and 4.20% using a similar architecture. \n\nIt is also noted that in \n\n- Wang, Y., Chen, Z., Zheng, C., Zhang, Y., Han, W. and Haghani, P., 2023, June. Accelerating rnn-t training and inference using ctc guidance. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.\n\nIt achieved almost the same WER as the initial conformer paper, but part of the conformer layers (10 out of total 17 layers) is running at much lower frame rate (about 120ms-160ms per frame). This is related to the temporal downsampling used in zipformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Es0ctYQdN4", "reviewer": "Reviewer_hvPD", "summary": "This paper offers a number of incremental improvements over the well-known Conformer architecture. In aggregate, these improvements set a new state of the art for supervised Librispeech.", "questions": "The comparison to the original conformer is not quite apples-to-apples, in that Zipformer-L is substantially larger than Conformer-L in both the original publication and in this paper's reproduction.  I think the authors should consider including a variant of Zipformer with ~120M parameters, as originally used by Gulati et al.   This would prevent the suspicion that the stronger WER simply reflects the fact that the Zipformer is larger.\n\nStill, I think Zipformer's substantially reduced computational needs demonstrate that superiority over Conformer does not only reflect model size.  So, I'll recommend this paper for acceptance despite the flaw in this comparison.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper makes the most significant type of contribution in ASR - it improves the state of the art on a well-studied benchmark.  Since it has done so with a collection of small improvements, it offers detailed ablations quantifying the usefulness of each modification.  Importantly, it appears that each modification by itself offers improvement over Conformer.", "weaknesses": "In my option, this paper has the same problem that it identifies in the original Conformer write-up: It's closed-source, complicated, and it will likely not be possible for the results to be reproduced.  This is not by itself disqualifying; it is the nature of the field that SOTA systems are often published without source code.  However, I expect that future work iterating on this design will struggle with direct comparison, much in the same way that this paper compares to a weaker reproduction of the original Conformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)