Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
Authors: Sehoon Kim
Amir Gholami
Albert Eaton Shaw
Nicholas Lee
Karttikeya Mangalam
Jitendra Malik
Michael W. Mahoney
Kurt Keutzer
Venue: NeurIPS 2022 Accept (2022)
Keywords: ASR
speech recognition
efficient model
Transformers

Abstract: The recently proposed Conformer model has become the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture that captures both local and global features. However, through a series of systematic studies, we find that the Conformer architecture’s design choices are not optimal. After re-examining the design choices for both the macro and micro-architecture of Conformer, we propose Squeezeformer which consistently outperforms the state-of-the-art ASR models under the same training schemes. In particular, for the macro-architecture, Squeezeformer incorporates (i) the Temporal U-Net structure which reduces the cost of the multi-head attention modules on long sequences, and (ii) a simpler block structure of multi-head attention or convolution modules followed up by feed-forward module instead of the Macaron structure proposed in Conformer. Furthermore, for the micro-architecture, Squeezeformer (i) simplifies the activations in the convolutional block, (ii) removes redundant Layer Normalization operations, and (iii) incorporates an efficient depthwise down-sampling layer to efficiently sub-sample the input signal. Squeezeformer achieves state-of-the-art results of 7.5%, 6.5%, and 6.0% word-error-rate (WER) on LibriSpeech test-other without external language models, which are 3.1%, 1.4%, and 0.6% better than Conformer-CTC with the same number of FLOPs. Our code is open-sourced and available online.

Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
Sehoon Kim"!, Amir Gholami"!, Albert Shawf!l, Nicholas Lee3l, Karttikeya Mangalam',
Jitendra Malik?, Michael W. Mahoney 123 Kurt Keutzer*
Unniversity of California, Berkeley PICSI 3LBNL
(sehoonkim, amirgh, micholas_les lee, mangalam, malik, mahoneymw, keutzer)aberkeley,
Albertshaw@google. com
Abstract
The recently proposed Conformer model has become the facio backbone model for various downstream speech tasks based on hybrid attention-convolution architecture that captures both local and global features. However, through series systematic studies, we find that the Conformer architecture's design choices are not optimal. After re-cxamining the design choices for both the macro and micro-architecture of Conformer, we propose Squeezeformer which consistently outperforms the state-of-the offthe-art art ASR models under the same training schemes. In particular, for the macro-are aachtearchite Squeezeformer incorporates (i) the Temporal U-Net structure which reduces the cost of the multi-head attention modules on long sequences, and (ii) a simpler block structure of multi-head attention or convolution modules followed up by feed-forward module instead of the Macaron structure proposed in Conformer. Furthermore, for the micro-architecture, Squeezeformer simplifies the activations in the convolutional block, (ii) removes redundant Layer Normalization operations, and (iii) incorporates an efficient depthwise down- sampling layer to efficiently sub-sample the input signal. Squcezeformer achieves state-of-ther -fethh-rrt results of 7.5%, 6.5%, and 6.0% word-error-rate (WER) on Lib- riSpeech test-other without external language models, which are 3.1% %. 1.4%, and 0.6% better than Conformer- CTC with the same number of FLOPs. Our code is open-sourced and available online [25].
1 Introduction
The increasing success of end-to-end neural network models has been huge driving force for the drastic udvancements in various automatic speech recognition (ASR) tasks. While both convolutional neural networks (CNN) [19, 27, 29, 36. 61] and Transformers [24, 31, 32, 56, 57] have drawn attention as popular backbone architectures for ASR models, each of them has several limitations. Generally, CNN models lack the ability to capture global contexts and Transformers involve prohibitive comput- ing and memory overhead. To overcome these shortcomings, Conformer [16] has recently proposed Transformer architecture. Due its ability synchronously capture global and local features from audio signals, Conformer has become the de facto model not only for ASR tasks, but also for various end-to-end speech processing tasks [17]. Furthermore, it has also achieved sate--ofttheart performance in combination with recent developments in self-supervised learning methodologies as well [37, 62]. While the Conformer architecture was introduced as an (RNN-T) [14] model in its original setting, it has been adopted with
novel canvolution-augmene
autoregressive
First author equal contribution.
Second author equal contribution.
Now at Google.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

71.7
72.0
58.4
58.4
42.7
6.5
GFLOPs
6.50
WER (%, log scale)
6.89
6.89
WER (%)
7,12
7.28
Simpliifee LayerNorm
DV Sep. Subsanpping
Um Activations
7.5
Temparal U-tte
Tic-Sylle Block
7,90
Madel Scale
10-
Squeszefomer (Ours)
CitriNet
Pasellng
11
Efficlent Conformer
Conformer CTC'
8.5
12
QuartzNet
13 10
30
50
100
200
300
Conformer-CTC.MM
Squeezeformer.M
GFLOPs (log scale)
Figure 1: (Left) We perform series of systematic studies macro and micro architecture to redesign the Conformer architecture towards our Squeezeformer architecture. The bars and the line indicate the WER LibriSpeech test-other dataset and the FLOPs, respectively. For each design modification, we strictly improve WER until our final Squeezeformer model outperforms Conformer by 1.40% WER improvement with the same number of FLOPs. See Tab. for the details. (Right) LibriSpeech test-other WER vs. FLOPs for Squeezeformer and other state-of-the-art ASR models. Conformer-CTC* is our own reproduction the best performance as possible and the others the reported numbers in their papers [4, 27, 36]. Our architecture scales well to smaller and larger models to constantly outperform other models by a large margin throughout the entire FLOPs range. See Tab. for the details. For both plots, the lower the WER, the better; however, we plotted in reverse for better visualization.
less critique to non-autoregressive schemes such as Connectionist Temporal Classification (CTC) [15] well [38].
Despite being a key architecture in speech processing tasks, the Conformer architecture has some limitations that can be improved upon. First, Conformer still suffers from the quadratic complexity of the attention mechanism limiting efficiency on long sequence lengths. This problem further highlighted by the long sequence lengths of typical audio imputs also pointed out in [46]. Further- more, the Conformer architecture is relatively more complicated than Transformer architectures used in other domains such in natural language processing [7, 44, 51] computer vision [9, 10, 49]. For instance, the Conformer architecture incorporates multiple different normalization schemes and activation functions, the Macaron structure [34], well as back-to-back multi-head attention (MHA) and convolution modules. This level of complexity makes it difficult efficiently deploy the model on dedicated hardware platforms for inference [26, 39, 55]. More importantly, this raises the question whether such design choices are necessary and optimal for achieving good performance in ASR tasks.
In this paper, we perform a careful and systematic analysis of each the design choices with the goal of achieving lower word-error-rate (WER) for given computational budget. We developed a much simpler and more efficient hybrid attention-coovvllttoo architecture in both its macro and micro-design that consistently outperforms the state-of thrtttrrrt ASR models. In particular, make the following contributions our proposed Squeezeformer model:
We find high temporal redundancy in the learned feature representations of neighboring speech frames especially deeper in the network, which results in unnecessary computational overhead. To address this, we incorporate the temporal U-Net structure in which downsampling layer halves the sampling rate at the middle of the network, and a light upsampling layer recovers the temporal resolution at the end for training stability ($ 3.1.1).
We redesign the hybrid attention-comvolutiin architecture based on observation that the back- to back MHA and convolution modules with the Macaron structure are suboptimal. particular, we propose simpler block strueture similar to the standard Transformer block [7, 51], where the MHA and convolution modules are each directly followed by single feed forward module 3.1.2).
We finely examine the simplify the model overall and greatly improve the accuracy and efficiency. This includes (i) activation unification that replaces GLU activations with Swish ($ 3.2.1), (ii) Layer Normalization
of the network and found several modifications that

---

NOSLH
Decoder
Pret
Conformer
Feed Fonward Module
1/
Squeezeformer
40ms
Feed Forward Module
PreUN
Conformer
Convolution Mocule
Squeezetormer Block
80ms
Convelution Medule
Block
40ms
Conformer Block
40ms
MHA Module
Squeezeformer Block
40ms
Feed Forward Module
PrefNN
Convelution
DW Sep. Conv Subsampling
Subsampling
Feed Forward Modulc
MHA Modulc
Conformer
Squeezeformer
Figure 2: (Left) The Conformer architecutre and (Right) the Squeezeformer architecture which comprises of the Temporal U-Net structure for downsampling and upsampling of the sampling rate, the standard Transformer-style block structure that only uses Post-Layer Normalization, and the depthwise separable subsampling layer.
simplification by replacing redundant pre-Layer Normalization layers with scaled post-Layer Normalization which incorporates learnable scaling for the residual path that can be merged with other layers to be zero-cost during inference ($ 3.2.2), and (iii) incorporation of a depthwise separable convolution for the first sub-sampling layer that results in significant floating point operations (FLOPs) reduction ($ 3.2.3).
We show that the Squeezeformer architecture scales well with both smaller and larger models and consistently outperforms other state-of-the-ee -ft----rt---tttt-- ASR models when trained under the settings (Tab. 4.2, $ 4.2). Furthermore, we justify the final model architecture of Squeezeformer with a reverse ablation study for the design choices (Tab, 4.3).
Related Work
The recent advancements in end-to-end ASR can be broadly categorized into (1) model architecture and (2) training methods.
Model Architecture for End-to-end ASR. The recent end-to-end ASR models are typically composed of an encoder, which takes as input speech signal (i.e., sequence of speech frames) and extracts high-level acoustic features, and a decoder, which converts the extracted features from the encoder into sequence of text. The model architecture of the encoder determines the representational power of an ASR model and its ability to extract acoustic features from input signals. Therefore, a strong architecture is critical for overall performance.
One of the popular choices for backbone model architecture is convolutional neural network (CNN). End-to-end deep CNN models have been first explored in [29, 61], and further improved by introducing depth-wise separable convolution [21, 47, 50] in QuartzNet [27] and the Squeeze- and-Excitatio module [23] in CitriNet [36] and ContextNet [19]. However, since CNNs often fail to capture global contexts, Transformer [51] models have also been widely adopted backbone architectures due to their ability to capture long-range dependencies between speech frames [24, 31, 32, 56, 57]. Recently, [16] has proposed a novel model architecture named Conformer, which augments Transformers with convolutions model both global and local dependencies efficiently. With the Conformer architecture as our starting point, we focus on designing next- seett-eeneattion model architecture for ASR that is simpler, more accurate, and more efficient.
The hybrid
architecture of Conformer has enabled the state-of-th ofte--f--he-arr results in many speech tasks. However, the quadratic complexity of the attention layer still proves to be cost prohibitive at larger sequence lengths. While different approaches have been proposed to reduce the cost of MHA in ASR [5, 46, 58, 59], their main focus is not changing the overall architecture design, and their optimizations can also be applied our model, as they are orthogonal to the developments for Squeezeformer. Efficient-Confone [4] introduces the progressive downsampling

---

Table I: Starting from Conformer as the baseline, we redesign the architecture towards Squeezeformer through series of systematic studies on macro and micro architecture. Note that for each design change, the WER on LibriSpeech test-clean and test-other datasets improves consistently. For comparison, we include the number of parameters and FL.OPs for 30s input in the last two columns.
Model
Design change
test-clean test-other
Params (M)
GFLOPs
Conformer-CTCa
Baseline
3.20
7.90
27.4
71.7
Temporal U-Net (5 3.1.1)
2.97
7.28
27.5
57.0
+ Transformer-style Block (s 3.1.2)
2.93
7.12
27.5
57.0
Unified activations 3.2.1)
2,88
7.09
28.7
58.4
+ Simplified LayerNorm 3.2.2)
2.85
6.89
28.7
58.4
Squeezeformer-Sh Squeezefommer-M
DW sep. subsampling (8 3.2.3)
2.79
6.89
28.2
42.7
+ Model scale-up (s 3.2.3)
2.56
6.50
55.6
72.0
scheme and grouped attention to reduce the training and inference costs of Conformer. Our work incorporates similar progressive downsampling, but also introduces an up-sampling mechanism with skip connections from the earlier layers inspired by the U-Net [45] architecture in computer vision and the U-Time [43] architecture for sleep signal analysis. We find this be critical for training stability and overall performance. In addition, through systematic experiments, we completely refactor the Conformer block by carefully redesigning both the macro and micro-architectures.
Training Methodology for End-to-end ASR. In the past few years, various self-supervised learning methodologies based contrastive learning [2, 52, 62] or masked prediction [1, 6, 22] have been proposed to push forward the ASR performance. While model pre-trained with self-supervised tasks generally outperforms when finetuned on target ASR task, training strategies are the main focus in this work as they can be applied independently to the underlying architecture.
3 Architecture Design
The Conformer architecture has been widely adopted by the speech community and is used as backbone for different speech tasks. At macro-level, Conformer incorporates the Macaron structure [34] comprised of four modules per block, as shown in Fig. 2 (Left). These blocks are stacked multiple times to construct the Conformer architecture. In this work. carefully reexamine the design choices in Conformer, starting first with its macro-architecture, and then its micro- architecture design. We choose Conformer-CTC-N. we compare word-error-rate (WER) on LibriSpeech test-other as a performance metric for each architecture. Furthermore, we measure FLOPs on a 30s audio input proxy for model efficiency. While we acknowledge that FLOPs may not always be a linear indicator of hardware and runtime efficiency, we choose FLOPs is hardware agnostic and is statically computable. However, do measure the final end-to-end throughput of our changes, ensuring up to 1.34x consistent improvement in runtime for different versions of Squeezeformer (Tab. 4.2)
the baseline model for the case study, and
3.1 Macro-Architecture Design
We first focus on designing the macro structure of Squeezeformer, i.e., how the blocks and modules are organized in global scale.
3.1.1 Temporal U-Net Architecture
The hybrid attention--comvllttion structure enables Conformer to capture both global and local interactions. However, the attention operation has a quadratic FLOPs complexity with respect to the input sequence length. We propose lighten this extra overhead by computing attention over a reduced sequence length. In the Conformer model itself, the input sampling rate is reduced from 10ms to 40ms with convolutional subsampling block at the base of the network. However, this rate kept constant throughout the network, with all the attention and convolution operations operating constant temporal scale.
To this end, we begin by studying the temporal redundancy in the learned feature representations. In particular, analyze how the learned feature embeddings per speech frame differentiated through

---

1.0
Contormer
0.9
Temporni
Module 2
Module 2
Cosine Similarity
Scaling
Blokk
0.5
PostLN
PostLN
0.4
d1st
2
Module
Module 1
Black
10 Conformer Block Index
Figure 3: Cosine similarity between two embed- ding vectors of neighboring speech frames with varying adjacency distances across the Conformer blocks. The temporal dimension downsampled after the 7th block and upsampled before the 16th block in the Temporal U-Net structure.
Figure 4: (Left) Back-to-back preLN and postLN the boundary of the blocks. (Right) The preI.N can be replaced with the learned scaling that readjusts the magnitude of the ac- tivation that goes into the subsequent module.
the Conformer model depth. We randomly sample 100 audio signals from LibriSpeech's dev-other dataset, and process them through the Conformer blocks, recording their per-block activations. We then measure the average cosine similarity between two neighboring embedding vectors. The results are plotted as the solid lines in Fig. 3. We observe that the embeddings for the speech frames directly next to each other have an average similarity of 95% at the topmost layer, and even those 4 speech frames away from each other have a similarity of more than 80%. This reveals that there is an increasing temporal redundancy as inputs are processed through the Conformer blocks deeper in the network. We hypothesize that this redundancy in feature embedding vectors causes unnecessary computational overhead and that the sequence length can be reduced deeper in the network without loss in accuracy.
As our first macro-arce aacht-architecture improvement step, we change the Conformer model to incorporate subsampling of the embedding vectors after it has been processed by the early blocks of the model. In particular, we keep the sample rate to be 40ms up to the 7th block, and afterwards we subsample to a rate of 80ms per input sequence by using pooling layer. For the pooling layer we use depthwise separable convolution with stride 2 and kernel size 3 to merge the redundancies across neighboring embeddings. This decreases the attention complexity by 4x and also reduces the redundancies of the features. This temporal downsampling shares similarities with computer vision models, which often downsample the input image spatially to save compute and develop hierarchical level features [10, 20, 30, 48], and with the approach of Efficient Conformer [4].
However, the temporal downsampling alone leads to an unstable and diverging training behaviour ($ 4.3). One possible reason for this is the lack of enough resolution for the decoder after subsampling the rate to 80ms. The decoder maps an embedding for each speech frame into a single label, c.g., character, and therefore requires sufficient resolution for successful decoding of the full sequence. Inspired from successful architectures for dense prediction in computer vision such as U-Net [45], we incorporate the Temporal U-Net structure to recover the resolution at the end of the network through upsampling layer as shown in Fig. 2. This upsampling block takes the embedding vectors processed by the 40ms and 80ms sampling rates, and produces an embedding with a rate 40ms by adding them together via skip connection. To the best of our knowledge, the closest work to our Temporal U-Net is the approach proposed [43], in which the U-Net structure is incorporated into a fully-convolutional model to downsample sleep signals.
This change not only reduces the total FLOPs by 20% compared to Conformer! but also improves the oestr WER by 0.62% from 7.90% to 7.28% (Tab. 1, 2nd row). Furthermore, analyzing the cosine similarity shows that the Temporal U-Net architecture prevents the neighboring embeddings
'The total FLOPs for the entire model. If we just study the attention block, the Temporal U-Net structure reduces the FLOPs by 2.31 and 2.53x FLOPs reduction for processing 30s and 60s audio signals as compared Conforee-CTT CTC-M baseline, respectively.
5

---

from becoming too similar to each others at the later blocks, in particular at the final block directly connected the decoder, as shown in Fig. the dashed lines.
3.1.2 Transformer-Style Block
The Conformer block consists of sequence of feed-forwant ('F',, multi-head attention (MHA, 'M'), convolution ('C'), and another feed-forwant module ('F).. We denote this as the FMCF structure. Note that the convolutional kernel sizes in ASR models rather large, e.g., 31 in Conformer, which makes its behaviour similar to attention in mixing global information. This is stark contrast to convolutional kernels in computer vision, which often have small 3 x kernels and hence benefit greatly from attention's 's global processing. As such, placing the convolution and MHA module with similar functionality back-to-back (i.e., the MC substructure) does seem prudent. Hence, consider an MF/CF structure, which is motivated by considering the convolution module as local MHA module. Furthermore, we drop the Macaron structure [34], MHA modules followed by feed-forward modules have been more widely adopted in the literature [7. 9, 44, 51J. In a nutshell, we simplify the architecture to be similar the standard Transformer network and denote the blocks MF and CF substructures, as shown in Fig. 2. This modification further improves the test-other WER by 0.16% from 7.28% to 7.12% and marginally improves the test-clean WER without affecting the FLOPs (Tab. 1, 3rd row).
3.2 Micro-Architecture Design
So far we have designed the macro structure of Squeezeformer by incorporating seminal architecture principles from computer vision and natural language processing into Conformer. In this subsection, we now focus on optimizing the micro structure of the individual modules. We show that we can further simplify the module architectures while improving both efficiency and performance.
3.2.1 Unified Activations
Conformer uses Swish activation for most of the blocks. However, it switches to Gated Linear Unit (GLU) for its convolution module. Such a heterogeneous design seems over-complicated and unnecessary. From a practical standpoint, the use of multiple activations complicates hardware deployment, as an efficient implementation requires dedicated logic design, look up tables, or custom approximations [26, 39, 55J. For instance, on low-end edge devices with no dedicated vector processing unit, supporting additional non-linear operations would require additional look up tables or advanced algorithms [12, 13J. To address this, we propose to replace the GLU activation with Swish, unifying the choice of activation function throughout the entire model. We keep the expansion rate for the convolution modules. As shown in the 4th row of Tab. 1, this change does not entail noticeable changes in WER and FLOPs but only simplifies the architecture.
3.2.2 Simplified Layer Normalizations
Continuing our micro- architecture improvements, we note that the Conformer model incorporates redundant Layer Normalizations (LayerNorm), as shown in Fig. 4 (Left). This is because the Con- former model contains both a post-LayerNorm (postLN) that applies LayerNorm in between the residual blocks, as well as pre-LayerNorm (preLN) which applies LayerNorm inside the residual connection. While it is hypothesized that preLN stabilizes training and postLN benefits perfor- mance [53], these two modules used together lead to redundant back-to-back operations. Aside from the architectural redundancy, LayerNorm can be computationally expensive [26, 55) due to its global reduction operations.
However, we found that naively removing the preLN or postLN leads to training instability and convergence failure ($ 4.3). Investigating the cause of failure, we observe that typical trained Conformer model has orders of magnitude differences in the norms of the learnable scale variables of the back-to-back preLN and postLN. In particular, we found that the preLN would scale down the input signal by large value, giving more weight to the skip connection. Therefore, it is important to use a scaling layer when replacing the preLN component to allow the network to control this weight. This idea is also on par with several training stabilization strategies in other domains. For instance, NF-Net [3] proposed adaptive (i.e., learnable) scaling before and after the residual blocks to stabilize training without normalization. Furthermore, DeepNet [53] also recently proposed to add non-trainable rule-based scaling to the skip connections to stabilize PreLN in Transformers.

---

Table Detailed architecture configurations for Conformer- CTC (baseline) and Squeezeformer, For comparison, we include the number of parameters and FLOPs for 30s input in the last two columns.
Model
Layers
Dimension
Heads
Params (M)
GFLOPs
Conformer-CTC--S
16
144
4
8.7
26.2
Squeezeformer-XS
144
15.8
Squeezeformer-S
18
196
4
18.6
26.3
Conformer-CTC CTC-M
256
27.4
71.7
Squeezeformer-She
16
256
4
28.2
42.7
Squeezeformer-M
20
324
4
55.6
72.0
Conformer-CTC-L
18
512
121.5
280.6
Squeczeformer-ML.
18
512
8
125.1
169.2
Squeezeformer--L 22
640
236.3
277.9
Inspired by these computer vision advancements, we propose to replace preLN with a learnable scaling layer that scales and shifts the activations, Scaling(x) - 7r + with learnable scale and bias vectors "y and 3 of the size of feature dimension. For homogeneity of architectural design, then replace the preI.N throughout all the modules with the postL.Ntthen-caalnn as illustrated in Fig. 2 (Right) and make the entire model postLN-only. Note that the learned scaling parameters can be merged into the weights of the subsequent linear layer, as the architecture illustrated in Fig. (Right), and hence have zero inference cost. With the learned scaling, our model further improves the test-other WER by 0.20% from 7.09% to 6.89% (Tab. 1, 5th row).
3.2.3 Depthwise Separable Subsampling
We now shift our focus from the Conformer blocks to the subsampling block. While easy to overlook this single module at the beginning of the architecture, we note that it accounts for sigmificant portion of the overall FLOPs count, up to 28% for Conformer-CTC-. with 30-second input. This is because the subsampling layer uses two vanilla convolution operations each of which has stride To reduce the overhcad of this layer, we replace the second convolution operation with depthwise separable convolution while keeping the kernel size and stride the same. We leave the first convolution operation as is since is equivalent to depthwise convolution with the input dimension 1. This saves additional 22% of the baseline FLOPs without test-other WER drop and even a 0.06% improvement in test-clean WER (Tab. 1, 6th row). An important point to note here is that generally depthwise separable convolutions in part due its low arithmetic intensity. However, given the large FLOPs reduction, we consistently observe overall improvement in the total inference throughput of up 1.34x reported in Tab. as compared to the baseline Conformer models.
hard to efficiently map to hardware accelerators,
We name our final model with all these improvements Squeezeformer-SM Compared to Conformer- CTC-M, our initial baseline, Squeezeformer-SM improves WER by 1.01% from 7.90% to 6.89% with 40% less FLOPs. Given the smaller FLOPs of Squeezeformer-SM. we also scale up the model to similar FLOPs cost as Conformer CTC-M.. In particular, we scale both depth and width of the model together following the practice in [8]. Scaling up the model achieves additional test-other WER gain of 0.39% from 6.89% to 6.50% (Tab. 1, 7th row), and we name this architecture Squeezeformer-M.
4 Results
4.1 Experiment Setup
Models. Following the procedure described in $ 3, construct Squeezeformer variants with different size and FLOPs: we apply the macro and micro-architecture changes in $ 3.1 and $ 3.2, respectively, to construct Squeezeformer-X: SM, and ML from Conformer-S, M. and retaining the model size. Afterwards, we construct Squeezeformer-S M, and L by scaling up each model to match the FLOPs of the corresponding Conformer. The detailed architecture configurations described in Tab. 2.
While there are multiple options available for the decoder such as RNN- Transduure (RNN-T) [14] and Connectionist Temporal Classification (CTC) [15],
CTC decoder whose non-autoregressive
7

---

Table 3: WER (%) comparison on LibriSpeech dev and datasets for Squeezeformer and other state-of state-of-theeart CTC models for ASR including Conformer-CTC QuartzNet [27], CitriNet [36], Transformer-CTCC [31], and Efficient Conformer-CTC [4]. For comparison, we include the number of parameters, FLOPs, and throughput (Thp) on single NVIDIA Tesla A100 GPU for a 30s input in the last three columns. *The performance numbers for Conformer-CTC: are based on our own reproduction to the best performance as possible. All the other performance numbers are from the corresponding papers. With and wwithout the grouped attention.
Model
dev-clean
dev-other
test-clean lest-other
Params (M)
GFLOPs
Thp (ex/s)
Conformer-CTC [16]
4.21
10,54
4.06
10.58
8.7
26.2
613
QuartzNet 5x5 [27]
5.39
15.69
6.7
20.2
Citrinet 256 [36]
3.78
9.60
10.3
16.8
Squeezeformer-XS
3.63
3.74
9.09
9.0
15.8
763
Conformer-CTC-MM [16]
2,94
7.80
3.20
7.90
27.4
71.7
463
QuartzNet 5x10 [27]
4.14
12.33
12.8
38.5
Quartz 5x15 [27]
3.98
11.58
3.90
11.28
18.9
55.7
Citrinet 512 [36]
3.11
7.82
37.0
63.1
Rff. Conformer-CTCC [4]
3.57
8.99
13.2
26.0
Rff. Conformer-CTC: [4]
3.58
8.88
13.2
32.5
Squezzeformer-S
2.80
7.49
3.08
7.47
18.6
26.3
602
Squeezeformer-SM
2.71
2.79
6.89
28.2
42.7
Conformer-CTC-L: [16]
2.61
6.45
2.80
6.55
121.5
280.6
200
Citrinet 1024 [36]
2.52
6.22
143.1
246.3
Squeezeformer-M
2.43
6.51
2.56
6.50
55.6
72.0
431
Squeezeformer-ML.
2.34
6.08
2.61
6.05
125.1
169.2
268
Transformer-CTC [31]
2.6
7.0
2.7
6.8
255.2
621.1
Squeezeformer-L.
2.27
5.77
2.47
5.97
236.3
277.9
207
decoding method benefits training and inference latency [36]. However, the main focus of this work the model architecture design of the encoder, which can orthogonal to the decoder type.
Another subtlety when evaluating models the use external language models (LM). In many prior works [18, 24, 35, 40, 54, 56], decoders often augmented with external LMs such as pre-trained 4-gram or Transformer, which boost the final WER by re-scoring the outputs in a more lexically accurate manner. However, compare the results without external LMS to fairly compare the true representation power of the model architectures alone external LMs can be incorporated as an orthogonal optimization afterward.
Training Details. Because the training recipes and codes for Conformer have not been open-sourced, train it to reproduce the best performance numbers possible. We train both Conformer-CTI CTC and [41] for 500 epochs on Google's cloud TPUs v3 with batch
Squeezeformer on the LibriSpeech-9600 size 1024 for the small and medium variants and 2048 for the large variants. We use Adam [33] optimizer with weight decay 5e-4 for all models. More details for the training and evaluation setup given in & A.1 and A.2.
4.2 Main Results
In Tab. 3 compare the WER of Squeezeformer with Conformer-CTC and other state-of-ther the-art CTC-based ASR models including QuartzNet [27], CitriNet [36), Transformer [31], and Efficient Conformer [4] the clean and other datasets. Note that the performance numbers for Conformer- CTC are based on our own reproduction to the best performance as possible due to the absence of public training recipes codes. For simplicity, we denote WER test-clean/test-tther without % throughout the section.
Squeezeformer vs. Conformer. Our smallest model Squeezeformer-Xt outperforms Conformer- CTC-S by 0.32/1.49 (3.74/9.09 vs. 4.06/10.58) with 1.66x FLOPs reduction. Compared with Conformer-CTC-A. Squeczeformer-S achieves 0.12/0.43 WER improvement (3.08/7.47 vs.
"The WER results exhibit some differences from the original paper [16] due to the difference in decoder. The original Conformer RNN-T decoder, which known generally result in better WER than CTC [4, 36, 60).

---

Table 4: Ablation studies for the design choices made in Squeezeformer, including Temporal U-Net, LayerNorm, and activation in the convolution module. "Without the upsampling layer, the model fails to converge.
Ablation
Model
dev-clean
dev-other
Ours
Squeezeformer-M
2.43
6.51
Temporal U-Net ($ 3.1.1),
No skip connection
2.78
7.38
No upsampling
N/A"
N/A"
LayerNorm ($ 3.2.2)
PosILN only
5.60
14.00
PreLN only
3.02
8.27
Convolution module ($ 3.2.1)
No Swish
2.53
6.73
3.20/7.90) with 1.47x smaller size and 2.73x less FLOPs, and Squeezeformer-S5 SM further improves WER by 0.41/1.01 (2.79/6.89 vs. 3.20/7.90 with comparable size and 1.70x less FLOPs. Com- pared with Conformer-CTCAl Squeezeformer-M shows 0.24/0.05 WER improvement (2.56/6.50 vs. 2.80/6.55) with significant size and FLOPs reductions of 2.18x and 3.90x, respectively, and Squeezeformer-MI. shows 0. 09/0.55 WER improvement (2.61/6.05 vs. 2.80/6.55) with similar size and 1.66 less FLOPs, Finally, our largest model Squeezeformer-L. improves WER by 0.33/0.58 upon Conformer-CTI-L with the same FLOPs count, achieving the oate-of-the--n result of 2.47/5.97.
Squeezeformer vs. Other ASR Models. As can be seen in Tab. 3, our model consistently outperforms QuartzNet, CitriNet, and Transformer with comparable or smaller model sizes and FLOPs counts. A notable result is comparison against Efffcient-Conformen designed Efficient Conformer by large margin of 0.79/1.99 (2. 79/69/9 vs. 3.58/8.88 888/8.88) with the same FLOPs count. The overall results are summarized plot in Fig. 1 (Right) where Squeezeformer consistently outperforms other models across all FLOPs regimes.
our model outperforms the efficiently-
4.3 Ablation Studies
In this section, we provide additional ablation studies for the design choices made for individual architecture components using Squeezeformer-M M as the base model. See Tab. 4. Unless specified, use the same hyperparameter settings as in the main experiment.
Temporal U-Net. In the 2nd row of Tab. 4, the model clearly underperforms by 0.35/0.87 without the skip connection from the downsampling layer to the upsampling layer. This shows that the high-resolution information collected in the early layers is critical for successful decoding. The 3rd row in Tab. shows that our model completely fails to converge without the upsampling layer due to training stability, even with several different peak learning rates of (0.5, 1.0, 1.5
LayerNorm. In the 4th line of Tab. 4, we show that WER drops significantly by 3.17/7.49 when apply the PostLN-only scheme without the learned scaling layer. Another alternative design choice is apply the PreLN-only scheme without the learned scaling, which also results in a noticeable WER degradation of 0.59/1.76 as shown in the 5th line of Tab. 4. In both cases, the model fails to converge. so we report the best WER before divergence. The results suggest that the learned scaling layer plays a key role for training stabilization and better WER.
Convolution Module. When ablating the GLU activation in the convolution modules, another possible design choice is to drop it without replacing it with the Swish activation. This, however, results in 0. 10/0.22 worse WER shown in the last line of Tab. 4.
Conclusions
In this work, we performed a series of systematic ablation studies on the macro and micro architecture of the Conformer architecture, and we proposed a novel hybrid that is simpler and consistently achieves better performance than other models for a wide range of computational budgets. The key novel components of Squeezeformer's macro- archhieccceetee is the incorporation the Temporal U-Net structure which downsamples audio signals in the second half of the network to reduce the temporal redundancy between adjacent features and save compute, well as the MF/CF block structure similar to the standard Transformer-style which simplifies the architecture and improves performance. Furthermore, the micro-architecture of Squeezeformer simplifies the
architecture

---

activations throughout the model and replaces redundant LayerNorms with the scaled postLN, which more efficient and leads to better accuracy. We also drastically reduce the subsampling cost the beginning of the model by incorporating depthwise separable convolution. We perform extensive testing of the proposed architecture and find that Squeezeformer scales very well across different model sizes and FLOPs regimes, surpassing prior model architectures when trained under the same settings, Our code along with the checkpoints for all of the trained models is open-sourced and available online [25].
Acknowledgments
The authors acknowledge contributions from Dr. Zhewei Yao, Aniruddha Nrusimha, and Jiachen Lian. We also acknowledge gracious support from Google Cloud, Google TRC team, and specifically Jonathan Caton, Prof. David Patterson, and Dr. Ed Chi. We would also like to acknowledge the Sky Pilot team from UC Berkeley. Prof. Keutzer's lab is sponsored by Intel corporation, Intel VLAB team, Intel One-API center of excellence, as well as funding through BDD and BAIR. Sehoon Kim would like to acknowledge the support from Korea Foundation for Advanced Studies (KFAS). Amir Gholami was supported through funding from Samsung SAIT. Michael W. Mahoney would also like to acknowledge the UC Berkeley CLTC, ARO, NSF, and ONR. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.
References
[1] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03553 2022.
[2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. Wav2vec 2.0: framework for sulf--upervvsed learning of speech representations. Advances in Neural Information Processing Systems, 33:12449-12466 12460, 2020.
[3] Andy Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Muchine Learning, pages 1059-1071. PMLR, 2021.
[4] Maxime Burchi and Valentin Vielzeuf. Efficient Conformer: Progressive downsampling and grouped attention for automatic speech recognition. arXiv preprint arXiv:2109.01163 2021.
[5] Xuankai Chang, Aswin Shanmugam Subramanian, Pengcheng Guo, Shinji Watanabe, Yuya Fu- jita, and Motoi Omachi. End-to-end ASR with adaptive span self-attention. In INTERSPËECII, pages 3595-3599, 2020.
[6] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et WavLM: Large-scat self-supervised pre-training for full stack speech processing. arXiv preprint arXiv:2 13,0000, 2021.
[7] Jacob Devlin, Ming-Wei, Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understanding. arXiv preprint 11X.v.1810004805, 2018.
[8] Piotr Dollár, Mannat Singh, and Ross Girshick. Fast and accurate model scaling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 924-932, 2021.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
[10] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale Vision Transformers. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 6824-6835, 2021.
10

---

[11] John S Garofolo. Timit acoustic phonetic continuous speech corpus. Linguistic Data Consortium, 1993, 1993.
[12] Xue Geng, Jie Lin, Bin Zhao, Anmin Kong, Mohamed M Sabry Aly, and Vijay Chandrasekhar Hardware-aware softmax approximation for deep neural networks. In Asian Conference on Computer Vision, pages 107-122. Springer, 2018.
[13] Xue Geng, Jie Lin, Bin Zhao, Zhe Wang, Mohamed M Sabry Aly, and Vijay Chandrasekhar. Hardware-aware exponential approximation for deep neural network. 2018.
[14] Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint 1rXiv:1211.3711, 2012.
[15] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006.
[16] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang. Yonghui Wu, and Ruoming Pang. Conformer: Convolution- augmented Transformer for speech recognition. arXiv preprint urXiv:2005.08100 2020.
[17] Pengcheng Guo al. Recent developments on ESPNet toolkit boosted by Conformer. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5874-5878. IEEE, 2021.
[18] Kyu Han, Ramon Pricto, and Tao Ma. State-of-the-an speech recognition using multi-strean self-attention with dilated Id convolutions. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 54-61. IEEE, 2019.
[19] Wei Han, Zhengdong Zhang, Yu Zhang. Jiahui Yu, Chung-Chers Chiu, James Qin, Anmol Gulati, Ruoming Pang, and Yonghui Wu. ContextNet: Improving convolutional neural networks for automatic speech recognition with global context. arXiv preprint arXiv:2005.0319/ 2020.
[20] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 04861, 2017.
[22] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov. and Abdelrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. /EEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-344 -460, 2021.
[23] Jie Hu, Li Shen, and Gang Sun. ggueeze-and-Exxitaee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018
[24] Shigeki Karita et al. comparative study on Transformer vs RNN in speech applications. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 449-456. IEEE, 2019.
[25] Schoon Kim.
[26] Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-BERT: Integer-only bert quantization. In International conference on machine learning, pages 5506- 5518. PMLR, 2021.
[27] Samuel Kriman, Stanislav Beliaev, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, and Yang Zhang. QuartzNet: Deep automatic speech recognition with 1 time-channel separable convolutions, In ICASSP, pages 6124-6128. IEEE, 2020.
11

---

[28] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint 10X.v0880800622:: 2018.
[29] Jason Li, Vitaly Lavrukhin, Boris Ginsburg, Ryan Leary. Oleksii Kuchaiev, Jonathan M Cohen, Huyen Nguyen, and Ravi Teja Gadde. Jasper: An end-to-end convolutional neural acoustic model. arXiv preprini arviv: MoXiv:994...0 1904.03884,, 2019.
[30] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved Multiscale Vision Transformers for classification and detection. arXiv preprint arXiv:21
2021.
[31] Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap. Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking evaluation in ASR: Are our models robust enough? arXiv preprint arXiv:2010. 1 2020.
[32] Chunxi Liu, Frank Zhang, Due Le, Suyoun Kim, Yatharth Saraf, and Geoffrey Zweig. Improving RNN Transducer based ASR with auxiliary tasks. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 172-179. IEEE, 2021.
[33] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
[34] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving Transformer from a multi-partich dynamic system point of view. arXiv preprint arXiv: 027622 2019.
[35] Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, and Hermann Ney. RWTH ASR Systems for LibriSpeech: Hybrid vs attention-w/o 2019.
data augmentation. urXiv preprint 1rXivv/905..03
[36] Somshubra Majumdar, Jagadeesh Balam, Oleksii Hrinchuk, Vitaly Lavrukhin, Vahid Noroozi, and Boris Ginsburg. Citrinet: Closing the gap between non-autoregressive and autoregressive end-to-end models for automatic speech recognition. arXiv preprint arXiv:2104.01721, 2021.
[37] Edwin G Ng, Chung-Cheng Chiu, Yu Zhang, and William Chan. Pushing the limits of non- autoregressive speech recognition. arXiv preprint arXiv:2104.03416, 2021.
[38] NVDIA Nemo. https://gituub.coo/nviiiiiii/iit
[39] NVDLA Primer. http://awell..g//prrinc.m.. 2021.
[40] Jing Pan, Joshua Shapiro, Jeremy Wohlwend, Kyu J Han, Tao Lei, and Tao Ma. ASAPP- ASR: Multistream CNN and self-attentive SRU for SOTA speech recognition. arXiv preprint arXiv:2005. 10469, 2020.
[41] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ASR corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210, IEEE, 2015.
[42] Daniel S Park, William Chan, Yu Zhang, Boung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arxiv:1944 08779, 2019.
[43] Mathias Perslev, Michael Jensen, Sune Darkner, Poul Jørgen Jennum, and Christian Igel. U- Time: A fully convolutional network for time series segmentation applied sleep staging. Advances in Neural Information Processing Systems, 32, 2019.
[44] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018
[45] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
12

---

[46] Kyuhong Shim, Jungwook Choi, and Wonyong Sung. Understanding the role of self attention for efficient speech recognition. In International Conference on Learning Representations. 2021.
[47] Laurent Sifre and Stéphane Mallat. Rigid-motion scattering for texture classification. arXiv preprint arXiv: 1403. 1687, 2014.
[48] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
[49] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image Transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[50] Vincent Vanhoucke. Learning visual representations at scale. 2014.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] Chengyi Wang. Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng. and Xuedong Huang. Unispeech: Unified speech representation learning with labeled and unlabeled data. In International Conference on Machine Learning, pages 10937-10947. PMLR, 2021.
[53] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling Transformers to 1,000 layers. arXiv preprint urXiv:2203.00555 2022.
[54] Yongqiang Wang et al. Transformer-brut based acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6874-6878. IEEE, 2020.
[55] Joonsang Yu, Junki Park, Seongmin Park, Minsoo Kim, Sihwa Lee, Dong Hyun Lee, and Jung- wook Choi. NN-LUT: Neural approximation of non-linear operations for efficient Transformer inference. arXiv preprini arXiv:21 2021.
[56] Frank Zhang, Yongqiang Wang, Xiaohui Zhang, Chunxi Liu, Yatharth Saraf, and Geoffrey Zweig. Faster, simpler and more accurate hybrid asr systems using wordpieces. arXiv preprint PrXiv:2005.09155 2020.
[57] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with Trans- former encoders and RNN-T loss. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7829-7833. IEEE, 2020.
[58] Shucong Zhang, Erfan Loweimi, Peter Bell, and Steve Renals. Stochastic attention head removal: A simple and effective method for improving Transformer based ASR models. arXiv preprint arXiv:2011.04004 2020.
[59] Shucong Zhang, Erfan Loweimi, Peter Bell, and Steve Renals. On the usefulness of self- attention for automatic speech recognition with Transformers. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 89 96. IEEE, 2021.
[60] Xiaohui Zhang, Frank Zhang, Chunxi Liu, Kjell Schubert, Julian Chan, Pradyot Prakash, Jun Liu, Ching-Feng Ych, Fuchun Peng. Yatharth Saraf, et al. Benchmarking If-mmi, ctc and rnn-t criteria for streaming asr. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 46-51. IEEE, 2021.
[61] Ying Zhang, Mohammad Pezeshki, Philémon Brakel, Saizheng Zhang, Cesar Laurent Yoshua Bengio, and Aaron Courville. Towards end-to and-to-end speech recognition with deep convolutional neural networks. arXiv preprint arXiv: /701. 707.002.222 02720, 2017.
[62] Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu. Pushing the limits of sun-ssuper learning for automatic speech recognition. arXiv preprint arXiv:2010.1088 10504, 2020.
13


Reviews:
[{"review_id": "nxY2UXJXGnC", "reviewer": "Reviewer_TyAu", "summary": "The paper considers the Conformer architecture popularly used in recent speech-input tasks. Under the CTC objective, they observe the following properties and motivate solutions to give Squeezeformer:\n- Temporal redundancy of features motivates a U-Net downsample/upsample structure\n- Adjacent PreLN/PostLNs motivate converting one to scale normalization\n- Adjacent convolution/MHSAs motivate intervening FFNs\n\nas well as other changes (unified activation fns., depthwise-separable convs.). They additively demonstrate each's improvements over Conformer-CTC and compare them under fixed GFLOPs or fixed parameter counts, and then at three overall model scales (vs. small, vs. medium, and vs. large models).\n\n**Post-rebuttal: I increase my soundness and presentation scores from 3/4 to 4/4, and my score from 6/10 to 8/10.** This paper is an exemplar of what proposals of new deep-learning architectures should look like.", "questions": "- Are Squeezeformer's gains over Conformer possibly from optimizing for a different objective (from Transducer to CTC)? This limitation should be early and upfront.\n\n- What if the reduction in cosine similarities is not meaningful, e.g., outputs give H__E_Y____ --> H_HEEY_Y_Y? What if U-Net outdoes downsampling for a simpler reason (the output token resolution argument?)\n\n- Why do you think similarity increased (vs. baseline) before the downsampling?\n\n- If unifying activations, why to Swish instead of GLU? (Also the point above re: multiple activations and efficiency)\n\nSome comments re: presentation concerns above would also be appreciated.", "limitations": "The original Conformer was optimized for the bi-encoder Transducer objective and not for the uni-encoder CTC objective. For example, the Conformer architecture must also operate well on tokens (in the Transducer-specific label encoder), on which it is harder to e.g., justify downsampling. This work shows that Squeezeformer-CTC outdoes Conformer-CTC; it *does not show* Squeezeformer outdoes Conformer generally, not even in the Conformer's original setting.\n\n(That said, the uni-encoder Conformer has become more typical, so these insights likely apply to many upcoming models.)", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Thanks to the authors for their extensive work.\n\nWhile none of the individual methods are novel, their careful application atop Conformer-CTC is. The motivations are verbally clear, at least, and the CTC comparisons are fair (at fixed GFLOPs, fixed counts) as well as the ablations (e.g. vs postLN or preLN only). Re: significance, I'm convinced that Squeezeformer-CTC outperforms Conformer-CTC, and would expect this to extend to other _uni_-encoder speech applications given the margin of improvement (see Limitations). Re: clarity, the process of development and the individual methods are very clear.\n\nHowever, some changes are motivated by debatable, possibly post-hoc \"intuitions\":\n- In 3.1.1, cosine similarity in adjacent frames is touted as redundancy which is implied as bad, even in the last layer (L164-166). But redundant embeddings are good for the CTC objective, where consecutive equal predictions can express staying in the same token. \n- Are multiple activation functions really a problem for efficient inference (L190)? There is still only one type of repeating block in the Conformer. Furthermore, Squeezeformer introduces new downsampling and upsampling layers, as well as variable memory consumption.\n\nAlso, some presentation concerns:\n- I disagree with flipping the WER y-axis in Fig. 1. It is jarring as an ASR practitioner; also, it is reasonable to expect readers to intepret lower=better, visually (for some, the jarring reversed order of y-axis values may outweigh any gain in visual intuitiveness). Even worse, _the GFLOPs are superimposed atop this \"higher-is-better\" WER plot, but GFLOPs are a \"lower-is-better\" quantity!_\n- In Table 3's last two rows, it is misleading to selectively include one non-CTC baseline. The original Conformer paper includes Transformer numbers (from your [22]) that are notably stronger (2.2, 5.6, 2.6, 5.7) than what you list. In fact, the original Conformer(-Transducer)'s numbers are far better (test = (2.1, 4.3)) with only 118.8M params.\n\nMinor issues:\n- FLOPs should defined before use, especially as it may be confused with FLOPS.\n- A sentence or two about the scaling proposed in [9] in L241 would be helpful.\n- L142-150 should mention that downsampling by itself is similar to Efficient Conformer\n- L34: \"it's\" --> \"its\", \"lengths\" --> \"lengths.\"\n- L35: \"inputs as alow pointed out\"?\n- L51: \"doubles\" --> \"halves\" (to match *down*sampling; the sampling rate is lower when fixed time is covered by fewer samples)\n- L80, etc.: \"depthwise\", \"depth-wise\", \"depthWise\" all used\n- L116, etc.: \"Librispeech\", \"LibriSpeech\" both used\n- L219: \"preLN-only\" --> \"postLN-only\"\n- Table 3: \"an NVIDIA’s Tesla a100\" --> \"an NVIDIA Tesla A100\"\n- L422: \"NVDIA\" --> \"NVIDIA\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FpJmwrxLAdb", "reviewer": "Reviewer_6bur", "summary": "This paper proposes Squeezeformer, a novel hybrid attention-convolution architecture for ASR through a series of extensive architectural studies and improvements over the Conformer architecture. Note, Conformer has been the de-facto architecture for E2E speech processing tasks and Squeezeformer simplifies the shows impressive improvements on the Conformer-CTC architecture. The authors also have open sourced the code and trained model checkpoints which should be supremely helpful to the community in re-using and building on these models.", "questions": "* I enjoyed reading through the paper, I wonder if the authors studied the positional embeddings in Conformer and thought of any improvements in a positional encoding scheme for self-attention in the Squeezeformer?", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n *  The authors extensively study the architectural design of Conformer and simplifies a lot of the choices in Conformer building out a simpler, cleaner and possibly better architecture, Squeezeformer.\n* The paper introduces U-Net style temporal scaling architecture and are able to show strong results at various different model scales improving upon earlier Conformer benchmarks which have been SOTA for possibly all speech processing tasks.\n* Given that Conformer has been the de-facto architecture for speech over the last few years, the results jumps in performance by Squeezeformer are significant and the extensive set of ablations and experiments in this paper are very helpful in providing insights to the community. \n\nWeaknesses:\n* Even though the results are impressive, there is not much originality in this paper as it builds upon extensively studying the existing Conformer architecture and combines standard temporal downsampling tricks from vision. But still, the results shown are strong and the experiments are significant that would be of huge help to the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Af5A0CL5wPo", "reviewer": "Reviewer_xM6i", "summary": "The paper improves upon a Conformer model for speech recognition. The paper finds experimentally that the subsequent activations are are highly correlated in the higher confomer blocks. Therefore, it proposes to subsample in the the temporal dimension. Then, the paper proposes to remove the Macaron block. Finally it proposes to optimize the micro-architecture of the conformer: to use the Swish activation instead of GLU, use the PostLN with an extra scaling, and use a separable convolution for the first layer.\n\nThe paper describes in detail the motivation for each improvement. Then, the experiments show that adding each change one-by-one gradually improves the performance of the proposed model.\n\nFinally, the paper reports the performance of the proposed model in comparison to prior publications.", "questions": "Suggestions\n- Please fix the typos\n- Add the previously reported conformer result into the table\n- Evaluating on another dataset would strengthen the paper\n- It seems that the suggestion to remove the Macaron architecture contradicts the original study https://arxiv.org/pdf/2005.08100.pdf Table 3. Am I missing something or this paper proposes to undo this change in the original?", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe paper is very well written and is a delight to read. I was able to follow the motivations for each step of the improvement and the proposed change.\n\nThat's being said, each proposal is sound and makes sense.  The experimentation is rigorous and follows the best practices.\n\n# Weaknesses\n\nMy main criticism is that the baseline has much worse performance than it is reported in [0]. As far as I can see the main difference between the 6.8% (test-other) reported here and 4.3% in [0] is the decoder. While I understand, that the baseline source code is not available, the paper needs to be more upfront about the differences and better performance of [0]. Perhaps, it can be included in the table as \"closed source SOTA with RNN-T decoder\" or something like this.\n\nThere is a paper which predates Jasper as an end-to-end CNN architecture: [1].\n\n[0] https://arxiv.org/pdf/2005.08100.pdf\n[1] https://arxiv.org/pdf/1701.02720.pdf\n\n# Typos\n\npage 7, 219 p -> alpha\nPage 8, line 255 work -> works\npage 8, line 258 \"-\" -> \"--\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept