Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes Squeezeformer, a novel hybrid attention-convolution architecture for ASR through a series of extensive architectural studies and improvements over the Conformer architecture. Note, Conformer has been the de-facto architecture for E2E speech processing tasks and Squeezeformer simplifies the shows impressive improvements on the Conformer-CTC architecture. The authors also have open sourced the code and trained model checkpoints which should be supremely helpful to the community in re-using and building on these models.
strengths_and_weaknesses: Strengths:
 *  The authors extensively study the architectural design of Conformer and simplifies a lot of the choices in Conformer building out a simpler, cleaner and possibly better architecture, Squeezeformer.
* The paper introduces U-Net style temporal scaling architecture and are able to show strong results at various different model scales improving upon earlier Conformer benchmarks which have been SOTA for possibly all speech processing tasks.
* Given that Conformer has been the de-facto architecture for speech over the last few years, the results jumps in performance by Squeezeformer are significant and the extensive set of ablations and experiments in this paper are very helpful in providing insights to the community. 

Weaknesses:
* Even though the results are impressive, there is not much originality in this paper as it builds upon extensively studying the existing Conformer architecture and combines standard temporal downsampling tricks from vision. But still, the results shown are strong and the experiments are significant that would be of huge help to the community.
questions: * I enjoyed reading through the paper, I wonder if the authors studied the positional embeddings in Conformer and thought of any improvements in a positional encoding scheme for self-attention in the Squeezeformer?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper improves upon a Conformer model for speech recognition. The paper finds experimentally that the subsequent activations are are highly correlated in the higher confomer blocks. Therefore, it proposes to subsample in the the temporal dimension. Then, the paper proposes to remove the Macaron block. Finally it proposes to optimize the micro-architecture of the conformer: to use the Swish activation instead of GLU, use the PostLN with an extra scaling, and use a separable convolution for the first layer.

The paper describes in detail the motivation for each improvement. Then, the experiments show that adding each change one-by-one gradually improves the performance of the proposed model.

Finally, the paper reports the performance of the proposed model in comparison to prior publications.
strengths_and_weaknesses: # Strengths

The paper is very well written and is a delight to read. I was able to follow the motivations for each step of the improvement and the proposed change.

That's being said, each proposal is sound and makes sense.  The experimentation is rigorous and follows the best practices.

# Weaknesses

My main criticism is that the baseline has much worse performance than it is reported in [0]. As far as I can see the main difference between the 6.8% (test-other) reported here and 4.3% in [0] is the decoder. While I understand, that the baseline source code is not available, the paper needs to be more upfront about the differences and better performance of [0]. Perhaps, it can be included in the table as "closed source SOTA with RNN-T decoder" or something like this.

There is a paper which predates Jasper as an end-to-end CNN architecture: [1].

[0] https://arxiv.org/pdf/2005.08100.pdf
[1] https://arxiv.org/pdf/1701.02720.pdf

# Typos

page 7, 219 p -> alpha
Page 8, line 255 work -> works
page 8, line 258 "-" -> "--"
questions: Suggestions
- Please fix the typos
- Add the previously reported conformer result into the table
- Evaluating on another dataset would strengthen the paper
- It seems that the suggestion to remove the Macaron architecture contradicts the original study https://arxiv.org/pdf/2005.08100.pdf Table 3. Am I missing something or this paper proposes to undo this change in the original?
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers the Conformer architecture popularly used in recent speech-input tasks. Under the CTC objective, they observe the following properties and motivate solutions to give Squeezeformer:
- Temporal redundancy of features motivates a U-Net downsample/upsample structure
- Adjacent PreLN/PostLNs motivate converting one to scale normalization
- Adjacent convolution/MHSAs motivate intervening FFNs

as well as other changes (unified activation fns., depthwise-separable convs.). They additively demonstrate each's improvements over Conformer-CTC and compare them under fixed GFLOPs or fixed parameter counts, and then at three overall model scales (vs. small, vs. medium, and vs. large models).

**Post-rebuttal: I increase my soundness and presentation scores from 3/4 to 4/4, and my score from 6/10 to 8/10.** This paper is an exemplar of what proposals of new deep-learning architectures should look like.
strengths_and_weaknesses: Thanks to the authors for their extensive work.

While none of the individual methods are novel, their careful application atop Conformer-CTC is. The motivations are verbally clear, at least, and the CTC comparisons are fair (at fixed GFLOPs, fixed counts) as well as the ablations (e.g. vs postLN or preLN only). Re: significance, I'm convinced that Squeezeformer-CTC outperforms Conformer-CTC, and would expect this to extend to other _uni_-encoder speech applications given the margin of improvement (see Limitations). Re: clarity, the process of development and the individual methods are very clear.

However, some changes are motivated by debatable, possibly post-hoc "intuitions":
- In 3.1.1, cosine similarity in adjacent frames is touted as redundancy which is implied as bad, even in the last layer (L164-166). But redundant embeddings are good for the CTC objective, where consecutive equal predictions can express staying in the same token. 
- Are multiple activation functions really a problem for efficient inference (L190)? There is still only one type of repeating block in the Conformer. Furthermore, Squeezeformer introduces new downsampling and upsampling layers, as well as variable memory consumption.

Also, some presentation concerns:
- I disagree with flipping the WER y-axis in Fig. 1. It is jarring as an ASR practitioner; also, it is reasonable to expect readers to intepret lower=better, visually (for some, the jarring reversed order of y-axis values may outweigh any gain in visual intuitiveness). Even worse, _the GFLOPs are superimposed atop this "higher-is-better" WER plot, but GFLOPs are a "lower-is-better" quantity!_
- In Table 3's last two rows, it is misleading to selectively include one non-CTC baseline. The original Conformer paper includes Transformer numbers (from your [22]) that are notably stronger (2.2, 5.6, 2.6, 5.7) than what you list. In fact, the original Conformer(-Transducer)'s numbers are far better (test = (2.1, 4.3)) with only 118.8M params.

Minor issues:
- FLOPs should defined before use, especially as it may be confused with FLOPS.
- A sentence or two about the scaling proposed in [9] in L241 would be helpful.
- L142-150 should mention that downsampling by itself is similar to Efficient Conformer
- L34: "it's" --> "its", "lengths" --> "lengths."
- L35: "inputs as alow pointed out"?
- L51: "doubles" --> "halves" (to match *down*sampling; the sampling rate is lower when fixed time is covered by fewer samples)
- L80, etc.: "depthwise", "depth-wise", "depthWise" all used
- L116, etc.: "Librispeech", "LibriSpeech" both used
- L219: "preLN-only" --> "postLN-only"
- Table 3: "an NVIDIA’s Tesla a100" --> "an NVIDIA Tesla A100"
- L422: "NVDIA" --> "NVIDIA"
questions: - Are Squeezeformer's gains over Conformer possibly from optimizing for a different objective (from Transducer to CTC)? This limitation should be early and upfront.

- What if the reduction in cosine similarities is not meaningful, e.g., outputs give H__E_Y____ --> H_HEEY_Y_Y? What if U-Net outdoes downsampling for a simpler reason (the output token resolution argument?)

- Why do you think similarity increased (vs. baseline) before the downsampling?

- If unifying activations, why to Swish instead of GLU? (Also the point above re: multiple activations and efficiency)

Some comments re: presentation concerns above would also be appreciated.
limitations: The original Conformer was optimized for the bi-encoder Transducer objective and not for the uni-encoder CTC objective. For example, the Conformer architecture must also operate well on tokens (in the Transducer-specific label encoder), on which it is harder to e.g., justify downsampling. This work shows that Squeezeformer-CTC outdoes Conformer-CTC; it *does not show* Squeezeformer outdoes Conformer generally, not even in the Conformer's original setting.

(That said, the uni-encoder Conformer has become more typical, so these insights likely apply to many upcoming models.)
review_title: null

Meta-review: metareview: The paper conducts thorough analysis of the Conformer architecture and brings insights and techniques from other fields to simplify and improve the model structure, which is also demonstrated to show nice gains. Though as pointed by reviewers the novelty is limited, the study is very useful to the field. 
confidence: Certain
Decision: Accept