MLP-based architecture with variable length input for automatic speech recognition
Authors: Jin Sakuma
Tatsuya Komatsu
Robin Scheibler
Venue: ICLR 2022 Submitted (2022)
Keywords: MLP
automatic speech recognition

Abstract: We propose multi-layer perceptron (MLP)-based architectures suitable for variable length input.
Recently, several such architectures that do not rely on self-attention have been proposed for image classification. 
They achieve performance competitive with that of transformer-based architectures, albeit with a simpler structure and low computational cost. 
They split an image into patches and mix information by applying MLPs within and across patches alternately. 
Due to the use of MLPs, one such model can only be used for inputs of a fixed, pre-defined size.
However, many types of data are naturally variable in length, for example acoustic signals. 
We propose three approaches to extend MLP-based architectures for use with sequences of arbitrary length. 
In all of them, we start by splitting the signal into contiguous tokens of fixed size (equivalent to patches in images). 
Naturally, the number of tokens is variable. 
The two first approaches use a gating mechanism that mixes local information across tokens in a shift-invariant and length-agnostic way.
One uses a depthwise convolution to derive the gate values, while the other rely on shifting tokens.
The final approach explores non-gated mixing using a circular convolution applied in the Fourier domain.
We evaluate the proposed architectures on an automatic speech recognition task with the Librispeech and Tedlium2 corpora. Compared to Transformer, our proposed architecture reduces the WER by \SI{1.9 / 3.4}{\percent} on Librispeech test-clean/test-other set, and 1.8 / 1.6 % on Tedlium2 dev/test set, using only 75.3 % of the parameters.

Under review as a conference paper at ICLR 2022
MLP-BASED ARCHITECTURE WITH VARIABLE LENGTH INPUT FOR AUTOMATIC SPEECH RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose multi-layer perceptron (MLP)-based architectures suitable for vari- able length input. Recently, several such architectures that do not rely on self- attention have been proposed for image classification. They achieve performance competitive with that of transformer-bas based architectures, albeit with a simpler structure and low computational cost. They split an image into patches and mix information by applying MLPs within and across patches alternately. Due to the use of MI.Ps, such a model can only be used for inputs of a fixed, pre-defined size. However, many types of data naturally variable in length, for example, acoustic signals. We propose three approaches to extend MLP-based architectures for use with sequences of arbitrary length. In all of them, start by splitting the signal into contiguous tokens of fixed size (equivalent to patches images). Naturally, the number of tokens is variable. The two first approaches use a gat- ing mechanism that mixes local information across tokens in a shift-invariant and way. One uses a depthwise convolution to derive the gate values. while the other relies on shifting tokens. The final approach explores non-gated mixing using circular convolution applied in the Fourier domain. We evalu- ate the proposed architectures on an automatic speech recognition task with the Librispeech and Tedlium2 corpora. Compared to Transformer, our proposed ar- chitecture reduces the WER by 1.2/0.3 % on Librispeech lest-clean/test-othe set, and 1.6/1.8 1.6 % on Tedlium2 dev/test set, using only 86.4 % of the parameters. addition, a hybrid of proposed architecture and self self-attention module reduces the WER by 1.9/3.4 44% % on Librispeech test-clean/test other set, and 1.8/1.6 on Tedlium2 dev/test set, using only 75.3 % of the parameters.
INTRODUCTION
Self-attention, the well-known building block of Transformer (Vaswani et al. 2017), appeared in natural language processing (NLP) where caused breakthrough. Soon enough, it propagated to the fields of computer vision and automatic speech recognition (ASR). In particular, recent end-to- end ASR systems based on self-atenntion architecture, e.g., Transformer (Karita et al.) 2019a) and Conformer (Gulati al. 2020), provide state-of-the-art performance.
Recently, several architectures entirely based on MLP have been proposed in the area of computer vision. MLP-based architectures have simple structure, and achieve performance competitive with that of Transformer-based architectures, despite having fewer parameters and lower computational complexity. They split image into patches and reshape it into a (channels patches) matrix used as input. An illustration of the process, and its analog for variable length sequences, is shown in Figure T MLP-based architecture such as MLP-Mixer (Tolstikhin et al. (20TT) and gMMP (Liu et al, (021)) consist of MLP across the channel dimension and MLP across the patch dimension (also referred to as spatial dimension in computer vision tasks). ML. across the channel dimension mixes information between channels like typical feed-forward network (FFN). MLP across the patches dimension mixes information between patches, All these different works demonstrate that this process capture sufficient information and that self-attention is not always necessary. The small model size and low computational cost of the MLP- based architecture are also useful for ASR. For acoustic data, the input sequence is typically first split into contiguous transformed to some kind of frequency domain representation, e.g. a Mel-spectrogram. Then, the time and frequency dimensions analogous to patch and channel, respectively, for images. We
overlapping blocks and

---

Under review as a conference paper at ICLR 2022
fixed images
variable lingth sequeences
Charrede
potches
Figure 1: Illustration of the difference between patching for fixed-size images (left) and variable length sequences (right). We rename patches as tokens adapt to the semantics of sequences.
adopt the terminology of "token" to describe the vector of channel values at a given time. It is more apt for sequences and consistent with that used in the MLP-Mixer and gMLP works. Now, in contrast to images, sequences will produce inputs with a variable number of tokens, making it impractical to apply MLP directly to this dimension.
In this paper, we propose three approaches that can work with variable length inputs. First, the input sequences are broken down into contiguous chunks that we call tokens, as explained earlier and shown in Figure | Building on previous approaches, we propose three new token mixing units. Two of them rely on gating, where the input is split into two parts, one is transformed and then multiplied with the other. In the first kind of unit, Convolutional MLP (C-MLP), we use depthwise convolution to transform the input. This operation is locally MLP-like along the token dimension and makes the network shift invariant. The second approach, Temporal-Shitt MLP (TS- MLP), concatenates shifted parts of the input. The third approach is the non mon-gating one. In this case, apply simple depthwise convolution, thus mixing tokens with their local neighbors. However, we make use of the fast Fourier transform (FFT) to do so efficiently. This approach termed Fourier MLP (F-MLP). We applied these MI.--aaed methods to the connectionist temporal classification (CTC) (Graves et (206) ASR model and evaluated them on two datasets, Librispeech (Panayotov al. 2015) and Tedlium2 (Rousseau et al. 2014). We found different trade-off's between accuracy in terms of word error rate (WER), the number of parameters, and inference speed. However, when matching the number of parameters, the proposed MLP architectures consistently outperform self- attention based models, decreasing the WER by as much as 3 %.
Related works.
Recently, end-to-end (E2E) ASR models, where single neural network produces a transcription directly from the input acoustic features, have come to dominate benchmarks. The E2E ASR model consists of an encoder that converts audio input into latent representation and decoder that con- verts the latent representation into text output (token). To build the encoderr architecture, attention based encoder-decoder models such as Listen, Attend and Spell (LAS) (Chan et al.) 2015) and recurrent neural network transducer(RNN-TT (Graves) 2012) have shown high performance, but they suffer from slow inference speed due the way they auto-regressively output tokens. Another choice to build E2E ASR is non-autoregreesive models with the audio encoder and a decoder using CTC (Graves al. 2006). In recent years, many studies have been conducted CTC-hased mod- els, and reported results demonstrate high inference speed for a performance comparable to that of attention-based encoder-decoder models. For the architecture of the audio encoder, recurrent neural networks (RNNs) and Convolution neural networks (CNNs) have been employed (Graves & Jaitly
2014 Li al. 2019 Kriman al. 2020). More recently, models applying the sell-attention based architecture (Karita et al.) 2212201. 9a; Zhang et al. 2020: Gulati et al., 2020) have shown promising performance and become the de facto architecture. However, self-attention is expensive terms of memory and computation, and reduction of both is an important endeavor.
Next, we summarize the recently proposed ML.--aaedd architectures. MIP--Mxer (Tolstikhin al., 2021) consists of channel-mixing MLPs and loken-mixing MLPs. It alternately applies these MLPs and mixes information between channels and information between tokens. gMLP (Liu et al. 2021 also consists of MLPs across the channel and token dimensions. To more effectively mix tokens, it proposes a spatial gating unit (SGU), a module containing MLP across the token dimension and used to modulate the output of the unit. S2MMLP (Yu et al. 222 and S S2-MLPv2 Yu et al. 202 mix information across the token dimension by shift operation along the width dimension and height dimension called spati operation. GFNet (Rao al. 2021) efficiently applies leamable filters
2

---

Under review as a conference paper at ICLR 2022
in the frequency domain. It performs 2D Fourier transform of the input image, multiplies by the filters, and transforms back to the image domain. Then, MLP is applied across the channels. In the area of NLP, FNet (Lee-Thorp al. (02I) has been proposed. It applies a 2D Fourier transform to the input data, retains the real part, and follows with MI.P across channels.
Contributions. Our contributions in this work as follows. (i) We propose three MLP-based architectures suitable for sequences of arbitrary lengths. (ii) We evaluate our architecture for ASR on two different datasets. We show that the proposed C-MLP architecture outperforms the celebrated self-attention, both in accuracy and inference speed. To the best of our knowledge, this is the first time that MLP-based architecture applied to ASR.
2 CONVENTIONAL MLP-BASED ARCHITECTURES
The so-called MLP-based architectures are built using two modules, each based on simple MLPs: the channel mixing and token mixing modules. The channel mixing module typically consists of linear projections on the channel dimension for each token, similar to the point-wine feed-forward network in the Transformer. The token mixing module mixes the information in the token dimension as replacement for self-attention, and how to design it is one of the keys to the MI.P-based models. We explain conventional architectures dividing them into two types according to how they stack the modules: MLP-mixe and gMLP types, corresponding to the architectures proposed in (Tolstikhin) ct al., 2021) and (Liu et al., 2021). respectively. We start by describing the outer construction, i.e., how the channel and token mixing modules combined. Then, focus on the inners of a few kinds of token mixing module. Throughout, we omit layer normalization, residual connections, and bias of linear projections for the sake of brevity. Figure shows diagrams of all the parts.
2.1 OUTER CONSTRUCTION
MLP-Mixer type. Figure 2 (a) shows the MLP-Mixer type which first applies the token-mixing module and then applies the channel-mixing module. It is adopted by its namesake (Tolstikhin et al. 2021), and GFNet (Rao et al. 2021). Let Xin E DxN denote the input data matrix with D channels and N tokens. It first goes through the token mixing module,
X = TokenMixinggMocuue(X...
E D
(1)
where Token LokknMixxngee MoingeMixinMMduueee is MLP-based module that usually consists of multiple linear projections on the token dimension. How to design this module is one of the key considerations for MLP-based models. Conventional approaches are introduced in Section 2.2
Next, the output of the token-mixing module, X, is fed to the channel-mixing module. The channel- mixing module is the same the point-wise feed forward network in Transformer, which performs two linear projection across the channel dimension,
Xout = W20 E DxN
where Wa E D'DD and W, Dx D' are the weights of the linear projections. We remark that the construction of the MLP-Mixer type is similar to that of a Transformer, with the token-mixing replacing self-attention. This highlights the critical importance of token mixing.
gMLP type. Figure (b) shows the gMLP type where the two layers of the -hannel-mixing module are split and the token-mixing module placed in-between. This is the structure adopted by gMLP, S2-MLP and S2-MLPv2. Let Xin E DxN denote the input data with D channels and N tokens. First, linear projection on channel dimension to D' is applied to Xin-
X = a (W, WWWKy Xin) E D'N
(3)
where W, D'DD refers to weights of linear projection. Next, the token-mixing module follows,
X = TooeenniixigggMMouulleee// E DN
(4)
Finally, linear projection across the channel dimension D" to D is applied as
X.nt W,X' E DRN
(5)
where W, DxD is the weight matrix of the linear projection.
Here, additional linear projections can be utilized for mixing the channel information. For example, S2-MLP stacks two additional linear projections after the token-mixing module.
3

---

Under review as a conference paper at ICLR 2022
LeWe
Adtration
Line
Iul Iunsaoe
sumnmbiin modise
Adtvalion
Adhiction
Liner
bosenwnidigg rodkle
Uniet
(e] takten miding M.P MLP Mbbe
(4) Spatial Gating Unit gMLP
(4) MLP-Mixer type
[b) gMLP type
Figure 2: Overview of (a) MLP-Mixer type, (b) gMLP type, (c) token-mixing MLP in MLP-Mixer, and (d) Spatial Gating Unit gMLP.
2.2 CONVENTIONAL TOKEN-MIXING MODULES AND CHALLENGES FOR ASR
There have been several studies on how to design the token-mixing module, such as token-mixing MLP in MLP-mixer, spatial gating unit (SGU) in gMLP, Fourier Domain Mixing in GFNet (Rao et al., 2021) and FNet Thorp et al. 2021). J. In this paper, we introduce these modules and discuss the issues in applying them variable sequence length input data. In the following, the input is always assumed to be a data matrix with D channels and N tokens, X E DDN
Token- MLP. The token-mixing MLP first transposes the input X and performs linear pro- jection across the token dimension to mix the token information,
X' = (W2o(W,X E PXN,
(6)
where W1 N'NN and W2 Nx'' refer weights of linear projections, and a is an element-wise activation function. Albeit the components are the same as the channel-mixing module of Equation 2 the linear projections are done on the token dimension by transposing the input. This module is shown in Figure
Spatial Gating Unit. The structure of the SGU is illustrated in Figure 2 (d). The SGU first splits the input X into X, R xN and Xg & In gMLP, X, is set as the first half of X and Xg for the second half of X. Then, performs gating operation. The output of SGU X' is,
X' X, HsGU E R with HSGU = (W2X!) E R
(7)
where denotes element-wise multiplication, and Wa 2 NxN refers to the weights of the linear projection. This gating operation mixes the information of the token effectively.
The effectiveness of the SGU has been experimentally shown with image and language experiments compared to non-gating linear projection and other variants.
Fourier Domain Mixing. GFNet and FNet the discrete Fourier transform (DFT) in the token- mixing module. GFNet applies a fixed length learnable filter by direct multiplication in the fre- quency domain. FNet mixes tokens by the application single forward 2D DFT, retaining only the real part of the output.
2.3 CHALLENGES FOR ASR.
The sizes of the weight matrices in Equation and 7 are fixed and have match the number of tokens in the input. Similarly, the learnable filter in GFNet is of fixed size equal to the length of the input sequence. These approaches are thus not suitable for variable length input. Nevertheless, given a fixed dataset, they could be applied by zero-padding all sequences to the size of the longest one. However, there are number of shortcomings. First and foremost, longer sequences that might be encountered at test time cannot be accommodated. Then, MLPs will introduce large number of parameters. Moreover, they are not shift invariant, an important property for sequences. While GF is shift invariant and only requires reasonable number of parameters, it still cannot deal with sequences longer than the maximum size set. FNet has neither the above limitations, however, find its performance somewhat lacking in the experiments of Section

---

Under review as a conference paper at ICLR 2022
fuon veter cory
dip Site
Conuuuanni Goting Ura
Canouaional Daiin
65 Thipoon Oean Guting lnd
The
Figure 3: Overview of (a) Convolutional Gating Unit, (b) Convolutional Gating Unit', (c) Temporal- Shift Gating Unit, and (d) Temporal-She operation.
3 MLP-BASED ARCHITECTURE FOR VARIABLE LENGTH INPUT
We describe here our three proposed MLP-based architectures for ASR. These approaches are suit-
able for sequences of any length and shift invariant. As described in Section 2.2 the design of the token-mixing module is important to apply MLP-based architecture to ASR. We first propose three token-mixing modules for ASR in 3.1 Then, we describe the overall architecture in 3.2
3.1
TOKEN-MIXING MODULE FOR VARIABLE LENGTH INPUT
3.1.1
CONVOLUTIONAL GATING UNIT (CGU)
Convolutional Gating Unit (CGU). The conventional token-mixing modules described in Sec- the full extent of the data. However, for long sequences, think it is sufficient to mix temporally local information. We do so by using a convolutional layer along the time dimension. We call this token-mixing module Convolutional Gating Unit (CGU). The structure of a CGU is shown in Figure 3 (a). CGU replaces linear projection across the token dimension in SGU (see Equation with depthwise convolution. Its output HCGU is,
tion 2.2 mix information globally
HCGU K &
where X E xNN K K is the D -immensiional kernel with kernel size k. The depth wise convolution operation denoted by and defined as,
b (K * X). = Σ K X.
j=1
Convolutional Gating Unit' (CGU'), We also propose variation where linear projection across the channel dimension is applied to the filter HeGU. In this case, information from both token and channel dimensions is mixed. The new filter H'GU is formulated as follows,
H'GGU WHoGU E 2
(10)
where W E X is the weight matrix of the linear projection. This structure is illustrated in Figure (b). We refer to this token-mixxe miking module as Convolutional Gating Unit' (CGU")
3.1.2 TEMPORAL-SHIRE GATING UNIT (TSGU)
In CGU, we used learnable kernels for convolution across the token dimension. Next, we consider the case of performing convolution using kernels with fixed parameters. If the input is a D N matrix, then the ith row of the fixed kernel Kahift is
[0,0,000,, if if D if D.
k;
T
(1I)
[1,0,0,0,0
This corresponds to the shift operation proposed in S Yu et al. 2021a). Half of the channels shifted by two forward, and the other half by two backward along the time dimension. The time shift action of this kernel on the input signal is illustrated in Figure 3 (d). The size of the shift is a parameter but fix it here the value taken in the experiments. We propose the Temporal Shift
5

---

Under review as a conference paper at ICLR 2022
Gating Unit (TSGU) as a token- token-mixing module with shift operation in the token direction. The inputs is split as in SGU, and the gating values derived as
HTSGU
shin X g:
(12)
which replaces HSGU. This gating mechanism is parameter-free and mixes time information by applying progressive time shifts to the input signal.
3.1.3 FOURIER FILTER UNIT (FFU)
In both CGU and TSGU, we use a gating unit based on the SGU, as in gML... To investigate the use- fulness of gating, introduce third architecture that does not use it. This is circular depthwise convolution constructed as follows. It can be understood as similar to CGU, but without the split, and where all the input is processed through the depthwise convolution. The fixed size filters are defined in the time domain, but applied in the frequency domain. Thus, the filter is easily applied to sequences of any length by applying an FFT padded to the length of the input sequence the filters, and working in the frequency domain. We call this token mixing module Fourier Filter Unit (FFU). The output of the circular depthwise convolution is computed as follows,
Z F-
X
F[K]] E R
(13)
where X E RDxN is the input signal and K E R Dxn contains the filters in its rows. The FFT and its inverse are denoted F[-] and [-),,), respectively. We assume that the operation zero-pads to size D x N to match X.
GFNet (Rao et al., 2021) also applies element-wise multiplication learnable filters and input features in the frequency domain. To apply GFNet to variable sequence length input, we have to find the maximum sequence length of the dataset and define filters to match that length. On the other hand, FFU defines filters in the time domain, zero-pads it to match the size of each input data, and perform FFT to the filter and input data. Therefore, FFU has fewer parameters than GFNet.
3.2 OVERALL ARCHITECTURE
We have proposed three types of token-mixing modules, but there is choice in the outer con- struction, described in Section 2.1 to build the whole architecture. We have conducted preliminary experiments (See Appendix A and finally propose the following four MLP-bbsed architectures.
Convolutional MLP (C-MLP): CGU + gMLP type outer construction
Convolutional MLP' (C-MLP'): CGU' + gMLP type outer construction
Temporal-Shift MIP (TS-MLP): TSGU + gMLP type outer construction
Fourier MLP (F-MLP): FFU + MLP-Mixer type outer construction
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We applied our architectures non-autoregressive CTC-based ASR. The experiments were con- ducted using the ESPNet tool kit (Watanabe et al. 2018). We use Transformer-eccoder as self- attention based baseline, which is a strong baseline used many prior works (Bai al., 2020 Higuchi et al. 2021 Lee & Watanabe 2021).
We also use FNet (Lee-Thorp al. 202T) and GFNet (Rao et al., 2021) as MI.P-based baseline. In order to justify the performance of the architecture itself, all parameters other than the architecture were kept the same in all experiments.
Encoder layer structure. For MLP-Mixer type architectures (FNet, GFNet, and F-MLP), input and output dimensions the token-mixing module are set to 256, input dimension, hidden dimension, and output dimension of the channel-mixing module set to 256, 1024, 256. For gMLP type architectures (C-MLP, C-MLP', and TS-MLP), input and output dimensions of the first channel- mixing module are set to 256 and 1024, input and output dimensions of the token-mixing module

---

Under review as a conference paper at ICLR 2022
Table I: Experimental results on Librispeech and Tedlium2.
WER (I.ibisppech)) test-clean
WER (Tedliumz)
metthod
test-other
dex
test
Raseline
Transfomer
FNet Lee Themp Cal. 2021
11.3M
33.8%
63.3%
5%
28.3%
GFNeT 1C30 2
16.2M
12.5%
31.45%
13.65%
12.7%
Own
F-MPE
11.55
16.05%
37.00%
16.34
15.8%
F-MLPtiay
13:80
12.11%
30.81%
13.25%
13.0%%
C-MPP
9.3M
133,4%
33.85%
C-MLP4nay art
12:2M
11.4%%
250.8%
12.64%
12.0%%
C-MLP
14.00
12.11%
31.91%
12.85%
12.01%
-tiny attn
11.1%
28.35%
11.55
11,44%
IS. MLLE'
9.1M
16.0%%
40.00%
18.11%
$7.55%
TS-MI Patiny ams
12.1M
13.55
32,69%
14.1%
13.55
are set to 512 and 256, input dimension, hidden dimension, and output dimension of the second channel-mixing module are set to 256, 1024, 256. For both types, we use Gaussian Error Linear Units (GELU) (Hendrycks & Gimpel 2016) as activation function. Convolution kernel size of C- ML.P and C-MLP' and filter size of F-MLP are 15 and shift size of TS-MLP encoder, follow the architecture proposed in Karita et al. 2019b). The number of heads and input dimensions of the self-attention module are 4 and 256. The intermediate dimensions of the feed-forward network are 1024. For all models, we use two CNN-based subsampling layers before encoder layers. We set the subsampling rate to 0.25. All models consist of 18-layers.
For Transformer
Tiny self-attention module. gMLP (Lu et al. 2021) demonstrate that adding a small self-attention module to the SGU improve performance the cost of a modest increase in resource usage. The structure of an SGU with a tiny self-attention mod- ule is shown in Figure 4 The input of the tiny self- attention module the input of the encoder. The out- put of the tiny self- module is added to the end of the right path of SGU. The tiny self-attention module has the same structure as the self-attention module Transformer encoder, but its hidden di- mension of linear projection d and the number of attention heads Thead are small. We also experimented with the proposed token mixing module 1, d 128, while we set
Linoifguee
Altentinn
Figure 4: SGU with a tiny self-attention.
combined with tiny attention. We set the tiny self-attention Nhead self-attention module in Transformer- based model to Theaa = 4, d 256.
Datasets. We measure performance two datasets: Librispeech and Tedlium2. These two datasets are widely used for the evaluation of non-autoregresssie ASR Tedlium2 contains utterances from English Ted Talks, and we used the 207-hour training data. We used the standard validation and test sets for tuning hyper-parameters and evaluating performance, respectively. For Librispeech, we used the 100-hour subset for training set and standard validation and test sets as (Park et al. 2020) Zhao et al.) 2020 Higuchi et al. 2021). Specifically, the validation and test sets of Librispeech are divided into "clean" and "other" based on the quality of the recorded utterances.
Feature Extraction. We use 80-dimensional log- mel--sectroorrmm features and 3-dimensional pitch features as inputs. For feature extraction, we use kaldi (Povey al., (01I). and shift size and length of the window are set to 10 ms and 25 ms, respectively. We apply SpecAugment (Park et al.) 2019) and speed perturbations (Ko et al., 2015) for data augmentation. For Librispeech, we tokenize text into 300 subwords, and for Tedlium2, we tokenize text into 500 subwords. We created subwords with SentencePiece (Kudo Richardson 2018))
Training and inference setup. All models are trained for 50 epochs. We set the batch size to 64. The optimizer is Adam with 3 = 0.9. _,2 0.98, <= 10 The scheduling method for the learning rate is the same (Vaswani et al. 2017). learning rate d 0.5 min(step_num, step_num warmup steps -1.3), -1.3), where we set warmup steps and d to 25000 and 1280, respectively. Dropout rate and label smoothing rate set to 0.1. For inference, we use the model parameters obtained
7

---

Under review as a conference paper at ICLR 2022
Table 2: Comparison of model size and WER. In order stabilize the learning of deep models, all models were trained using InterCTC (Lee & Watanabe 2021). regularization technique used in CTC-based ASR.
(Librispeecl)
(Tedlium2)
method
layers
parums
test-cleun
lest-uther
dev
test
Transfoemes model
9.1M
14.1%
15.2%
F-MPP
15
9.9M
15.5%
57. 17.3%
17.38
16.4%
C-MLP
18
9.5M
11.5%
$1.2%
13.69%
12.7%
12
10.0M
11.4%
13.5%
MLP
18
9.IM
14.5%
38.0%%
17.6%
16.8%
F F-MLF+iny aun
12
9.9M
11.9%
30.3%
13.7%
12.8%
C-MPPriny
12
8.8M
12.9%
4tay attn
94M
11.698
13.2%
12.8%
TT-MLPP P-tiny alln
12
8.7M
13.3%
32 32.9%
14.5%
14.0%
Transformer based model
18
16.2M
10.9%
27.8%
11.5%
11.1%
F-MPP
27
16.3M
13.9%
34.6%%
15.7%
14.8%
C-MLP
36
10.1%
16.0M
10.195
11,4%
10.5%
TS-MLP
36
16.3M
11.256
$2.7%%
13.54
12.79
F- P+inyy all
21
15.8M
10.4%
27 77.0%%
11.9%
11.4%
27
17.3M
10.4%
C-MLP' Iayy atto
18
16.9M
9.4%5
24.9%
10.5%
10.24
TS Mityy atin
27
16.3M
9.9%
26.216
11,0%5
10.3%
TransBemesbeedd model
56
30.4M
9.998
25.8%
10.7%
9.7%
F-MPP
51
29.0M
13.1%
32.5%%
16.6%
15.4%
C-MEP
72
C-MLP'
42
30.0M
9.146
26.09%
9.5%
8.7%
TS-MEP
72
30.5M
10.0%
30.696
11,698
10.89
F-MLP+iny
42
29.8M
24.1%
9.8%
Patiny am
9.3%6
8.5%
C-ML P' tiny stin
33
29.4
8.1%
21 191899
10.25
9.3%
TS-MI Patiny atmn
51
30.6M
8.8%
23.5%
9.8%
9.1%
by averaging the 10 models with the best validation scores. The outputs are decoded by greedy decoding for CTC, without using any external language model.
4.2 RESULTS
Main results. Table provides a comparison of the parameter sizes and word error rates (WER) on Librispeech and Tedlium2. In Table we see that F-MLP is 71.0 % the size of GFNet, which also uses Fourier transform, and the Transformer-based model. Compared to the Transformer-based model, F-MLP degrades WER by 2.7 /4.8 % on Librispeech fest-clean/test-othe sets and 1.9/2.2 % on Tedlium2 dev/test sets. However, F-MLP with a tiny self-attention improves WER by 3.9/6.2 % on Librispeech test-clean/test-othh outperforms Transformer based model and GFNet. In addition, even when combined with tiny self- attention, its model size is smaller than that of Transformer-basee model and GFNet.
set and 3.1/2.8 % on Tedlium2 dev/test set from F-MLP and
C-MLP achieves competitive performance with Transformer-base model with only 57.4 % of its parameters. C-MLP' achieves the best WER in Table 1 and improves WER by 1.2/0.3 % on Lib- rispeech cest-clean/test-other set, and 1.6/1.6 6 % on Tedlium2 dev/test set. It increases the model size little but is still only 86.4 % the size of the Transformer-basee model. C-MLP with a tiny attention improves WER by 1.9/3.4 % on Librispeech lest-clean/test-othe set, and 1.8/1. 1.6 % on Tedlium2 dev/test set, while using only 75.3 % of the parameters. TS-MLP, which can be said to be a special case of C-MLP, has the smallest number of parameters. TS-MLP has only 56.2 % of parameters of Transformer-basee
CTC while degrading WER by 2.7/7.8 % on Librispeech rest-clean/test-thhee set and 3.7/3.9 % on Tedlium2 dev/test set.
Model Scaling Analysis. Table 2 shows the performance when the models are scaled to have ap- proximately the same number of parameters. We evaluate three model sizes corresponding to the number of parameters for Transformer-bast
model with 9, 18, and 36 layers. Under most condi- tions, C-MLP shows the highest accuracy. When the number of layers is scaled up and deepened, C-MLP and TS-MLP show large performance improvement compared to the other models. This may be because the number of layers of C-MLP and TS-MLP is larger than other models due to their smaller size. We conjecture that the larger number of layers allows better mixing between a wider range of locations, leading to improved representations.
8

---

Under review as a conference paper at ICLR 2022
barboma F-MPP
burabored
3.0
1.0
T-ALP-ath
etth
2.5
TS-MP
T5-NL7
2.0
2.0
1.5
Inference
2.5
2.0
0.5
055
00
13
30
60
30
30
50
60
88
Sequence length [4]
Sequence length [4]
Figure 5: Comparison of inference speed for different sequence lengths
Table Computational cost of architectures. N is the sequence length and D the size of channel dimension of a feature. I is the size of filters in Fourier Filter Unit. à is the convolution kernel size.
Method
compulational complexity
number of parameters
projection
ND² 4ND + 2N
D2
self-attention
4/7
Fourier
log, N ND KN N + ND
ID
Convolutional Gating Unit Temporal Shift Gating Unit tiny sclf-attention
kD
2ND + 2N
2D2
Computational Cost Analysis. We investigate inference speed with randomly created 83- dimensional inputs with sequence lengths of 512, 1024, 2048, 4096, 8192 frames. A sequence of length 512 corresponds to 5.12s of audio. We use 18-layer encoder for every method. We conduct our experiments with a batch size 1 on a single Intel Xeon@ Gold 6230 CPU @ 2.10GHz with 4 cores. The reported times are the average of ten measurements. We show the inference speed for each input sequence length in Figure 5 and show computational characteristics of each architecture in Table 3 N denotes the input sequence length and D) the number of channels. From Table 3] we see that when sequence length N is small, the influence of the size of D on operation complexity is strong. As shown in Figure 5] the inference speed of C-MLP and C-MLP' is slower than Transformer-bast operation complexity small as shown in Table So, Figure we see that the inference speed of Transformer- bbasee model is slower than any of our architectures when N is large. TS-MIP, which can be said to special case C-MLP, only performs a simple shift operation and glement-wist multiplication. The TSGU has parameters. Therefore, it achieves the fastest inference speed in Figure 5) regardless of the input sequence length. When combined with tiny self-attention, the inference time increases, but the trend remains.
model when N is small. On the other hand, the influence of the size of D on
CONCLUSION
We proposed three new network architectures based MLP-mixing for sequences of variable size. Each uses different mechanism to mix information at long-range across the sequence. C-MLP relies on convolutional gating, TS-MLP on time-shift gating, and F-MLP simply on circular con- volution without gating. Extensive experiments revealed that these MLP architectures are sufficient to outperform Transformer-based models for aoneeetoreeeessvvv ASR. Among the different models, C-MLP was the best, suggesting that gating is useful and necessary to squeeze out the best perfor- mance. When the proposed models alone were not sufficient, adding a tiny self-attention layer was enough to bridge the gap, while keeping the parameter count low. However, this was not always necessary when matching the number of parameters of the different networks. We thus conclude that all three proposed MLP-like architectures not only suitable for ASR, but also highly practi- cal due to their simplicity and good performance. In the future, we will explore their application to other tasks such as natural language processing acoustic event detection.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ye Bai, Jiangyan Yi, Jianhua Tao, Zhengkun Tian, Zhengqi Wen, and Shuai Zhang. Listen at- tentively, and spell once: Whole sentence generation via a non-autoregressive architecture for low-latency speech recognition. In INTERSPEECH, pp. 3381-3385, 2020.
William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint FrXiv:/508..////1 2015.
Alex Graves. Sequence transduction with recurrent neural networks.
arXiv preprint
2012.
Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In Proceedings of International Conference on Machine Learning (ICML), pp. 1764 1772. PMLR, 2014.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of International Conference on Machine Learning (ICML), pp. 369-376. PMLR, 2006.
Anmol Gulati, James Qin, Chung-Chena Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmeett transformer for speech recognition. In Proceedings of Interspeech 2020, pp. 5036-5040, 2020.
Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 2016.
Yosuke Higuchi, Nanxin Chen, Yuya Fujita, Hirofumi Inaguma, Tatsuya Komatsu, Jaesong Lee, Jumon Nozaki, Tianzi Wang, and Shinji Watanabe. A comparative study -ooooooorerressvvee modelings for speech-to-text generation. ArXiv, abs/21 10.05249, 2021.
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, al. A com- parative study on transformer vs rnn in speech applications. In Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 449-456. IEEE, 2019a.
Shigeki Karita, Nelson Enrique Yalta Soplin, Shinji Watanabe, Mare Delcroix, Atsunori Ogawa, and Tomohiro Nakatani. Improving transformer- mandfrderr-baeedd end-to-end speech recognition with con- nectionist temporal classification and language model integration. In Proceedings of Interspeech 2019, pp. 1408-1412, 2019b.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. Audio augmentation for speech recognition. In Proceedings of Interspeech 2015, 2015.
Samuel Kriman, Stanislav Beliaev, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, and Yang Zhang. Quartznet: Deep automatic speech recog- nition with 1d time-channel separable convolutions. Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7829-7833. IEEE, 2020.
Taku Kudo and John Richardson. Sentencepicce: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 66 71, 2018
Jaesong Lee and Shinji Watanabe. Intermediate loss regularization for CTC-based speech recog- nition. In Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105 03824,, 2021.
Jason Li. Vital Lavrukhin, Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M. Cohen, Huyen Nguyen, and Ravi Teja Gadde. Jasper: An end-to-end convolutional neural acoustic model. arXiv preprint arXiv: / 190400004.30 003888, 2019.
10

---

Under review as a conference paper at ICLR 2022
Hanxiao Liu, Zihang Dai, David R So, and Quoe V Le. Pay attention to mips. arXiv preprint arXiv:2105.08055 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ast corpus based public domain audio books. In Proceedings International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015.
Daniel S Park, William Chan, Yu Zhang, Chung-Chems Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Proceedings of Interspeech 2019, pp. 2613-2617, 2019.
Daniel S. Park, Yu Zhang. Ye Jia, Wei Han, Chung- Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V. Le. Improved noisy student training for automatic speech recognition. Proceedings of Inter- speech 2019, pp. 2817-2821, 2020.
Daniel Povey, Arnab Kumar Ghoshal, Gilles Boulianne, Lukás Burget, Ondrej Glembek, Nagen- dra Kumar Goel, Mirko Hannemann, Petr Motlickk, Yanmin Qian, Petr Schwarz, Jan Silovský, Georg Stemmer, and Karel Vesely. The kaldi speech recognition toolkit. In 2011 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2011.
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. arXiv preprint Siii:2107.0064555 2021.
Anthony Rousseau, Paul Deléglise, Yannick Esteve, et al. Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks. In Proceedings of the Ninth Interna- tional Conference on Language Resources and Evaluation (LREC), pp. 3935-3939, 2014.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mip-mixer: all-mip architecture for vision. arXiv preprint arXiv:2 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems (NeurIPS), pp. 5998-6008, 2017.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson-Enriee Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. Espnet: End-to-end speech processing toolkit. Proceedings of Interspeech 2018, pp. 2207-2211, 2018.
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlp: Spatial-shift mlp architecture for vision. arXiv preprint arXiv:2106. 00470070.7777.. 2021a.
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlpv2: Improved spatial-shift mlp architecture for vision. arXiv preprint arXiv:2106.07444 2021b.
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer en- coders and mn-t loss. In Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7829-7833. IEEE, 2020.
Y. Zhao, Chongjia Ni, C. C. Leung, Shafiq R. Joty, Chng Eng Siong, and Bin Ma. Cross attention with monotonic alignment for speech transformer. In INTERSPEECH, 2020.
11

---

Under review as a conference paper at ICLR 2022
A APPENDIX A: ARCHITECTURE TYPE EXPERIMENT
Our proposed token-mixing modules can be applied to either of two types of architectures described in 2.1 MLP-Mixer type and gMLP type. We conducted preliminary experiments to decide which type of architecture to adopt for each of the proposed token-mixing modules. We show the results of the preliminary experiments in Table
Table 4: Comparison of architecture type and WER.
WER (Librispeech)
architecture type
params
test-clean
test-other
FFU MI.--Mxxrr type (FMI.P)
11.5M
16.0%
37.0%
FFU gMLP type
9.2M
18.1%
41.1%
FFU MLP-Mixer type (FMLP) tiny attn
13.8M
12.1%
30.8%
FFU gMLP type
12.2M
13.2%
32.1%
CGU MLP-Mixer type
11.5M
13.6%
34.6%
CGU gMLP lype (C-MLP)
9.3M
13.4%
33.8%
CGU + MLP-Mixer type + tiny attn
13.9M
11.8%
29.4%
CGU + gMLP type (C-MLP) + tiny attn
12.2M
11.4%
28.8%
CGU' + MLP-Mixer type
12.7M
13.6%
34.4%
CGU' + gMLP type (C-MLP')
14.0M
12.1%
31.9%
CGU' MLP-Mixer type + tiny atin
15.1M
11.9%
29.9%
CGU' gMLP type (C-MLP') tiny
16.9M
11.1%
28.3%
TSGU MLP-Mixer type
11.5M
18.7%
43.2%
TSGU + gMLP type (TS-MLP)
9.1M
16.0%
40.0%
TSGU + MI.-MMxer type tiny attn
13.8M
13.6%
33.3%
TSGU + gMLP type (TS-MLP) + tiny atmn
12.1M
13.5%
32.6%
B APPENDIX B: HYPERPARAMETERS FOR OUR EXPERIMENTS
We summarize the hyperparameters described in section in Table and Table
Table 5: Hyperparameters
hyperparameter
value
Epoch batch size
50
64
dropout rate
0.1
label smoothing rate
0.1
subsampling rate
0.25
Optimizer
Adam (3: = 0.9, 81 = 0.98, 10-9)
learning rate
0-5.5
(step num, num warmup steps 1.5)
warmup steps
25000
1280
window length (Feature Extraction) window shift (Feature Extraction)
25ms
10ms
12

---

Under review as a conference paper at ICLR 2022
Table 6: input and output channel dimensions of the architectures
module
parameters
self-attention
num head = 4, hiddendim 256
Transformer encoder
Linear in FFN
input 256, output 1024
Lincar in FFN
input 1024, output = 256
Activation
GELU
loken-mixing module
input 256, output 256
Linear (1)
input 256, output 1024
chanmel
MLP-Mixer type (F-MLP, GFNet, FNet)
Linear channel
input 1024, output 256 GELU 15 mam_head 1, hiddendim 128
Activation
filter size (F-MLP)
tiny self-altention
filter size (GFNet)
512
Linear (1)
changel
input 256, output 1024
token-mixing module
input 1024, output 512
gMLP type (C-MLP, C-MLP', TS-MLP)
Linear(2)
channel
(2)
input 512, output 256 GELU 15
Activation
convolution kernel size (C-MLP, C-MLP')
shift size (TS-MLP)
2
tiny self-attention
nem_head = 1, hiddendim 128
13


Reviews:
[{"review_id": "vJxbrT8HuRO", "reviewer": "Reviewer_uaQ8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Please revise the paper in terms of writing. Equation 1 seems to have typo. In section 2, please correct the repetition \"the long range dependencies\"\nIt is mistakenly referred to Table 1 instead of Table 2.\n- how do we split the dimension in SGU? Randomly?\n- you may want to elaborate more on the term \"tiny attention\"\n- have you run the conventional CTC model? Would be good to see that result as baseline too.", "summary_of_the_paper": "The paper proposes three mlp based structures to deal with sequences of variable size. The experiments are conducted on two datasets and campare to transformer based models some improvements are reported.", "summary_of_the_review": "While it seems the paper presents technically interesting topics, it requires major revision in terms of writing. It includes many grammatical errors, wrong references and punctuation are not respected. Unfortunately, it makes the paper difficult to follow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uBdegwlcUhD", "reviewer": "Reviewer_vrtS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - The approach is novel and significant. The MLP approach for image classification is very promising and it's a important step to make it work on sequential data and speech-related tasks.\n- The evaluation is thorough, clearly showing the capability of the proposed approaches. The model scaling analysis is a very nice addition and clear showcases the potiental of the MLP architectures.\n\nWeaknesses:\n- The clarity of the paper is quite bad. Section 2 especially is hard to read and hard to understand (see detailed comments below). Section 3 is also not very clear. They also are a lot ot typos. The author should improve the writing, as these sections are critical to understand the paper. The experimental setup section is also missing important details about data split, training details and model hyper-parameters.\n- I am not convinced about the use of the terms \"channel\" and \"token\" for the two dimensions of the input. These keywords are use mainly in CV for the first one and in NLP for the second one. For speech, these two dimensions already have a name: the temporal dimension and the spectral (or cepstral depending on the features) dimension. Moreover, \"token\" can be misleading as it usually refers to words, but in this case the input is an audio signal segmented by duration, so there are no word boundaries involved. I thus encourage the authors to revisit the naming scheme.\n\nDetailed comments:\n- Section 2:\n    - \"The long range dependencies The latter allows to model ...\" -> \"The latter allows to model ...\"\n    - Figure 2: \"Linear_toekn\" -> \"Linear_token\"\n    - Equation (1) is using \"Linear_channel()\" but the text mentioned \"Linear_token()\", which one is correct?\n    - SGU is used in Section 2.1 but only explained in Section 2.2. It's confusing. \n- Section 4.4:\n    - Table 1 should be Table 2\n- The term \"sandwiched\" is used several time, it's not wrong but there are better ways to write that, like \"in between\" for instance.\n- I am a bit confused about the FNet and GFNet baselines in Table 1: according to Section 2 and 3, these two approaches are not suitable for variable length input (which is the key motivation of the paper), so how did the author make them work for ASR? please clarify. ", "summary_of_the_paper": "This paper presents an approach for Automatic Speech Recognition based on MLP architectures. Those architectures were recently proposed for image classification and yielded promising results. In this paper, three new MLP-based architectures able to handle variable length sequences are proposed. The approaches are then evaluated on the Librispeech and Tedlium corpora and are compared to baselines from the literature. The proposed approaches are shown to yield better peformance than the baseline while keeping a low complexity. ", "summary_of_the_review": "The proposed approaches are novel and significant, and clearly evaluated. They are shown to be a promising new approach for ASR. However the clarity and quality of writing of the paper are quite below the standards.  Hence in the current shape i cannot recommend acceptance, it's just below the threshold. I will be happy to increase my score if the writing is improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nqvJjqYtH-c", "reviewer": "Reviewer_4cme", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(1)\tThe novelty of the paper is somewhat weak. C-MLP, which exploits a depth-wise convolution for time-axis feature aggregation, is a commonly used approach in ASR. (Convolution module in Conformer, or Time-Depth Separable Convolution) TS-MLP seems to bring an idea from S^2-MLP. An additional tiny-attention module is also taken from gMLP.\n\n(2)\tFor the baseline performance, is there any reason for using only a clean-100h subset of LibriSpeech? I’m not sure the performance is in a sufficient range; it would be clear if you could provide some references that also trained the Transformer only on clean-100h.\n\n(3)\tNot enough experimental/architectural details are provided. For example, the embedding dimension, the number of heads in Transformer, convolution kernel size, frontend subsampling stride, type of activation function, STFT window size, optimizer, learning rate, etc., are missing.\n", "summary_of_the_paper": "This paper proposes three architectural modifications on recently introduced MLP-based neural networks, such as MLP-Mixer or gMLP. These networks could not handle the variable-length input because the token-level mixing was performed by MLP. The proposed three modifications are: C-MLP (w/ depth-wise convolution), TS-MLP (w/ shift operator), and F-MLP (w/ Fourier transform and w/o gating). The authors evaluated their model on two ASR datasets, LibriSpeech-100h and TedLium-2, and achieved better WER compared to Transformer-based models with a similar number of parameters.", "summary_of_the_review": "Although the problem that this paper tries to solve is an important problem, the novelty of the proposed solutions seems to be weak. Experiments could be more powerful with commonly used benchmarks (LibriSpeech-960h), and details are missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lwbl61AUjXd", "reviewer": "Reviewer_h5Ds", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper contains many typos and ungrammatical sentences which make it difficult to read. Therefore, even if the experiments seem rigorous,  the lack of rigor in the writing casts a doubt on the results. The reference to previous or related work is scarce and the results of a standard ASR methods that can deal with variable input length should be reported for comparison.\nThe experiments are not easily reproductible due to the lack of details and errors in the description of proposed models. \n\nThe quality of the paper is not sufficient for publication in ICLR.\n", "summary_of_the_paper": "This paper deals with the problem of having variable length inputs in deep neural network architecture using only MLPs. Recently, there has been interest in these architectures because they could reduce the number of parameters in deep NN models. However, these architectures have so far only been applied to images with a fixed size input. In order to apply them to sequences such as speech recognition or NLP, these models must be able to deal with variable input size. This paper presents 3 different solutions to allow MLB based deep NN to cope with variable input. The proposed solutions are based on either a modification of the gating mechanism or replace the gating mechanism with FFT. The proposed models are tested for a speech recognition task on 2 standard databases. Compared to transformer-based models, the proposed models obtain comparable performance with less parameters, and better performance with similar number of parameters.", "summary_of_the_review": "The paper contains many typos and ungrammatical sentences which make it difficult to read. To list a few:\n* Page 2: \"The long range dependencies The latter allows to model long-range dependencies just like self-attention would.\n* Page 3:  \"MLP-Mixer simply applies the MLP across the token dimension gMLP has a module called Spatial Gating Unit (SGU)\"\n* Page 6: \"For Librispeech, We tokenize\"\n\n\n* The related work section is very short, focuses only on models for images and gives no reference to ASR. \n\n* Section 2\n    * Equation 1, `Linear_{channel}` is referenced in the following sentence as `Linear_{token}`\n    * Equations in 2.1 are wrong or wrongly written. Some variable are not explained, and the dimensions don't add up when you consider all the projections.\n    * SGU is presented in section 2.2 but has been used in equations of section 2.1\n* Section 3\n  *  C-MLP', which will be the best model, is not clearly descibed (only equation 8)\n  *  \"This operation to obtain filter HCGU is independent of the channel dimension.\" : this is unclear\n* Section 4\n  * In the text, reported WER seem to have been copied from one to another, and don't match the table.\n  * `C-MLP achieves competitive performance with Transformer-based model with only 57.1 % of its parameters`, but after computing the value, it seems to be `57.4 %`\n  * Should be `Table 2` not `Table 1` in section 4.4\n  * C-MLP' is the best model based on table 1 but is not presented in  table 2\n  * In table 5, RTF is not defined\n  * In the claims (abstract), it is said that the proposed architecture reduces WER by 1.9/3.4 % but this result seems to be mainly due to the tiny attention module, which is not clearly stated.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper adopts the recently developed MLP-based architectures for image classification to Automatic Speech Recognition with 3 different modifications to handle variable length sequences. The three architectural modifications are: C-MLP (w/ depthwise convolution), TS-MLP (w/ shift operator), and F-MLP (w/ Fourier transform and w/o gating). The approaches are then evaluated on the Librispeech-100h and Tedlium-2 corpora and are compared to baselines from the literature. The proposed approaches are shown to yield better performance than Transformer-based models.

As pointed by the reviewers, there are 3 major concerns:
clarity: the initial version of the paper needed more improvement in writing, the authors did improve the writing a lot, which led to increased ratings by the reviewers;
reproduction: many experimental details were missing in the initial version, but the authors added those in the revision and shared the code
novelty: as agreed by all the reviewers that The novelty of the paper is somewhat weak. It is mainly an application of a known technique to a new use-case and the modifications are commonly used in ASR. 

The decision is mainly due to the limited novelty.
Decision: Reject