Speech-MLP: a simple MLP architecture for speech processing
Authors: Chao Xing
Dong Wang
Lirong Dai
Qun Liu
Anderson Avila
Venue: ICLR 2022 Submitted (2022)
Keywords: MLP
transformers
speech signal processing

Abstract: Overparameterized transformer-based architectures have shown remarkable performance in recent years, achieving state-of-the-art results in speech processing tasks such as speech recognition, speech synthesis, keyword spotting, and speech enhancement et al. The main assumption is that with the underlying self-attention mechanism, transformers can ultimately capture the long-range temporal dependency from speech signals. In this paper, we propose a multi-layer perceptron (MLP) architecture, namely speech-MLP, useful for extracting information from speech signals. The model splits feature channels into non-overlapped chunks and processes each chunk individually. The processed chunks are then merged together and processed to consolidate the output. By setting the different numbers of chunks and focusing on different contextual window sizes, speech-MLP learns multiscale local temporal dependency. The proposed model is successfully evaluated on two tasks: keyword spotting and speech enhancement. In our experiments, we use two benchmark datasets for keyword spotting (Google speech command V2-35 and LibriWords) and the VoiceBank dataset for the speech enhancement task. In all experiments, speech-MLP surpassed transformer-based solutions, achieving state-of-the-art performance with fewer parameters and simpler training schemes. Such results indicate that oftentimes more complex models such as transformers are not necessary for speech processing tasks. Hence, they should not be considered as the first option as simpler and more compact models can offer optimal performance. 

Under review as a conference paper at ICLR 2022
SPEECH-MLP: A SIMPLE MLP ARCHITECTURE FOR SPEECH PROCESSING
Anonymous authors Paper under double-blind review
ABSTRACT
Transformers have shown outstanding performance in recent years, achieving otate-of-the-art results in speech processing tasks such as speech recognition, speech synthesis and speech enhancement. In this paper, their success, such complex models are not needed for some important speech re- lated tasks, which be solved with much simpler and compact models. Thus, we propose a multi-layer perceptron (MLP) architecture, namely speech-MLP, useful for extracting information from speech signals. The model splits feature chan- nels into non-overlapped chunks and processes each chunk individually. These chunks are then merged together and further processed consolidate the output. By setting different numbers of chunks and focusing on different contextual win- dow sizes, speech-MIP learns multiscale local temporal dependency. The pro- posed model is successfully evaluated on two tasks: keyword spotting and speech enhancement. In our experiments, two benchmark datasets are adopted for key- word spotting (Google speech command V2-35 and LibriWords) and one dataset (VoiceBank) for speech enhancement. all experiments, speech-MLP surpassed the transformer-based solutions, achieving better performance with fewer param- eters lower GFLOPS. Such results indicate that more complex models, such as transformers, oftentimes not necessary for speech processing tasks. Hence, simpler and more compact models should always be considered as an alternative. specially in resource-constrained scenarios.
show that, despite
INTRODUCTION
As in many machine learning disciplines, speech processing is embracing more and more complex models, where transformer (Vaswani et al., (017) is a particular example. It was first proposed to tackle machine translation, and afterwards
successfully applied multiple research fields such as natural language processing (NLP) (Devlin et al. 2018) and computer vision (CV) (Dosovitskiy al. 2020). The of the transformer model is self-attention mechanism, by which any two elements in a sequence can interact with each other, hence capturing long-range dependency. Con- sidering that speech signals naturally cappora--depeneent researchers in the speech community recently explored transformer-basee models in multiple speech processing tasks, and remarkable performance was reported in speech recognition (Dong et al. 2018 Karita al. 2019 Huang al,, 2020), speech enhancement (SE) (Kim et al.) Fu Fu et al.) 2020), keyword spotting (KWS) (Berg jtt al. 202 Vygon & Mikhaylovskiy 2021) and speech synthesis (Li et al., (019). Recently, the conformer architecture, which combines convolution and self-attention, achieved excellent success in speech processing tasks and attracted much attention Gulati et al. (2020).
In this paper, ask the following question: Do need complex models such transformers for certain speech processing tasks?
This question is closely related to the principle of 'parsimony of explanations', a.k.a., Occam's zor (Walsh 1979). According to this principle, if there is any possibility, we should seek the models that can represent the data with the least complexity (Rasmussen & Ghahramani 2001 B Blumer et al. 1987p. However, in the public benchmark tests, complex and elaborately designed models are often ranked higher, due to the better reported performance. For example, the KWS benchmark Google

---

Under review as a conference paper at ICLR 2022
speech command and the SE benchmark YoiceBank-DEMAND transformer-bat based models are among the top ranks. Although the good performance is celebrating, the increased model complexity implies potential over-tuning and over-explanation, the risk that the Occam's 's razor principle intends to avoid.
We, therefore, attempt to discover the simplest neural architecture, that is powerful enough to achieve comparable performance as the best existing models, in particular transformers, while eliminating unnecessary complexity. Our design is based on domain knowledge, in particular, three properties of speech signals: (1) temporal invariance, (2) frequency asymmetry, and (3) short-term depen-
dency (Huang et al. 2001 Benesty et al. 2008 Furui 2018). Based on these knowledge, we build the speech-MLP, a simple multi-layer perceptron (MLP) )) architecture, shown in Fig. [ Besides the normalization components, the architecture involves simple linear transformations only. The core of the architecture is the Split & Glue layer, which splits the channel dimension into multiple chunks, processes each chunk separately, and finally merges the processed chunks in order to attain the out- put. Speech-MLP processes each time frames independently (compatible to temporal invariance), and the splitting & gluing procedure allows different treatments for different frequency bands (com- patible frequency asymmetry), and involves the local context of multiple scales (compatible to short-term dependency),
We tested the model two speech processing tasks: keyword spotting with the Google speech command V2-35 and Libriword benchmark datasets; and speech enhancement with the VoiceBank benchmark dataset. Results showed that both tasks the proposed speech-MLP outperforms com- plex models, in particular models based on transformers. Such results demonstrate that by utilizing domain knowledge and employing appropriate normalization techniques, it is possible design simple yet powerful models. In some cases, these simple models even beat complex models on open benchmarks, where complex models are more likely to obtain good performance by careful tuning.
In summary, we proposed Speech-MLP, simple yet effective neural model to represent speech sig- nal. On the KWS and SE tasks, we demonstrated that the simple model can achieve performance comparable to or even better than transformers with less parameters and inference time. Our work shows that by taking nomain-knowledde into account, it possible to remove unnecessary complex- ity (e.g.. modeling for the long-range dependency in KWS and SE) in model design, as advocated by the Occam's razor.
2 RELATED WORK
Recent research has shown that a simple model can be as effective as complex and task specific models such as transformers in some important tasks. In (Tolstikhin et al., (02T). for example, the authors proposed a simple architecture for vision, namely MLP-Mixer. The model receives sequence of image patches and performs channel-wise and patch-wise linear projection alterna- tively and iteratively. Without using convolutions or self-attention, the Mixer architecture separates the per-location (channel-mixing) and cross-location (token- (token-mixing) operations (Tolstikhin et al.) 2021]. While the mixing MLPs enable communication between different channels, the token-mixing MLPs allow communication between different spatial locations (tokens). Tested on image classification benchmarks, MIP-Mixer achieved performance comparable to SOTA models, in particular the vision transformer model (Tolstikhin et al. 2021).
In another recent work (Liu et al. 2021). the authors investigated the need of the self-attention mech- anism in transformers, proposing an alternative MLP-based architecture, namely gMLP. The model, based on MLP layers with gating, consists of a stack of L identical blocks. Each block comprises normalization layer, channel projection, followed by an activation function and spatial gating unit. followed by another channel projection (Liu et al. 200T). It achieves similar performance when compared the vision transformer model (Touvron et al.) (021b), being 3 % more accurate than the aforementioned MLP-mixter model with 66 % fewer parameters. The model was also success- ful on language modeling in the BERT setup (Liu et al. 2021), minimizing perplexity as well as Transformers. The authors also found that perplexity reduction was more influenced by the model capacity than by the attention mechanism.
2

---

Under review as a conference paper at ICLR 2022
Inspired by vision transformers Touvron al. 202Ib al. 2020), in (Touvron al. 202Ia), the authors apply the skip connection technique from ResNet's to MLP layers and propose the so-called Residual Multi-Layer Perceptrons (ResMLP). The model receives non-overlapping image patches, typically 16 x 16. These patches go through a linear transformation in order to attain d-dimensional embeddings. The emheddings then fed sequence of ResMLP blocks to produce set of d-dimensional output embeddings. An average pooling is applied on the d- dimension output vector represent the image, linear classifier used then to predict the label associated with the image (Touvron et al. 2021a).
Differently from Mixer-MLP: gMLP and ResMLP, CycleMLP can process inputs of arbitrary reso- lution with linear computational complexity as its receptive fields are enlarged for context aggrega- tion (Chen et al., 202T). The model is based on Cycle Fully-Connected I Layer (Cycle FC), serving architecture. Results show CycleMLP outperforming
as a generic, plug-and-play transformer- existing MLP-like models on ImageNet classification, achieving good performance on object detec- tion, instance segmentation and semantic segmentation (Chen et al. 2021
The aforementioned research highlights that, despite their success, convolution and self-attention mechanisms are not mandatory for some CV and NLP tasks, and can be replaced by simpler layers such as MLP with a customized design. Although typical convolution operations are not used by these MLP solutions (but rather 1 x 1 convolution as pointed out in (Chen et al., 202T) and (Tol- stikhin et al. 202T)), these MI.P approaches are inspired by CNN architectures for computer vision related tasks. Their building block, nonetheless, is similar and based applying linear transforma- tion on spatial locations and feature channels.
Although inspired by these new MLP architectures. speech-MLP focuses on speech signals rather than images. This implies in processing different input resolutions given the nature of the input signal. The split & glue layer is very similar separable CNN (Chen et al. 2018). if we re- gard the frame-inde -iteneiiddeeddennt processing as 1-D convolution in time. In particular, it is essentially group-wised CNN (Romero et al., 2020) with different kernels for each group. However, from the perspective of feature learning, the entire split & glue is an MLP if our focus is a particular frame (within a context). That is why 1-D convolution is often called time-delay neural net (TDNN) (Waibel et al., 1989). We follow this convention and name our structure as specch-MLP.
key motivation of the speech-MLP structure is to respect the properties of speech signals. It should be emphasized that almost all successful techniques in speech processing take these properties into account, for instance the hidden Markov model (HMM) assumes short-term dependency (Rabiner & Juang 1986). TDNN assumes temporal invariance (Waibel et al.) 1989), and frequency asymmetry is explicitly implemented in the famous MFCC feature (Mermelstein) 1976). In this paper, the role of knowledge of speech signals is to help remove unnecessary complexity, i.e., seeking the minimum structure that make reflect these basic properties.
Finally, MLP is not new in speech processing; in fact the neural models used in early days in speech processing are all general MIPs, e.g., (Bourlard & Morgan 2012). Speech-MLP is a special de- signed MIP, by taking the properties of speech signals into account.
3 METHODOLOGY
Our model, referred to as speech-MLP, is presented in Figure I Note that for a given speech wave- form, a sequence of acoustic features, denoted by X { X22, J. are first extracted. These features are then fed into N stacked speech-MLP blocks and the output of the last speech-MLP block speech representation that needs to undergo susk-specific layers in order perform spe- cific tasks, such as the ones addressed in this study: SE and KWS.
Inside of each speech-MLPP block, there are three components: pre-projection of the extracted acoustic features; (2) a Split & Glue layer for processing the pro- jected acoustic features while addressing frequency asymmetry and temporal dependency, and (3) another linear transformation for post-projection of the final representation. Two residual connec- tions are also adopted to encourage gradient propagation. The first one maps the input features onto the output of the last linear transformation (i.e., the output of the post-projection operation). The second residual connection maps the output of the first linear transformation (i.c., the output of the pre-projection operation) onto the output of the Split& & Glue layer. Note that normalization tech-
(1) linear transformation for a
3

---

Under review as a conference paper at ICLR 2022
Task Specific Layers
Norm
Linear
GELU
Linear
Concaten.aedd Feutures
GELU
Linear
Norm
Linear1
Linear
Unfolded Factures
Split & Glue
Unfolded Features 1
Unfolded Feotures 0
Chunked Features
Chunked Chunked
Linear
Feature Extraction
Figure 1: Proposed architecture consisting of N speech-MLP blocks, comprising three components: (1) linear transformation as pre-projection; (2) a Split Glue layer the main processing module; and (3) another linear transformation as post-processing.
niques are also applied regulate the feature distribution (by layer norm) and temporal variance (by instance norm). In the next section, give more details the Split & Glue layer, followed by discussion on the normalization methods adopted in this work.
3.1 SPLIT & GLUE
Figure depicts how the Split & Glue layer operates. The sequence of acoustic features is denoted by X E R Mx,, with T and being, respectively, the length and the number of channels of the input seguence. The first step is split X into K non-overlapping chunks,
illustrated in both Figure and Figure 2 The split referred to as > {X1 X :... ), is performed along the channel dimension. In our experiments, the channel dimension of each chunk is considered the same, leading to X R For each chunk, X context expansion is then performed through the so-called unfolding operations. This results context-expanded chunks, denoted by X E RR" H/KXT where w is the size of the context window induced by the unfolding operation.
Note that the number of chunks K and the window size w can be arbitrarily selected for each chunk. This flexibility allows to represent multi-scale contexts by adopting different window sizes for different chunks. In Figure for instance, the input channels are split into two chunks, and the window sizes set to 3 and 5, respectively. This leads to the model learning from small and large contexts simultancously.
The unfolded chunk X is projected by linear transformation. leading to a new representation for the initial chunk, y E RTT where A could be set arbitrary and is called the number of Glue channels. We highlight that the linear transformation used in the above chunk-wise operation is shared across all the time steps for single chunk, and each time frame is processed independently. This setting reduces the number of parameters and is compatible with the temporal invariance prop- erty of speech signals. Nevertheless, different weight parameters are adopted for different chunks, to provide sufficient flexibility.
Finally, all the learned speech representations, Y are concatenated along the channel dimension, forming glued feature matrix yo {Y), y2 yk). Following, another linear transformation

---

Under review as a conference paper at ICLR 2022
X
0
1
3
windoww
Liwer
1
2
4
1
2
4
P
y1
X
4
6
T
P
5
7
nithow
Uniold
6
8
Uncar A
9
X
y2
8
9
P
XG
T
H
6
y6
Figure 2: The Split Glue layer, best viewed color. The input feature X is split to chunks (denoted by blue and green box respectively), and the window sizes of the unfold operation are set to and respectively for the two chunks. The red box indicates a single time step, and 'P' X and X? represents padding.
is applied in order to obtain the output feature Y R"xT, Again, the linear transformation is shared across all the time steps, to reflect temporal invariance.
3.2 NORMALIZATIONS
Normalization plays important role in our speech-MLP model. We employed two normaliza- tion approaches: (1) layer normalization (LN) (Ba et al. 2016) and (2) instance normalization (IN) (Ulyanov et al. 2016).
Layer normalization is applied across the channel dimension at each time step. Thus, it computes statistics (mean and variance) on each column of X R#*T, and then uses these statisties to normalize the elements in the same column. With this normalization technique, the distribution of the feature vector at each time step is regularized.
Instance normalization is used to perform per-channel normalization. That is, the statistics are com- puted on each row of X #xT and applied across the time steps to normalize the elements of each row. Thus, the temporal variation of each channel is normalized. Note that IN extends the conventional cepstral mean normalization (CMN) approach (Liu et al. 1993), by normalizing not only acoustic features, but also features produced by any hidden layer.
Empirically, we found that was only effective for the SE task while the LN was more important for the KWS task. Therefore, we apply LN only for KWS and IN for SE.
EXPERIMENTS
We evaluate the proposed speech-MLP model in two speech processing tasks: speech enhancement and keyword spotting. In this section, we introduce these tasks and their respective datasets, used in our experiments, followed by experimental settings, experimental results. and the ablation study
Thhe code will be available on github. To respect the double-blind review, the link will be sent to the reviewers when the discussion is open.
5

---

Under review as a conference paper at ICLR 2022
Table 1: Adopted architectures for the KWS and SE tasks. In the KWS setting, (S) and (L) denote small model and large model respectively.
KWS
SE
Input Channels
40
257
Linear 0
Output Channels
128(S/L)/26(XXL)
256
Bias
true
true
#Blocks
4(S/LV12(XL)
10
Input Channels
128($11)/256(X1.
256
Glue Channels
(015)/0001_)120000
60
Speech-MI.P
Hidden Channels
401S/80(LL1000XX)
40
Bias
true
true
Context Window
{3, 9, 11}
{3. 9, 11]
Normalization
Layer Norm
Instance Norm
GELU
MaxPooling
Input Channels
128
256
Linear 1
Output Channels
128
257
Bías
true
true
GELU
Input Channels
128
Linear 2
Output Channels
#Keywords true
Bias
4.1 KEYWORD SPOTTING
Keyword spotting aims at detecting predefined words in speech utterances (Szöke al. 2005
Mamou et al. 2007] Wang 2010 Mandal et al. 2014). In our experiments, we explore two KWS
datasets: (1) the Google speech commands V2 dataset (Warden 2018), and (2) the LibriWords (Vy-
gon & Mikhaylovskiy. 2021). The Google speech commands V2 dataset (here, referred to as V2-35)
consists of T05,829 829 utterances of 35 words, recorded by 2,618 speakers. The training. validation and test sets contain 84, 843, 11, 005 and 9, 981 utterances respectively. The LibriWords dataset, derived from 1000-hours of English speech from the LibriSpeech
larger and more complex. dataset (Panayotov et al., 2015). Signal-to-word alignments were generated using the Montreal Forced Aligner (McAuliffe et al. 2017) and are available (Lugosch et al., 2019). The averaged duration of the keywords are 0.28 seconds. The provider defined four benchmark tests, based on the number target keywords: LW-10, LW-100, LW-1K and LW-10K, where the target keywords are 10, 100, 1k and 10k respectively. More details this dataset are presented in Appendix.
4.1.1 SETTINGS
We used the same architecture in all the KWS tasks, except that the dimension of the output layer was adapted to the number of keywords, as shown in Table Note that we set the window size w be {3, 7, 11). This allows to exploit multi-scale contexts. Additionally, we the stride to be 1 and appropriately set the padding list to ensure that all the expanded features are in the same length and equal to that of the input feature.
Prior to the feature extraction step, each speech recording is resampled to 16 kHz. Then, 40- dimensional Mel-Frequency Cepstral Coefficients (MFCC) are attained the acoustic features. The MFCC features are then projected target dimensional feature vector by linear layer and then for- warded to speech-MI. blocks. The output features are then passed through a max-pooling operation collects the information across time steps. Finally, two linear layers with GELU activation function in the middle and a softmax activation are employed in order to attain the posterior probabilities that the input speech belongs to each keyword. For regularization we used SpecAugment (Park al., 2019), dropout (Baldi & Sadowski 2013). and label smoothing (Müller et al. 2019 were used to prevent overfitting.
Three model architectures have been verified in all the experiments: a 180k small model denoted by Speech-MI. Miach-MMIP-S.. A.S.SS 480k large model denoted by Speech-MI.P-L, and 2375K extra large model

---

Under review as a conference paper at ICLR 2022
Table 2: Performance comparison on KWS tasks in terms of accuracy (he size reported from the entire KWS model on the V2-35 task. Due to the different numbers of keywords, the value varies from task to task, although the backbone is the same. # #The Gflops is calculated on V2-35 task.
Accuracy%
Models
GFlops
Size
V2-35
LW-10
LW-100
LW-IK
LW-TOK
AR-RNN (de Andrade et 2018
2024
93,90
KWT-I (Berg al..
2021
0.108
607k
96.85
0.07
KWT-2 (Berg et al.
2021
0.469
2394k
97.53
0.07
KWT-3 Berg et al.
2021
1.053
5361k
97.51 + 0.14
Res15-CE Vygon & Mikhaylovskiy 2021)
237k
95,96
88.8
82.3
78.2
69.3
Res15-TL Vygon & Mikhaylovskiy 2021
237k
97.00
91.7
86.9
84.3
81.2
Speech-MLP-S (Ours)
0.016
180k
97.15
1.155:0..00 0.07
95.03
90.91
90.16
89.16
Speoch-MLP-L (Ours)
0.046
480k
97.36
-: 0.16
95.37
92.11
91.50
90.82
Speech-MLP XL (Ours)
0.228
2375K
97.56 + 0.09
95.80
93,22
93,27
93.01
denoted by Speech-MLP-XL The three models are different in the number channels of the hidden layer (i.e., after the pre-projection) and the channels within the Split & Glue block (i.e., channels after Linear A, and layers Fig. 2). as shown in Table
For the experiments on the Google speech commands dataset, applied the following data aug- mentation techniques: time shifting, audio re-sampling and noise perturbation: as (Berg et al.) 2021 Vygon & Mikhaylovskiy 2021). After augmentation, the data increased 10 times the size of V2-35. We set the hatch size to be 256 and trained the model for 100 epochs on 4 cards V100 Nvidia GPU.
For the experiments on the LibriWords, the batch size was set to 1024, and we trained the model for 20 epochs 2 cards V100 Nvidia GPU which showed to be enough for this dataset. The training schemes were set differently simply because Libriwords is huge and long-term training is not economic.
The performance of the proposed model is compared to three benchmarks. The first one referred as Att-RNN, CNN-LSTM architecture with the attention mechanism introduced in (de An- drade et al. 2018). The model has approximately 202k trainable parameters and attains reasonable performance. Another recent solution, based on transformer architecture is adopted as the second benchmark (Berg et al. 2021). We refer to this benchmark as KWT-K where K refers to differ- ent size of models. Res 15 (Vygon & Mikhaylovskin 2021], another recent work based on ResNet reports high performance on both V2-35 and Libriwords. The authors reported results with two figurations, one trained by cross entropy (Res15-CE) and the other based on triple loss (Res15-TL) We use them as the third benchmark.
4.1.2 RESULTS
Table[preseett the results of the benchmarks discussed in the previous section and the performance of the proposed Speech-MLDD the experimental results are presented by mean value and 95% con- fidence of 5 trials with different random seeds on V2-35. It can be observed that the Speech-MLP models outperform all the benchmarks with comparable model sizes. Note that the small version of speech-MLP, which contains less than half of the parameters of its large version, still maintain reasonable performance, providing higher accuracy than most benchmarks. The performance of our solution the Libriword dataset is even more significant. It outperforms Res15-CE and 1es15-TL. while being able to maintain performance across all LibriWord dataset sizes. Our conjecture that by the knowledge-driven design, we can use the parameters more efficiently, which allows for the use of smaller models to handle large-scale tasks.
4.1.3 ABLATION STUDY
To investigate how each module impacts the performance of speech-MLP, we conducted an ablation study, in order to fair compare each model we fixed random seed 123 in all ablation study experiments, we show that window list to {3} equivalent to use TDNN with kernel size to 3, and window list to (3,
3, 3,3) 3) equivalent the TDNN with 4 groups convolution operation with kernel size
7

---

Under review as a conference paper at ICLR 2022
to 3 in split & glue layer, and our proposed speech-MLP with variance of window sizes outperform these existing solutions. We particularly focus on the chunk splitting, specially the number of chunks and the context window of each chunk. They are the only hyperparameters that need to design in speech-MLP, by using domain knowledge.
The results reported in Table It can be observed that the setting for the number of chunks and the context window does matter. A longer context window is clearly beneficial, and setting differ- context windows for different chunks can further improve the performance. This confirms our conjecture that contextual information is important for representing speech signals, and exploiting multi-scale contextual information is especially important.
An interesting comparison is between the Preech-MLP.. model with window {3, 7,9, 11} and the Speech-MLP-L model with window {1}. The parameters of the two models are comparable, but the latter model does not involve any chunk splitting and context expansion. The clear advantage of the Speech-MIL MLP-- model demonstrated that the performance improvement with larger and multi-scale context windows (ref. performance of Speech-MLP-S or Speech-MLP-I with different windows) due to the newly designed Split& Glue structure, rather than the increase in parameters. This in turn demonstrated the value of domain knowledge: if we can exploit it appropriately, it is possible to design very parsimonious models.
Table 3: Performance of speech-MLP on KWS tasks with different configurations.
Accuracy%
Models
Window
Size
V2-35
LW-10
LW- 100
LW 1000
LW: 10000
(3,7,9111)
180k
97.14
95.03
90.91
90.16
89.16
Speech-MLP-S
(3,3,33)
138k
96.74
94.83
90.25
89.20
87.89
(3)
108k
96.46
94.31
89.11
87.81
85.69
(1)
88k
87.67
66.06
36.96
21.87
22.03
13,7,9,11}
480k
97.31
95.37
92.11
91.50
90.82
Speech-MLP-L
(3,3,333
337K
96.96
95.29
91.52
90.83
89.89
(3)
239k
96.95
94.79
90.26
89.26
87.52
{1}
175k
82.43
66.12
32.36
28.42
22.85
(3,7,9.11)
2375K
97.60
95.80
93.22
93.27
93.01
Speech-MLP-XL
(3,3,3.3
1727K
97.39
95.65
92.97
92.75
92.24
(3)
1291K
97.50
95.45
92.44
92.16
91.55
(1)
1003K
85.69
61.03
45.88
28.59
22.51
4.2 SPEECH ENIANCEMENT
Speech enhancement, which aims at inferring clean speech from its corrupted version (Benesty let al.] 2006 Loizou 2007| Das et al. 2020)) is another fundamental task used to evaluate model. We choose the Voicebank Demand datase
et al. (2016) to perform the SE test. It contains clean speech signals from the Voicebank dataset, includes 28 speakers for training and speakers for testing. Noise signals of 40 types from the DEMAND atabase TThemann et al. (2013) were selected and were mixed into the clean speech. After the mixing, the training and testing set involve 11,572 and 824 clips respectively. We split the training utterances into segments of seconds without overlap. This resulted into 17,989 training samples, each sampling consisting of noise corrupted segment and the corresponding clean segment. The goal of SE is to learn mapping function that converts a noisy segment to a clean segment.
4.2. SETTINGS
The architecture of our SE model is shown in Table [ As input, the model receives 257- dimensional log-magnitude spectrum. The extracted features are first projected by linear layer and reduced to 256-dimensional feature vector, which are then forwarded to 10 stacked speech- MLP blocks. The output from the last speech-ML. MMP block is re-projected to 257-dimensional feature vector. After hard- sigmodd function Corionnniuuaaa et al. (2015). the value of the output units corre- spond to the ratio masks on the dimeennnni input log-magnitude spectrum. The clean speech signal is estimated by applying the ratio masks onto the noisy spectrum and reusing the noisy phase.
8

---

Under review as a conference paper at ICLR 2022
Table 4: SE results on VoiceBank=Demand dataset. "The size of T-GSA was estimated from the structure reported in the original paper, which involves 10 transformer layers, and the dimension of the input/output is 1024.
Models
PESQ
CSIG
CBAK
COVL
Size
Unprocessed speech
1.97
3.37
2.49
2.66
SEGAN Pascual al.
2017
2.16
3.48
2.94
2.8
TF-GAN Soni et al.
2018
2.53
3.8
3.12
3.14
MDPHID Kim
2018
2.7
3.85
3.39
3.27
Metric-GAN
(Fu et al. 2019
2,86
3.99
3.39
3.42
Metric-GANt [uu al. 202
3.15
4.14
3.16
Metric-GAND [Fu et al.
20216
2.45
3.47
2,63
2.91
PHASEN YYn et 2020
2.99
3.55
T-GSA (Kim al.
2020
3.06
4.18
3,59
3.62
60M
Speech-MLP (Ours)
3.08
4.28
3.50
3.70
636K
More details the settings can be found in Appendix. The performance of the proposed model compared to six benchmarks. Note that we focus models trained without extra data, extra models for knowledge distillation. The reader can find details on these enhancement methods in the references presented in Table 4 Following the convention on this test set, we report the results of four metrics: PESQ, BAK, SIG and OVL. (Hu Loizou 2007).
4.2.2 RESULTS
The results are shown in Table where we choose baseline systems for comparison. Among these systems, T-GAS (Kim et al., 2020 is based on a transformer model. Similar to speech-MLP, the authors of T-GAS also noticed the importance of local context and designed an annealing approach to encourage attention neighbour frames. However the attention is still global in nature, and the improvement with T-GAS was still attributed to the capacity of transformers in learning (not so) long-range dependency. Note that the size of the T-GAS model was not reported in the original paper, so we made an estimation according to the structure description.
The results shown in Table 4 demonstrated that speech-MLP model outperformed all the six baselines. In particular, without modeling any long-range dependency, it outperformed T-GSA by almost 100 times smaller of model size. This comparative results challenge the assumption that the better performance of T-GSA over other baselines is due to its capacity of capturing long-range dependence in speech. Moreover, the model size of Speech-MLP and due to the concise architecture, the training is simple and fast. It provides strong support for our argument that complex models are not necessarily the best, and a knowledge-base based model may easily beat complex models with parsimonious parameters.
much smaller than T-GSA,
5 CONCLUSIONS
In this paper, we propose the speech-MLP model, a simple MLP architecture for speech processing tasks. Our main motivation was to find a compact solution that eliminates unnecessary complexity while being able to capture essential information from speech signals. By utilizing domain knowl- edge of speech, and normalization. The main ingredient is a split & glue structure, which splits input features into multiple chunks and makes them accounting for different contexts. This knowledge-bert based design reflects several properties of speech signals, including temporal variance, frequency symmetry, and short-term dependency. The experimental results on keyword spotting and speech enhancement demonstrated that speech-MLP is highly effective: with much less parameters and computation, can beat larger and more elaborately designed models including transformers.
designed a simple yet effective structure that involves only linear transform
Much work remains, for example, how to design better chunking and context; how to make the model even smaller (e.g., removing unnecessary residual connections); how to trade off the com- plexity in chunks and in depth. The ultimate goal is to design a liggt-weighted, sufficiently powerful and generalizable component for speech feature extraction. We believe the knowledge-drive feature extractor benefits general speech processing tasks, such as speech recognition and understanding.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We made the following efforts to ensure that the results reported in the paper can be reproduced by other researchers.
We will release the code on github, everyone can download
The datasets used in this paper are all publicly available for researchers
We documented the required python environment and provided step-by-step guidance for the reproduction
We fixed the random seed in the code, so that others can reproduce our result exactly.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv: arXiv:1607.00 1607706077600 2016.
Pierre Baldi and Peter J Sadowski. Understanding dropout. Advances in neural information pro- cessing systems, 26:2814-2822 2013.
Jacob Benesty, Shoji Makino, and Jingdong Chen. Speech enhancement. Springer Science & Busi- ness Media, 2006.
Jacob Benesty, M Mohan Sondhi, Yiteng Huang, al. Springer handbook of speech processing, volume Springer, 2008
Axel Berg, Mark O' Connor, and Miguel Tairum Cruz. Keyword transformer: A self-attention model for keyword spotting. arXiv preprint arXiv:2104.00769 2021.
Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred Warmuth. Occam's razor. Information processing letters, 24(6)3377-380, 1987.
Herve Bourlard and Nelson Morgan. Connectionist speech recognition: hybrid approach, vol- ume 247. Springer Science & Business Media, 2012.
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV). pp. 801-818, 2018.
Shoufa Chen, Enze Xie, Chongjian Ge, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architec- ture for dense prediction. arXiv preprint arXiv:2107.10224 2021.
Matthieu Courbariaux, Yoshua Bengio, and
David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Nabanita Das, Sayan Chakraborty, Jyotismita Chaki, Neelamadhab Padhy, and Nilanjan Dey. Funda- mentals, present and future perspectives of speech enhancement. International Journal of Speech Technology, pp. 1-19, 2020.
Douglas Coimbra de Andrade, Sabato Leo, Martin Loesener Da Silva Viana, and Christoph Bernkopf. A neural attention model for speech command recognition. arXiv preprint 1rXi::1808.0892,, 2018
Jacob Devlin, Ming- bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
Linhao Dong, Shuang Xu, and Bo Xu. Spasch-transformme:: no-recurrence soquence-to-sequene model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884-5888. IEEE, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11020.0...199............99 2020.
Szu-Wei Fu, Chien- Liao, Yu Tsao, and Shou-De Lin. Metricgan: Generative adversarial net- works based black-box metric scores optimization for speech enhancement. In International Conference Machine Learning, pp. 2031-2041. PMLR, 2019.
Szu-War Wei Fu, Chien-Feng Liao, Tsun-An Hsieh, Hio--Huaan Hung, Syu-Siang Wang. Cheng Yu, Heng-Cheng Kuo, Ryandhimas E Zezario. You-Jin Li, Shang- Yi Chuang, et al. Boosting ob- jective scores speech enhancement model through metricgan post- processoc....ng arXiv preprint arXiv:2006. 10296, 2020.
11

---

Under review as a conference paper at ICLR 2022
Szu-Wei Fu, Cheng Yu, Tsun-An Hsieh, Peter Plantinga, Mirco Ravanelli, Xugang Lu, and Yu Tsao. Metricgan+: An improved version of metricgan for speech enhancement. arXiv preprint TrXiv:2/04.03538, 2021a.
Szu-Wei Fu, Cheng Yu, Kuo-Hsuan Hung, Mirco Ravanelli, and Yu Tsao. Metricgan-u: Unsu- pervised speech enhancementdeeeevettraiii based only on noisy/reverberated speech. arXiv preprint arXiv:2110.05866 2021b.
Sadaoki Furui. Digital speech processing, synthesis, and recognition: 2018.
Synthesis, and recognition.
Anmol Gulati, James Qin, Chung- Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmentéé transformer for speech recognition. arXiv preprint arXiv:2005.08100 2020.
Yi Hu and Philipos C Loizou. Evaluation of objective quality measures for speech enhancement. IEEE Transactions audio, speech, and language processing, 16(1):229-23) 2007.
Wenyong Huang, Wenchao Hu, Yu Ting Yeung, and Xiao Chen. Conv-transformer trans- ducer: Low latency, low frame rate, streamable end-to-end speech recognition. arXiv preprint arXiv:2008.055 05750, 2020.
Xuedong Huang, Alex Acero, Iisiao-Wuen Hon, and Raj Reddy. Spoken language processing: A guide to theory, algorithm, and system developmens. Prentice hall PTR, 2001.
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al. A com- parative study on transformer rnn in speech applications. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 449-456. IEEE, 2019.
Jaeyoung Kim, Mostafa El-Khamy, and Jungwon Lee. T-gsa: Transformer with gaussian-weighted self-attention for speech enhancement. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6649-6653. IEEE, 2020.
Jang-Hye Hyuu Kim, Jaejun Yoo, Sanghyuk Chun, Adrian Kim, and Jung-Woy Woo Ha. Multi-domain pro- cessing via hybrid denoising networks for speech enhancement. arXiv preprint arXiv: / 1a0::089120000 2018.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with trans- former network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6706-677 671 13, 2019.
Fu-Hua Liu, Richard Stern, Xuedong Huang, and Alejandro Acero. Efficient cepstral normaliza- tion for robust speech recognition. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993, 1993.
Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le. Pay attention to mlps. arXiv preprint orXiv:2105.08050 2021.
Philipos C Loizou. Speech enhancement: theory and practice. CRC press, 2007.
Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. Speech model pre- pre-training for end-to-end spoken language understanding. arXiv preprint arXiv:1904.03670 2019.
Jonathan Mamou, Bhuvana Ramabhadran, and Olivier Siohan. Vocabulary independent spoken term detection. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 615-622, 2007.
Anupam Mandal, KR Prasanna Kumar, and Pabitra Mitra. Recent developments in spoken term detection: survey. International Journal of Speech Technology, 17(2):183-198, 2014.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pp. 498-502, 2017.
12

---

Under review as a conference paper at ICLR 2022
Paul Mermelstein. Distance measures for speech recognition, psychological and instrumental. Par- tern recognition and artificial intelligence, 116:374-388, 0874488,..1 1976.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv:1906. 0262000699.000 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, Librispeech: an ast corpus based on public domain audio books. In 20/5 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-52022222 5210. IEEE, 2015.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprini arXiv: 2019.
Santiago Pascual, Antonio Bonafonte, and Joan Serra. Segan: Speech enhancement generative adversarial network. arXiv preprint arXiv:1703.09452 2017.
Lawrence Rabiner and Biinghwang Juang. An introduction hidden markov models. ieee assp magazine, 3(1):4-16, 1986.
Carl Edward Rasmussen and Zoubin Ghahramani. Occam's razor. Advances in neural information processing systems, pp. 294-300, 2001.
David Romero, Erik Bekkers, Jakub Tomezak, and Mark Hoogendoorn. Attentive group equívari- ant convolutional networks. International Conference on Machine Learning, pp. 8188-8199. PMLR, 2020.
Meet Soni, Neil Shah, and Hemant A Patil. Time-frequency masking-based speech enhance- ment using generative adversarial network. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5039-5043. IEEE, 2018
Igor Szöke, Petr Schwarz, Pavel Matejka, Lukás Burget, Martin Karafíát, Michal Fapso, and Jan Cernckký. Comparison of keyword spotting approaches for informal continuous speech. In In- terspeech, 633-636, 2005.
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments
acoustic noise database: database multichannel environmental noise recordings. The Journal of the Acoustical Society of America, 133(5):3591-359) 2013.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung. Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601, 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmip: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv: 1ee:22105.00..0 03404, 2021a.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021b.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 100.00160...8022. 2016.
Cassia
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Investigating rnn- based speech enhancement methods for noise-robust text-to-speech. SSW, pp. 146-152, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Roman Vygon and Nikolay Mikhaylovskiy. Learning efficient representations for keyword spotting with triplet loss. arXiv preprint arXiv:2101.04792. 2021.
13

---

Under review as a conference paper at ICLR 2022
Alex Waibel, Toshiyuki Hanazawa, Geoffrey Hinton, Kiyohiro Shikano, and Kevin J Lang. Phoneme recognition using time-delay neural networks. IEEE transactions on acoustics, speech, and signal processing. 37(3):328-33 -398, 1989.
Dorothy Walsh. Occam's razor: A principle of intellectual elegance. American Philosophical Quar- terly, 16(3):241-244, 1979.
Dong Wang. Out-of-vocabulary spoken term detection. 2010.
Pete Warden. Speech commands: A dataset for limited-vocabulaan speech recognition. arXiv preprint urXiv:180.003209 2018.
Dacheng Yin, Chong Luo, Zhiwei Xiong, and Wenjun Zeng. Phasen: A phase-and-harmonics--a speech enhancement network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 9458-9465, 2020.
14

---

Under review as a conference paper at ICLR 2022
A APPENDIX A: DETAILS OF KWS EXPERIMENT
In this section, we present the details of the KWS experiment. We start with the system architecture, followed by the data preparation. We then present the training methods and the hyperparameters used in the experiments.
Table 5: Experimental settings on keyword spotting
Parameter
Value
win
512
win_hop
160
len
480
Feature Extraction
n.mel
80
n_mfec
40
window
hann
Time masks
Time mask bin
[0, 15]
Spec Aug
Frequency Masks
Frequency mask bin
[0, 7]
init Ir
1.0c-03(V2-35/11 100-02(LW))))
label smoothing
0.1
Schedule
cosine
Learning Parameters
end Ir
1.0e-05(V2-35)1..oe (ee-00LWWW
Optimizer
AdamW
Weight Decay
1.0e-4
Dropout
0.1
A.1 SYSTEM ARCHITECTURE
Prior to feature extraction, speech signals resampled 16 kHz if needed. Then, librosa" to extract a 40-dimensional MFCC features. The parameters used to extract these features are pre- sented in Table Global mean and variance is also applied to normalize the extracted features. These statistics are calculated using the respective training set of each task. After that, the features fed into the model shown in Figure 3
Specifically, a linear transformation (Linear 0) operates on the normalized MFCC features, project- ing them 128- -i-dimensional embeddings. These embeddings are then forwarded to stacked Speech- MLP blocks (4 blocks in our KWS study) to extract multiscale contextual representations. For each speech utterance, the last Speech-MI.P block outputs a sequence of context-rich representations, and then max pooling operation is adopted to aggregate this sequence to a single utterance-level representation. This representation is then passed to 128 x 128 lincar transformation and a GELU nonlinear activation function. It is then further processed by 128 x M linear transformation and a softmax nonlinear activation, where M the number of keywords. The final output of the above process is vector that represents the posterior probabilities that the original speech utterance be- longs to each keyword.
A.2 DATA PREPARATION
A.2.1 GOOGLE SPEECH COMMANDS
The google speech commands V2-35 contains 35 classes. The data can be obtained at the provider's w There are 84, 843 training samples in total, with strictly no overlapping between training, validation and test sets.
http://downl http:///eenn.......eroooooooroo
15

---

Under review as a conference paper at ICLR 2022
On, Up,...
Softmax
Linear 2
GELU
Linear 1
MaxPooling
Speech MLP Blocks
Linear
MFCC
Figure 3: The system architecture used in the keyword spotting experiment.
Data augmentation techniques have been used to increase the training data by 9 times. Combined with the original data, we have 848, 430 training samples in total. We fixed the random seed to be 59185 when producing the augmented samples. Following are the augmentation strategies adopted in this work:
Noise perturbation: the noise perturbation script provided by the organizer of the DNS challenge used add background noise to clean speect The SNR factor is randomly sampled from [5, 10, 15] with equal probabilities;
Time shifting: time shifting is applied in the time domain. It shifts the waveform by a time- shift factor sampled from [-T, In our experiments we set T 100. When < 0, the waveform is shifted left by f samples and zeros are padded to the right side. When t > 0, the waveform shifted right by samples and zeros are padded to left side;
is used to perform resampling augmentation, which changes the sampling rate slightly. Specifically, given parameter R, resampling factor is drawn from [1 R,, 1 + R). and the augmented sample is obtained by changing the sampling rate to x 16000. R is set to 0.15 in our experiments.
Resampling:
the resample function from scipy heaipyssignal.ressaane
We the segmental_enr_mizze function from DNS- Cus-Challin T
https: thu..htubb..... com/ wtt.b//cco//mroof.......
16

---

Under review as a conference paper at ICLR 2022
After the above augmentation, the original speech and the augmented speech further corrupted by SpecAug (Park et al., 2019). The setting of SpecAug is shown Table Note that SpecAug does not enlarge the dataset.
A.2.2 LIBRIWORDS
The LirbriWords dataset is larger and more complex dataset. The samples extracted from the json files provided by the provider We follow the task definition of the dataset provider, and the details are given below.
LibriWords 10 (LW-10): this task contains 10 keywords. including "the", "and", "of", "to", "a", "in", "he", "I", "that", and "was". There are 1, 750k samples in total, and they are spit to a training set (1, 400k), validation set (262, 512) and test set (87, 501).
LibriWords 100 (LW-100): a more challenging task that contains 100 keywords. There are 1,5128 2k training samples, 189, 010 validation samples and 188, 968 samples, totalling 1, 890k samples.
LibriWords 1000 (LW-1K); with increased difficult, this task contains 1000 keywords. The training set involves 2, 178k samples, and the validation set and the test set contain 272, 329 samples and 271, 858 samples respectively.
LibriWords 10000 (LW-10K): The most challenging task presents 9998 keywords. The training set contains 2, 719k training samples, 339, 849 validation samples and 335, 046 test samples.
Given the large number of samples, data augmentation was not required for this task. We only performed SpecAug (Park et al. 2019) based on the settings presented in Table
A.3 TRAINING PARAMETERS
The parameters used during training are specified in Table Further details are presented below.
the cross entropy between the model prediction and the ground truth is used loss func- tion;
The optimizer used in all the experiments is AdamW. The initial learning rate set to 0.01, and cosine annealing is applied to adjust the learning rate from 0.01 to 0.0001;
Dropout is applied onto the residual connections within the speech-MLP block, with the dropout rate set to 0.1;
Label smoothing employed to prevent the over-confidence problem. The smoothing factor is set to 0.1;
In the V2-35 experiment, the models are trained for 100 epochs and 10 epochs warmup is applied, In the LibriWords experiment, the models are trained for 20 epochs without warmup;
In both the experiments, the model of each epoch is evaluated on the evaluation set, and the checkpoint that performs the best on the validation set is saved to report the performance test set;
We fix the random seed to be 123 in all the ablation study experiments, for the sake of reproducibility.
B APPENDIX B: DETAILS OF SE EXPERIMENT
B.1 SYSTEM ARCHITECTURE
The model architecture has been presented Figure The primary goal is learn mapping function that converts noisy magnitude spectrum to clean magnitude spectrum. The model output
httees///ttthaoocom///mmoooogggp 1 oos__ws
17

---

Under review as a conference paper at ICLR 2022
Table 6: Experimental settings on speech enhancement
Parameter
Value
win len
512
win_len
480
Feature Extraction
win hop
160
fft
512
window
hann
init Ir
1.0e-02
Schedule
cosine
3000
Learning Parameters
end Ir
1.0e-04
Warmup
30
Optimizer
Adam W
Epoch
1000
Inverse STFT
HardSigmoid
Linear
GELU
InstanceNorm
Speech MLP Blocks
Linear
Magnitude
Phase
STFT
Figure 4: The system architecture used in the speech enhancement experiment.
predicts the soft ratio masks, that can be applied to the noisy magnitude spectrum to estimate the mangitude spectrum the clean speech. Combining the denoised magnitude spectrum and the phase spectrum of the original noisy speech, one can attain the denoised waveform by inverse STFT
We used the STFT class implemented in the torch-mfee tookkinh http::///ggttuub.cccccccccottttt
18

---

Under review as a conference paper at ICLR 2022
More specifically, 257-dimensional log oog-mannitude spectrum is firstly extracted from the noisy speech as the acoustic features, following the configuration shown in Table 6 Then a linear layer follows and transfers the input features 256-dimensional vectors PreX. The transformed feature vectors are then forwarded to 10 Speech-MI.3 blocks, and the output from the last block, denoted by PostXX, involves multiscale contextual information. Afterwards, residual connection adds PreX and Post X together, and instance normalization is applied to regulate temporal variance. Finally, another linear transform and non-linear HardSigmoid activation projects the normalized feature to masking space where the dimensionality is the same as the input feature, corresponding the ratio mask M [0, on the noisy magnitude spectrum.
B.2 LOSS FUNCTIONS
The loss function of model is computed based the discrepancy between the denoised speech Xa and the clean speech Xe. The entire loss consists two parts: (1) the distance on power- composed magnitude spectrum, denoted by Lmag. and (2) the distance on power-compressed STFT, denoted by ft We use a single frame to demonstrate this computation, where the real loss should compute the average of on all the frames.
Dreat, Dimag STFT(XX) Creat, Cimag = STFT(XX)
Dmag
Dreat
D Dimag 2
Omag
+ Cimag
Lmag
(Chig
Cmngy )2
mag
D0.3 mag
D
Dreal
Dmag
D0.3
0.3 D smag
x Dimag
Dmag
G
CO.3
Creat
Cmag
C0.3
C0.33 Cimag
x Cimag
Cmag
Lifett = {CC - D2221 + (Cimeg D0.3
immg) )2}
10 X Lmag + Lefft
B.3 TRAINING PARAMETERS
The parameters for model training are summarized in Table 6 Specifically, the model was trained for 1000 epochs using the adamw optimizer. The initial learning was set to 0.01, and a cosine annealling learning scheduler was used adjust the learning rate from 0.01 to 0.0001 in 3000 steps. Warmup was applied and involved 30 epochs The model was evaluated on the evaluation set every epoch, and the best checkpoint (in terms of PESQ) on the evaluation set was saved. The results are reported in terms of four metrics: PESQ, BAK, SIG and OVL (Hu & Loizou 2007
C APPENDIX C: PSEUDO CODE FOR SPLIT & GLUE
The evaluation script pysepm httpps//ggthbbbcommm//m/p//////p
19

---

Under review as a conference paper at ICLR 2022
Algorithm Pseudo code for Split & Glue
Input Sequence: X R NxT; sequence of acoustic features of frames and H dimensions (wo w ... w }: window sizes of the K chunks
Input Parameter:
Input Parameter: p = {p" .pp "".""""'' K) }: padding definition for the chunks
Input Parameter:
stride in context expansion
Output Y E RXT, sequence of output features of I' frames and H dimensions
Ensure: H%K- {X) ..... X chunk chunk(X, K) for k in range (K) do
Split X to K pieces on the channel dimension
X unfold(X* w xp,,))
Context expansion by unfolding
y = + by Linear projection A for each chunk, where W [ft, w H/K]
end for
[r [K yG
Concatenate Y along channel dimension
yc GFLU((G))
Y = WBY + bb
Linear projection B glue the chunks, where WB H, K A
20


Reviews:
[{"review_id": "x0lt596G-QC", "reviewer": "Reviewer_Qsws", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- The paper is well written and clearly describes the model used and the experiments performs. The descriptions are accompanied with high-quality diagrams and specifications for hyperparameters. The motivations behind model choices are also explained. \n\nWeaknesses:\n- One main contribution of the paper, the \"split and glue\" layer, seems like a groupwise convolution layer. A groupwise convolution layer can be implemented by unfold (im2col) followed by a linear layer and concatenation. An unfold followed by a linear layer *is* a convolution. This significantly reduces the novelty and undermines the claim that the \"architecture involves simple linear transformations only\". Prior art on keyword spotting with convolutional networks is not hard to find (e.g. [0]). If my analysis is incorrect, and \"split-and-glue\" *cannot* be reduced to a groupwise convolution or at least something very similar to it, the paper can be much improved by a comparison to groupwise convolution. \n- The paper compares KWS results on Google Speech Commands and LibriWords. Although Google Speech Commands is a well known dataset for KWS, LibriWords is not commonly used and as far as I can tell has only ever been evaluated with once in a single paper that proposed this dataset. I would focus on Google Speech Command results, as that dataset has been widely studied and benchmarked against. Although extra datasets do not *reduce* the quality of the paper, evaluating on LibriWords does not add much to the paper.\n- A key part of the claim is that speech-MLP is \"SOTA\" on KWS. \"It can be observed that the speech-MLP-S and speech-MLP-L outperform all the benchmarks for all task\". However, upon cursory examination of recent literature, this claim seems dubious. Matchboxnet ([1]) achieves 97.37% on V2-35 with 140k parameters. Keyword Transformer (KWT) [2] achieves 97.51%. Note that KWT is even *cited* and compared against, but the result from KWT that the authors choose to cite is KWT1 (which speech-MLP outperforms) and not KWT2 or KWT3 (which outperform speech-MLP). (Arguably, this is because KWT2 and KWT3 have way more parameters, but this is not clear at all in the paper, and it's unclear if this is important). More recent works (Audiomer and Audio Spectrogram Transformer [4]) *significantly* outperform speech-MLP with 99.74 and 98.1%. The paper would be much improved by citing all relevant results rather than cherry-picking the results that support the claim of \"speech-MLP is SOTA\" and omitting others. \n- Much of the introduction and motivation is based around model aesthetic cleanliness, that is, around claims that one model is 'more complex' than another and that simpler models are preferable. However, model complexity is subjective and not well defined. While the authors of the paper believe that transformers are a complex model and that the prosed speech-MLP model is \"simple\", this may not match the expectations of others, for whom a single architecture commonly known and used across industry and research is simpler than a custom architecture with a variety of dataset-specific tweaks. It may be best to leave subjective judgments of complexity out and focus on other benefits of the proposed model, e.g. parameter efficiency, inference speed, etc. \n- One reason that speech-MLP may make a good architecture is parameter size and efficiency. In comparison to KWT2, which achieves similar / better results, it is much smaller. Other KWS papers will make comparisons of inference time as the model scales; this may help make the case for this architecture as well.\n- Unlike many of the cited papers, none of the values have standard deviations or confidence intervals. It would help understand the results to have these.\n- May be good to compare to more recent VoiceBank-DEMAND papers as well.\n\n[0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43969.pdf\n[1] https://arxiv.org/pdf/2004.08531.pdf\n[2] https://arxiv.org/pdf/2104.00769.pdf\n[4] https://arxiv.org/pdf/2104.01778v3.pdf", "summary_of_the_paper": "The paper proposes a new architecture for speech processing, dubbed \"speech-MLP\". Besides input layers and output task-specific layers, speech-MLP consists of linear layers, residual connections, instance or layer normalization, GELU activations, and a layer called the \"split-and-glue\" layer. The split-and-glue layer splits a [batch, time, channels] tensor into N chunks of shape [batch, time, channels / N] along the channels dimension, then applies an unfold operation to provide temporal context, then a linear layer (with a different matrix for each chunk), then concatenates the results back together. Speech-MLP is tested on keyword spotting (with 2 datasets: Google Speech Commands and LibriWords) as well as speech enhancement (voicebank + demand).", "summary_of_the_review": "The paper is well written and easy to understand with clear diagrams. However, I have some significant concerns about the split-and-glue layer (which seems to me to be a groupwise or separable convolution) and the keyword spotting comparisons (which omit results which outperform speech-MLP, then claim that speech-MLP surpasses the SOTA results). With these (and other more minor) concerns, it is hard to evaluate the novelty and impact of the paper, but these issues would be fixed with improved writing, literature review, and hyperparameter search / comparison.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ogt8LGxRc8B", "reviewer": "Reviewer_b8HX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n- novel neural network architecture by revisiting an MLP\n- simple but effective architecture with fewer parameters than transformer.\n- shows the effectiveness in two different speech processing tasks (especially the speech enhancement task only with 600K parameters looks very strong).\n\nweaknesses\n- the effectiveness of the Split & Glue layer is similar to the convolution operation with different kernel/stride sizes. \n- the method cannot be applied to a sequence-to-sequence task like ASR (but the method can be used as an encoder of seq2seq tasks).\n- although the performance is strong, the task is rather simple and limited. It does not attract a machine learning researcher in general.\n- the paper needs more surveys", "summary_of_the_paper": "This paper proposes an MLP-based neural network, which is designed for speech processing. The method is an alternative architecture to a transformer encoder and is applied to several speech processing tasks (command recognition and speech enhancement). The novel Split & Glue layer is used to capture multi-resolution speech characteristics. The method achieved state-of-the-art performance in both command recognition and speech enhancement tasks.\n\nOther comments\n- I don't think (Huang et al., 2020) is a representative work for transformer ASR. The following papers are more appropriate:\n  - Dong, Linhao, Shuang Xu, and Bo Xu. \"Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition.\" 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018.\n  - Karita, Shigeki, et al. \"A comparative study on transformer vs rnn in speech applications.\" 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019.\n- It's better to mention how it is simple with quantitative measures in the abstract and introduction (model size, computational cost, or the number of code lines).\n- the paper should refer conformer and discuss it. It becomes SOTA in many speech processing tasks now. Also, historically, many deep-learning-based speech processing methods were started from MLPs but the paper does not have enough surveys.\n- please discuss the online/streaming capabilities. This is an important function for speech processing.\n\n\n", "summary_of_the_review": "This paper proposes a novel neural network architecture based on MLP. The neural network architecture in speech processing becomes more complicated, and the proposed method can provide simple and alternative solutions. The effectiveness is also shown with command recognition, keyword search, and speech enhancement. My concern of this paper is that the application is rather simple. The method has a lot of potentials and may have more attention if they are applied to sequence-to-sequence tasks like ASR and TTS.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WbbNUTBO24A", "reviewer": "Reviewer_EFsM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I begin with the experiments because that is the positive part, at least for keyword spotting.  The performance is good, and that is in itself interesting.  This is especially true for the smaller architecture, where there are far fewer parameters.  Although the evaluation on speech enhancement produces good objective results, the measures are hardly significant.  A 0.02 improvement in PESQ is not regarded as significant, and the others are derivatives of PESQ.  Modern measures such as STOI and frequency weighted segmental SNR are missing.  The enhancement results can only be taken as preliminary, suggesting that a subjective test is worthwhile.\n\nI find the method part less persuasive, beginning with the claim that the contribution is threefold.  The combination of propose, test and demonstrate is *one* contribution.  Thereafter I have multiple difficulties:\nThe authors claim early in the manuscript that they do not expect Speech-MLP to work for speech recognition.  Given the speech recognition is *the* main application of machine learning in speech processing, Speech-MLP is not really a good name.  Pragmatically, the longer range dependencies required for speech recognition are likely to come from other architectures.\nThe authors introduce their architecture simply by describing it.  Lacking is a description of Mixer MLP and how the new architecture differs from it.  My reading of this section is that Speech-MLP *is* Mixer-MLP with some minor modifications (it is not clear what they are).  Further, where in the introductory material, the authors claim that their solution is based on domain knowledge, I do not see how the proposed architecture addresses this where Mixer MLP would not.\nThe descriptive comparison with transformers is selective; it is true that transformers discard this type of domain knowledge, but *every* other speech processing architecture does take it into account.  A case in point is the TDNN used in many solutions.\nIn general, The method section should be rewritten to explain how and why the proposal differs from Mixer-MLP, and to place it in the context of other common signal processing techniques that also take such structure into account.\nAll results should include significance tests.", "summary_of_the_paper": "The paper presents Speech-MLP, an architecture based on Mixer MLP but specific for speech signals.  The Mixer MLP architecture is argued to be appropriate for the particular structure of speech.  The architecture is compared to others on keyword spotting and speech enhancement.  In each case, the architecture outperforms other competitive solutions, often with fewer parameters.", "summary_of_the_review": "Whilst some results are persuasive, showing that the architecture can perform as well as much larger ones, the method lacks rigor and contains no clear advance over other the methods on which it is based.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "RUHSSCEfUTu", "reviewer": "Reviewer_zWqe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written, it is clearly motivated and presents impressive results with such a simple architecture. If the code becomes publicly available as the authors promised it will be very useful for the research community.\n\nThe ablation study is useful but it would be more complete if additional parameters were investigated, e.g., number of blocks. In Table 1 they are set to 4 and 10 somehow arbitrarily. In Table 3, more window sizes can be considered, e.g., 7 and 9. It is also not clear how the number of chunks impacts performance in Table 3. \n\nWhy not considering some intermediate architectures between small and large? It would be useful to report resorts for a M(edium) architecture as well.\n\nAlso have the authors considered other ways to create the chunks, e.g., across the time dimension?\n\nSome typos\nAppendix B.3 Specificaaly\nA.2.2 closing quotation marks are used as opening quotation marks\n", "summary_of_the_paper": "The paper proposes a simple architecture based on the multi-layer perceptron for extracting information from speech signals. The architecture is based on a new layer called split and glue and can capture multi-scale local temporal dependecies. It is evaluated on two different problems, keyword spotting and speech enhancement and achieves state-of-the-art performance.", "summary_of_the_review": "Overall, this is an interesting study, the paper is clearly written and motivated and presents a novel architecture which leads to state-of-the-art results for keyword spotting and speech enhancement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M6sDPl8vA4Q", "reviewer": "Reviewer_cq7P", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The proposed model is relatively simple and easy to implement. It leverages some of the important characteristics of human speech such as temporal invariance, frequency asymmetry and short-term stationarity. The model is modular and can be easily integrated into any existing pipeline to process the features in a desired manner. Finally, it shows improvements on multiple tasks and across multiple datasets. \n\n\nWeaknesses: This model appears to be convolution in disguise. The splitting operation is akin to breaking down the signal into multiple frequency bands and learning a convolution kernel for each band separately, i.e, separable convolutions. The unfolding operation or context expansion is where the model resembles the convolutional structure the most. It is similar to creating a circulant matrix and applying linear transformation to the expanded signal. The linear transformation operator is analogous to the convolutional kernel. Therefore, the authors, unknowingly, are proposing a convolutional neural network with residual connections and labelling it as 'split and glue' model. Further, convolutions can actively take advantage of temporal invariance or short-term stationarity of speech signal which seems to be one of the main drivers of performance gains in this model. Finally, the authors have not clarified if the data augmentation was done for the baseline models or not. The augmentation procedure can have a huge impact on the generalization of neural networks. \n\nThe experiments on keyword spotting use cross entropy loss for backpropagation. My suggestion to the authors would be to use the triplet loss for training as it outperforms the entropy based model in baseline comparisons. Additionally, the baseline methods in the speech enhancement task are mostly generative models which are perhaps not trained in a supervised setting. Therefore, comparison with some recently proposed supervised methods for speech enhancement would provide a good sense of the performance improvements. In the ablation study, it would be valuable for the speech and machine learning community to see how the proposed split and glue model performs without the outer residual connection in each block. While the paper compares the proposed approach to a convolutional model (Res-15), the presence of residual connections (inner+outer) and normalization (layer/instance) adds a lot of confounders in the analysis. Finally, a statistical test or error bars on the speech enhancement metrics will help validate the claims made by the authors. ", "summary_of_the_paper": "The authors have proposed a new general-purpose MLP architecture for speech processing and learning speech representation for tasks such as keyword spotting and speech enhancement. The proposed model processes the spectral representation of speech in multiple frequency/channel bands ('split' operations). The unfolding operation expands the context of each banded signal and applies linear transformations to it. These transformations are not shared across the chunks/bands which allows the model to learn chunk-wise relevant representations. Further, the unfolding procedure (context expansion) provides the benefit of learning segmental and supra-segmental properties of speech. The transformed outputs are concatenated together which is referred to as the 'glue' operation. Finally, the learned representations are passed into task-specific layers for generating output predictions. There are two residual connections in each split and glue block which ensures that there is a proper flow of gradients during the backpropagation to train the model. \n\nThe authors demonstrate the performance of the proposed mlp architecture on two problems, namely, keyword spotting, and speech enhancement. The model shows improvement over the state-of-the-art baseline techniques on multiple datasets (Google V2 and Libriwords) for keyword spotting. The ablation studies show how the choice of multiple linear operator kernels in 'split and glue' perform better than no split model. The speech enhancement experiment has been carried out on VoiceBank+Demand dataset and the authors show improvement by proposed technique across multiple metrics and baseline models. Therefore, the main contribution of this paper is to show that mlp architectures that are heuristically driven and are derived from domain knowledge can outperform state-of-the-art models for various prediction tasks. ", "summary_of_the_review": "The proposed architecture is similar to a convolutional neural network. The context expansion and chunk-wise processing is analogous to the concept of separable convolution. The experiments however, show the effectiveness of the model on keyword spotting and speech enhancement tasks. The proposed approach appears to take advantage of the structure of speech signals, specifically, temporal continuity. The authors have also conducted ablation studies to show how splitting the model into multiple frequency bands has desirable effects. It would be interesting to see how the 'split and glue' output performs on the relevant tasks without any residual connection. Further, comparison with supervised methods and error bars on the evaluation metric for speech enhancement will provide additional insight into the proposed technique.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an MLP-based neural network specifically designed for speech processing. The proposed Split & Glue layer is used to capture multi-resolution speech characteristics. The method achieved better performance in both command recognition and speech enhancement tasks.

Two major concerns raised by the reviewers:
The proposed split & glue layer is similar to convolution. Although the authors revised the paper with more clarification on the differences, the op is equivalent to frame-wise convolution which has been explored in speech literature. This limits the novelty of the paper.
The experimental justifications are relatively simple and limited. On the voice command and speech enhancement tasks presented in the paper, stronger and better baselines would be more convincing to justify the benefit of the proposed method. Moreover, testing on large scale ASR tasks instead of the relatively simple voice command task would be more convincing.  

The decision is mainly based on the limited novelty and experimental justification.
Decision: Reject