Introduction
Deep convolutional neural networks (CNNs) with 2D convolutions and small kernels BIBREF1, have achieved state-of-the-art results for several speech recognition tasks BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6. The accuracy of those models grows with their complexity, leading to redundant latent representations. Several approaches have been proposed in the literature to reduce this redundancy BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, and therefore to improve their efficiency.
Octave convolutional layers BIBREF0 address the problem of spatial redundancy in feature maps by learning feature representations at high and low resolutions. The low resolution processing path increases the size of the receptive field in the original input space, which is a plausible explanation of the improved performance for image classification. We extend the octave convolution concept to multi-scale octave convolutional layers, which include lower resolution feature maps with a higher compression rate (reduction by more than one octave), and the use of more than two feature map tensor groups in order to be learn representations at multiple scales.
Multi-scale processing have been previously proposed for a variety of speech recognition tasks BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16. In deep CNN acoustic models, some of the feature maps may need to represent information which varies at a lower rate, such as the characteristics of the speaker or background noise, compared to the information necessary for phonetic discrimination. Spatial average pooling in a low resolution group of feature maps can be interpreted as a form of low-pass filtering, providing smoothed representations of the observed data, potentially leading to improved performance.
We investigate the use of multi-scale octave convolutional layers for robust speech recognition, and attempt to shed more light on the explainability of the models by evaluating the robustness of the learned representations using an affine transformation loss to measure the similarity between clean and noisy encodings.
Multi-scale octave convolutions
An octave convolutional layer BIBREF0 factorizes the output feature maps of a convolutional layer into two groups. The resolution of the low-frequency feature maps is reduced by an octave – height and width dimensions are divided by 2. In this work, we explore spatial reduction by up to 3 octaves – dividing by $2^t$, where $t=1,2,3$ – and for up to 4 groups. We refer to such a layer as a multi-octave convolutional (MultiOctConv) layer, and an example with three groups and reductions of one and two octaves is depicted in Fig. FIGREF1.
In a vanilla CNN the convolutions have the same spatial resolution throughout the network. An octave convolutional (OctConv) layer is divided into high- and low-frequency feature maps and a multi-octave convolutional (MultiOctConv) layer has feature maps reduced by multiple octaves. Let the input feature tensor be $X \in \mathbb {R}^{c_{in} \times h \times w}$, where $c_{in}$ denotes the number of input channels and $h$ and $w$ correspond to the spatial dimensions. In a MultiOctConv layer working at 3 resolutions, $X$ is factorized along the channel dimension into $X = \lbrace X^1, X^2, X^3\rbrace $. The first tensor group tensor, $X^1$, is a representation at the same spatial scale as $X$. The spatial dimensions of the second and third group tensors, $X^2$ and $X^3$, are reduced by one and two octaves respectively.
The dimensions of the input tensors $X^1$, $X^2$ and $X^3$ are described in Fig. FIGREF1. The fraction of the channels for each group is denoted with $\alpha _{n} \in [0,1]$, where $\sum _{n=1}^{N} \alpha _{n} = 1$ for $N$ resolution groups in the MultiOctConv layer. For simplicity, we use the same $\alpha _{n}$ for input and output representations within the same scale group. Similarly, the output tensors are also factorized into $Y = \lbrace Y^1, Y^2, Y^3\rbrace $. Their dimensions are analogous to the dimensions of the input tensors and are described in Fig. FIGREF1. To compute $Y^1$, $Y^2$ and $Y^3$ we operate directly on the factorized input tensors $X^1$, $X^2$ and $X^3$. Inter-frequency information update is implemented as a sum of feature maps from different resolution groups. To be able to sum those representations for a desired output scale, the spatial dimensions of the input tensors must be the same. For this reason, two operations are employed: spatial average pooling pool($X, p$) and bilinear interpolation upsample($X, u$), where $p$ is the kernel size and stride for the the 2D pooling layer and $u$ is the upsampling factor. The output MultiOctConv representations are therefore computed as
where $f(.)$ is the convolution function and $W^{n_{in}\rightarrow {n_{out}}}\in \mathbb {R}^{c_{in} \times k \times k \times c_{out}}$ is the convolution filter for a $k \times k$ kernel. We call the information update “intra-frequency” when $n_{in} = n_{out}$, and “inter-frequency” when $n_{in} \ne n_{out}$. It is important to note that the convolution $f(.)$ operates on the tensors compressed with average pooling and on the tensors before upsampling, making the design more efficient. The number of parameters in the MultiOctConv layer is the same as in a vanilla convolutional layer.
Multi-scale octave convolutions ::: Robustness of learned representations
To evaluate the robustness of learned representations, we compare the projections of clean and noisy Aurora-4 samples. The similarity between them is measured using the mean squared error (MSE) loss of an affine projection $y$ of $N$ clean to noisy samples (Eq. DISPLAY_FORM3), to take into account permutations of hidden representations and to ensure invariance of the metric to affine transformations of the encodings. The number of units in layer $y$ and the dimensionality $D$ of $\mathbf {x}_{h}$ is 1024.
We use the Aurora-4 test sets and compare clean encodings $\mathbf {x}_{h,clean}$ with noisy encodings $\mathbf {x}_{h,noise}$, obtained as the activations from the last convolutional layer with a forward pass through a trained model. Both hidden representations were obtained for CNN and octave CNN (OctCNN) models in order to compare representations between the models. Also, for intra-model comparison, we evaluate the loss with the encodings from high and low-resolution groups (paths $Y^{1\rightarrow 1}$ and $Y^{2\rightarrow 1}$). This analysis aims to evaluate if the low-resolution groups for noisy samples are indeed more similar to the clean ones than the high-resolution encodings, suggesting more robust representations. We optimize the parameters of $y$ with back-propagation using a fixed number of 3 epochs and we report the validation loss for Aurora-4 test sets.
Experimental setup
Aurora-4 BIBREF17: We evaluate our models on the simulated multi-condition Aurora-4 dataset, consisting of $\sim $15h of audio for training and $\sim $9h for testing. The test set is divided into 4 subsets: A, B, C, and D. Subset A contains clean-condition recordings, subset B has 6 noise types added to the recordings (car, babble, restaurant, street, airport, train), subset C is recorded with a mismatched microphone, and subset D is recorded with a mismatched microphone and with noise added. In our experiments, we use multi-condition GMM-HMM forced alignments as targets for CNN training. The number of CD states for Aurora-4 is 3422.
AMI BIBREF18: AMI contains $\sim $100h of meeting recordings, captured by an independent headset microphone (IHM), single distant microphone (SDM), and multiple distant microphones (MDM), where the mics are combined using the BeamformIt BIBREF19 toolkit. We train our models using the MDM data and evaluate the models for all 3 types of recordings to analyze the effect of mismatched training/testing conditions. We use the suggested train/dev/eval data split BIBREF20, and we evaluate the models on both dev and eval sets. The number of CD states for AMI is 3984.
Features: In our experiments, we use 40-dimension mel-scaled filterbank (FBANK) features with {-5,..,5} context for splicing, resulting in a $40\times 11$ input feature map.
Models: Our baseline CNN model BIBREF21 consists of 15 convolutional and one fully-connected layer. We use $3\times 3$ kernels throughout the network. We start with 64 output channels in the first layer and double them after 3 and 9 layers. We use batch normalization in every convolutional layer, and ReLU afterwards (unless a reverse order is noted). The initial learning rate is 0.001. We use early stopping for training.
Results
We present our results in terms of accuracy and robustness on Aurora-4 and AMI, as well as in terms of the computational cost, which is calculated as the number of multiply-accumulate operations (MACCs) performed for a single input feature map. The cost reduction when using octave convolutions stems from reduced dimensions $c_{in}$, $c_{out}$, $h$, and $w$ compared to a vanilla convolutional layer.
Aurora-4: Results for Aurora-4 are presented in Table TABREF4. We replace vanilla convolutional layers of our baseline model (CNN) with OctConv and MultiOctConv layers. We first evaluate which layers can be replaced and find that all but the first layer, operating directly on the input representation, should be replaced for the best performance. This approach (L2-L15) is also the least costly. Reducing the ratio of low-resolution representations to 0.125 improves the WER for the mismatched microphone scenario C, but not for all test conditions. Applying batch normalization after ReLU is beneficial for test set C and D. For OctCNN models, the WER for test set D dropped by $\sim 0.4\%$ with a compression by one octave, and by another $\sim 0.4\%$ with a reversed batch normalization and ReLU order.
The biggest differences between the MultiOctCNN models can be observed for test set D. The models with the lowest WERs are the ones with a spatial reduction by 2 or 3 octaves, and with 2 or 3 groups. This indicates that multi-scale octave convolutions seem to be an effective, as well as an efficient design for processing speech with background noise and channel mismatch. For MultiOctCNNs, batch normalization after ReLU also gives a performance boost for test set D, with a drop to $13.57\%$.
To further evaluate the robustness of the latent representations we measured the MSE between the (projected) representations, described above (Fig. FIGREF5). The loss for the activations at the output of Conv15 ("all") is similar for CNN and OctCNN models for test sets B and C, but lower for test set D for OctCNN, indicating that the learned representations are more robust, contributing to lower WERs. As expected, within-model comparison of the loss show that the representations at low resolution are more similar to the clean encodings from test set A than the ones at high resolution. We believe that this effect improves the robustness of latent representations and results in a decreased WER.
AMI: Results for AMI are presented in Table TABREF6. In contrast to the Aurora-4 findings, better performance was achieved with an all OctCNN model (L1-L15). This is an interesting finding, and we believe that the multi-scale processing of the input feature space is beneficial for AMI MDM because of the reverberation in the data. The reverbarated input time$\times $freq representation can be viewed as a spatially redundant one, therefore the OctConv layer applied to the input representation is effective. Unfortunately, the only MultiOctConv model superior to the baseline CNN is the one with 3 groups with a spatial reduction by 1 and 2 octaves. This result indicates that the spatial redundancy for this architecture for AMI MDM is not degrading the performance. However, in terms of the computational cost, we can reduce the #MACCs by a factor of 1.8 with only a small WER increase for a model with 4 resolution groups.
Conclusions
We have presented multi-scale octave CNN models for robust and efficient speech recognition. We build on Chen et al BIBREF0, applying the method to robust ASR and extending it to multiple resolution groups with a spatial reduction of more than one octave. Our experiments confirm that multi-scale processing of the hidden representations is not only more computationally efficient, but also improves the recognition. Similarity measures between clean and noisy encodings indicates that multi-scale processing in a deep CNN acoustic model improves the robustness of learned representations, especially in the additive noise and mismatched microphone scenario. The gain of the octave convolutions was also observed for AMI MDM data with significant reverberation, when applied to the input feature space. However, the model performance for AMI MDM was not improved with multi-octave convolutions. More careful tuning of the $\alpha $ hyperparameter could improve the results, as it controls the ratio of multi-scale feature maps in the model, enabling both learning of fine-grained representations preserving the details necessary for phonetic discrimination, as well as smoothed, more invariant representations improving the robustness of the model. It would also be possible to set $\alpha $ layer-by-layer to enable the fractions of channels at different resolutions to vary according to the depth of the representation.
We proposed a single projection layer MSE loss to measure the affine relationship of clean and noisy hidden representations. With this approach, we evaluated the robustness of the encodings and improved the explainability of our models. More thorough analysis of the representations learned is an interesting future direction. We confirmed that the noisy lower resolution representations are more similar to the clean counterparts than high resolution representations, and thus are more robust. However, we did not investigate the reason for the increased similarity, leaving future work to ascertain if the lower resolution group corresponds to better speaker or noise characteristics, or more invariant phonetic representations.

Question: Is model compared against state of the art models on these datasets?
Answer: Yes

Question: How is octave convolution concept extended to multiple resolutions and octaves?
Answer: The resolution of the low-frequency feature maps is reduced by an octave – height and width dimensions are divided by 2. In this work, we explore spatial reduction by up to 3 octaves – dividing by $2^t$, where $t=1,2,3$ – and for up to 4 groups. We refer to such a layer as a multi-octave convolutional (MultiOctConv) layer,