Understanding Adaptive, Multiscale Temporal Integration In Deep Speech Recognition Systems

∗ These authors contributed equally . 2 Code available at : https : //github.com/naplab/PyTCI 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . A central challenge of representing natural signals , such as speech and music , is that they are structured across many different timescales ( Chomsky and Halle , 1968 ; Lerdahl and Jackendoff , 1985 ; Hickok and Poeppel , 2007 ) . Speech , for instance , is composed of phonemes , syllables , and words spanning tens to hundreds of milliseconds , and the duration of these structures varies widely due to a myriad of factors ( e.g. , talker , speaking rate , prosody , etc . ) ( Figure 1 ) . Intelligent systems , whether biological or artificial , must learn to flexibly integrate across these different timescales to derive meaning from natural signals . DNNs have proven remarkably effective at learning complex temporal structures from natural signals , such as speech , and the representations learned by these networks show non-trivial similarities to those present in the brain ( Kell et al. , 2018 ) . However , relatively little is known about how deep speech recognition systems flexibly integrate across multiple timescales . Here , we address this gap by using a recently developed method for studying temporal integration in biological neural systems – the temporal context invariance ( TCI ) paradigm ( NormanHaignere et al. , 2020 ) – to probe how a widely used deep speech recognition system ( DeepSpeech2 ) learns to integrate across multiple timescales . We study temporal integration by measuring how training alters the integration window of DNN units . Integration windows are defined as the time window when stimuli alter the neural response and outside of which stimuli have little effect ( Theunissen and Miller , 1995 ; Hasson et al. , 2008 ) . Because this definition is simple and general , integration windows can be used to measure and compare the analysis timescale of responses from different models , stimuli , and modalities . While convolutional neural networks ( CNNs ) have a well-defined upper bound on their integration window , determined by the size of the temporal kernels , the effective integration window , when stimuli empirically alter the response , can differ substantially from this upper bound ( Luo et al. , 2016 ) and change substantially with training , as we show . Recurrent neural networks ( RNNs ) can in principle integrate over arbitrarily long time windows ( Hochreiter and Schmidhuber , 1997 ; Sussillo and Barak , 2013 ) , but in practice , RNNs also show time-limited effective integration windows within which stimuli have a much greater impact on the measured response ( Mahto et al. , 2020 ; Chien and Honey , 2020 ) . We have recently designed a method to estimate the effective integration window of an arbitrary response by presenting segments of natural stimuli ( here speech ) in two different random orders such that each segment occurs in two different contexts ( i.e. , is surrounded by different stimuli ) ( Figure 2a ) ( Norman-Haignere et al. , 2020 ) . If the integration window is less than the segment duration , there will be a moment when the response is the same across the two contexts . We can thus vary the segment duration in order to find the smallest segment duration outside of which stimuli have little effect on the response . The random ordering of segments makes it possible to use a simple correlation measure to quantify at each moment in time the degree to which the response is context invariant ( Figure 2b ) , as detailed below . The goal of this paper is to introduce this paradigm to the machine learning community and to use this paradigm to understand how deep speech recognition systems learn to flexibly integrate across multiple timescales in speech . 2 Relationship with prior work . Understanding latent representations of DNNs is a highly active area of research . A variety of feature visualization techniques have been developed to understand image-trained networks ( Zeiler and Fergus , 2014 ; Mahendran and Vedaldi , 2015 ; Yosinski et al. , 2015 ) with less work focused on understanding speech-trained networks ( Huang et al. , 2015 ; Krug and Stober , 2019 ; Beguš and Zhou , 2021 ; Keshishian et al. , 2020 ) . Many visualization methods are based upon calculating the gradient of the network ’ s response with respect to the input ( Springenberg et al. , 2014 ; Bach et al. , 2015 ; Selvaraju et al. , 2016 ; Shrikumar et al. , 2017 ; Sundararajan et al. , 2017 ; Samek et al. , 2021 ) . While for a linear system , the temporal extent of the gradient is equivalent to the integration window , for a nonlinear system , the link between the local gradient and the overall computation is unclear ( Balduzzi et al. , 2017 ) , and as a consequence , there is a need for empirical methods that can estimate the effective integration window of highly nonlinear systems ( Norman-Haignere et al. , 2020 ; Chien and Honey , 2020 ) . Probing tasks have been used to examine the representation of particular structures , such as phonemes , across different layers of speech recognition networks ( Belinkov and Glass , 2017 ; Nagamine et al. , 2015 ; Nagamine and Mesgarani , 2017 ) , and a recent study examined the geometry of how invariant feature representations evolve over time ( Stephenson et al. , 2020 ) . However , to our knowledge , little is known about how speech-trained networks learn to integrate across time , despite the centrality of temporal integration to the task of speech recognition . Neuroscience has produced a variety of experimental methods for studying sensory timescales . Scrambling methods have been used to examine selectivity for intact temporal structure at different timescales ( Hasson et al. , 2008 ; Overath et al. , 2015 ) . Chien and Honey ( 2020 ) recently introduced an experimental method that estimates the delay needed for a response to become context-invariant after a stimulus change , and they used this method to analyze neural language models ( Chien and Honey , 2020 ) . However , this method does not directly measure the time window in the stimulus that is needed to achieve an invariant response , which requires measuring responses to segments of differing durations . Moreover , speech recognition presents many unique challenges that are not present in language modeling . For example , a key contribution of our paper is showing that there is a transition from fixed , time-yoked integration to adaptive , structure-yoked integration in deep speech recognition models , a distinction that is not meaningful in language models where the input is inherently structure-based ( i.e. , words ) . 3 Temporal context invariance ( TCI ) paradigm . 3.1 Stimuli . The input to our analysis is a set of sound segments of varying duration . We tested durations ranging from 20 milliseconds and 2.48 seconds in pseudo-logarithmic steps ( 20 , 40 , 60 , 80 , 100 , 120 , 140 , 160 , 200 , 240 , 280 , 340 , 400 , 480 , 580 , 700 , 840 , 1000 , 1200 , 1440 , 1720 , 2060 , 2480 ms ) . Sound segments were excerpted from the LibriSpeech corpus ’ dev-clean and test-clean sets . We attempted to choose segments that were composed of words spoken at a moderate rate and which also had a moderate amount of natural variation in their duration , such that stretching or compressing the word would not place the word outside of its typical duration ( stretching and compression analyses described below ) . To this end , we first selected all words spoken at least 100 times in the corpus . For each word , we computed a histogram of durations , and we selected all words where the difference between the 5th and 95th percentile duration was at least a factor of 2 ( see Figure 1c for a histogram of this durational variability metric across words ) . For each word , we then randomly selected up to 100 of its utterances that fell within the central 20 % of its duration distribution . This resulted in a pool of 6,184 utterances across 1,095 unique words . Finally , for each utterance , we excerpted 23 segments , one per duration , with the middle of the word at the center of the segment ( if the segment duration was shorter than the word , we excerpted just the middle portion of the word ) . This selection procedure resulted in 6,184 unique segments per segment duration . We note , that while we used a speech-specific recipe to extract segments , having in mind the types of experiments we would conduct , this is not necessary or intrinsic to the TCI method . The segment extraction recipe can change depending on the nature of the stimuli and the intended experiments . Time alignments for phonemes and words of the LibriSpeech dataset were extracted from ( Lugosch et al. , 2019 ) , which used the Montreal Forced Aligner to align the transcripts to sounds ( McAuliffe et al. , 2017 ) . Syllables were identified based on the extracted phonemes using the syllabify python library ( Gorman , 2013 ) . All of the segments from a given duration were randomly ordered and then concatenated with crossfading at the boundary to minimize transition artifacts ( 20ms half-Hanning window ) . We used two different random orderings of the segments such that each segment occurred in two different contexts ( Fig 2a ) . Due to limited memory , we divided the randomly ordered segments into 48-second stimuli with 16-seconds of overlap between consecutive stimuli . The responses to the first and last 8 seconds of each stimulus were discarded to avoid padding artifacts due to batching . 3.2 DeepSpeech2 model . We applied our TCI paradigm to measure integration windows from a DeepSpeech2 model ( Amodei et al. , 2016 ) ( trained for 20 epochs ) . We used a publicly available implementation of this model ( Naren , 2019 ) . Our DeepSpeech2 model consists of two 2D convolutional layers ( Conv1/2 ) , five bi-LSTM layers ( LSTM1/2/3/4/5 ) , and a softmax readout that is trained to predict graphemes using a CTC loss ( Graves et al. , 2006 ) . All layers except the readout included batch normalization . The model reached a word error rate ( WER ) of 7.6 % on the LibriSpeech test-clean set after training for 20 epochs . We analyzed logarithmically transformed probabilities for the softmax readout ( to avoid the high sparsity of probabilities ) . We compared results from our trained model with a randomly initialized model to assess effects of training . All models were implemented in PyTorch ( Paszke et al. , 2019 ) and trained using PyTorch Lightning ( Falcon , 2019 ) on the training set of the LibriSpeech corpus ( Panayotov et al. , 2015 ) . We used the CTC loss ( Graves et al. , 2006 ) , the Adam optimizer ( learning rate : 1.5e−4 , weight decay : 1e−5 ) ( Kingma and Ba , 2014 ) , and a batch size of 64 . To improve model generalization , we augmented the stimuli by adding background noise , reverberation , and frequency masking . Augmentations were performed using the Sound eXchange ( SoX ) backend of the audio library for PyTorch ( torchaudio ) . Training and inference of all models were performed on NVIDIA A40 GPUs ( one per training/inference ) at the internal cluster at the Zuckerman Institute of Columbia University . Each epoch of training took approximately 2.5 hours for each model , for a total of 110 epochs across all models . For each experiment , we ran inference across three time-stretches and 23 segment durations , for an average of 22 hours per experiment . Inference was substantially faster for the convolutional model .

Summary: This paper addressed the topic of identifying the integration window of hidden layers within ASRs. Specifically, the work applies temporal context invariance proposed in “Multiscale integration organizes hierarchical computation in human auditory cortex“ to study the integration window of each layer in ASRs. The integration window is defined as the latency for which neurons provide saturated responses to a certain signal. In other words, the time a unit needs for identifying the underlying temporal signals is its integration window. The approach infers integration window through comparing the correlation between the response to the same signal segment within different utterances, where the window length is the time when the correlation exceeds a predefined threshold (0.75 in the paper). The paper also studies the effect of stretching and shrinking the audio, and showed that upper layers’ integration windows are more proportional to time scaling while lower layers are more invariant. The time-variance study provides an insight on whether the approach is effective on inferring the integration window of each layer: if the approach or the underlying unit simply reflect the time scale of the signal then it should scale proportionally with the input, while if the approach or the unit only reflect a fixed time response then they should be completely invariance.  The time-variance study shows that all layers within an untrained ASR are invariant to the time scaling, where the trained ASR exhibits different variance at different layers.