Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, the authors conduct analyses to understand how ASR models -- specifically, DeepSpeech2 -- integrates temporal information in various layers of the model. The analysis is based on the temporal context invariance (TCI) paradigm proposed for biological neural systems which is based on studying the output from a neuron from segments of different lengths in two different contexts; correlated outputs indicate sensitivity to the particular input and the length of the input segments indicate how long the segments need to be to produce relatively invariant outputs. The analyses indicate that the networks become increasingly sensitive to longer time scales and that earlier layers in the network are more responsive to static time windows whereas deeper layers are sensitive to the duration of the structures to which it responds.
main_review: Overall, I found this paper to be really interesting and intriguing. Generally speaking, the paper is excellently presented. The work is extremely well motivated, with detailed discussion of related work. The experiment in Section 4 was particularly interesting. However, I do have a number of comments that I would like the authors to address:

- **Dataset creation and SAR Matrix Description**: While I generally understood the gist of the argument, one aspect of the paper that I found hard to follow, was the fact that the paper presents all ideas in prose rather than through mathematical notation which makes the work somewhat imprecise and ambiguous. In particular, Section 3.3 which describes the creation of the SAR matrix is somewhat vague (e.g., “. First, we organize the response timecourses to all segments of a given duration as a matrix, which we refer to as the segment-aligned response (SAR) matrix (Figure 2b). Each row of the SAR matrix contains the response timecourse surrounding a single segment, aligned to segment onset. Different rows thus correspond to different segments and different columns correspond to different lags relative to segment onset.”) I wasn’t sure what the authors meant by “timecourses”. I think this section would be much clearer and easier to understand if this was described with precise mathematical notation. Currently, this is described in Section 3.3 and the caption to Figure 2 which together take up almost 2 pages of the paper. 

- **Specific choice of stimuli**: I think that Section 3.1 which describes how the dataset used to build the SAR matrix was constructed could be re-written to be a bit more precise. In particular, something I was curious about was what fraction of the words chosen for the analysis involve common versus uncommon words. For example, I could imagine that the behavior is very different for common words such as “a”, “an”, “the”, etc. versus more uncommon words. I would be curious to see some analysis along these lines.

- **Bidirectional Recurrent Layers**: In the caption to figure 2, the authors state: “If the segment duration is longer than the integration window (top panel), there will be a moment when the window is fully contained within each segment. As a consequence, the response at that moment will by definition be unaffected by the surrounding context segments.” Given that the model uses bidirectional recurrent layers, is it not still possible for the model to be affected by surrounding context segments? Could the authors please clarify if this is an assumption in the TCI paradigm, or if this still holds in some way.

- **Limitations**: A couple of limitations of this work is that the analyses are all limited to a particular model architecture. It would be interesting to conduct similar analyses on a variety of different architectures (e.g., only LSTMs, only Feedforward layers, etc.), and to vary the output units (e.g., sentence-pieces/words instead of graphemes) to see if the observations are similar different in these cases.

limitations_and_societal_impact: Apart from the limitations mentioned above, I do not see any negative societal impacts of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes an method of analyzing individual neurons in a neural network based on monitoring the output of a particular neuron while changing the size of input segments at the center and their context. The concept of integration window is introduced and defined as the size of the center segment that achieves a maximum correlation of 0.75 while changing the context. The paper uses DeepSpeech2 as a model to demonstrate the approach, and finds that the integration window significantly increases as we move deeper into the network and further away from the input. Further experiments are done to study the impact of stretching and compression the input spectrogram on the integration windows. The results show that the integration windows "adapt" better at the higher layers, meaning that stretching the input also stretches the integration windows, while the lower layers adapt less.
main_review: I am giving the paper a score 7, because the approach is novel and the results of the analyzing DeepSpeech2 themselves are a significant contribution.

# Novelty

The approach is novel, though there are a few closely related concepts not mentioned in the prior work. Changing the context while measuring the output in the limit is in fact measuring the gradient. A few gradient-based approach are listed below.

Deep inside convolutional networks: visualising image classification models
Karen Simonyan, Andrea Vedaldi, Andrew Zisserman
Workshop at ICLR, 2014

Striving for simplicity: the all convolution
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, Martin Riedmiller
arXiv:1412.6806

Grad-CAM: Why did you say that?
Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, Dhruv Batra
ICCV, 2017


# Approach

The major assumption of the approach is that analyzing the response of segments with arbitrary context concatenated is meaningful. The network is trained on continuous speech without the artifacts of concatenation, so in principle the concatenated input segments are considered out of domain and the results could really be anything. However, from the experiments, it seems that we have enough evidence to believe that the results are reliable.

The same applies to the stretching and compression experiments. I believe the input to the network is in the log magnitude domain, and a simple interpolation might introduce artifacts that are simply not natural and can be considered out of domain. Again, the experiments seem to suggest that the results are reliable.

Another relative strict constraint not considered in the approach is that fact that first two convolution layers have finite receptive fields. In other words, the integration windows are capped by the hard constraints.

As mentioned before, the approach is reminiscent to the gradient-based approaches. It would be great to use another set of analysis approaches to at least see if the findings are consistent.


# Presentation

The paper is a nice read.

I find the caption in Figure 2 a bit too long, and some of them are actually repeated in the main text.

I also find the text description in 3.3 extremely hard to follow. This is a good place to use mathematical notations to precisely define what is being computed to improve understanding.
limitations_and_societal_impact: The authors have addressed the limitation of the approach and its potential social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors investigate the DeepSpeech2 speech recognition model using temporal context invariance (TCI), which was recently introduced and used to study temporal integration in biological systems. The analysis shows that the "integration windows" for which the intermediate outputs of DeepSpeech2 are approximately independent of additional context are organized hierarchically, with lower layers integrating across shorter, fixed, time windows, and progressively higher layers integrating over longer timescales, which they show are progressively more governed by the structural evolution of speech rather than absolute time.

main_review: Strengths:

- Solid analysis of the temporal response properties of a well known speech recognition model.

Limitations:
- While interesting, the results are as expected, making them perhaps less significant. Also, DeepSpeech2 is an older speech model, and the findings fall short of identifying shortcomings or limitations of the model, or deep learning more broadly, to inform future research.
- The TCI analysis is applied only to one model, leaving contrastive analysis across architectures and applications for future work. 
- I would have really liked to see the authors open source a TCI analysis framework for ML researchers to analyze their models with...

Current Assessment:

- A solid analysis of the DeepSpeech2 model with the TCI framework. Lower significance. 

Post-rebuttal:

Thank you to the authors for their detailed responses. Based on the responses to the reviewers' questions regarding the analysis, and the clarification regarding the release of a TCI analysis tool, I have increased my score by 1 to 6. The rebuttal includes several promissory updates to the paper, including clearer algorithmic descriptions, and a contrastive analysis across architectures, which I expect will further strengthen the paper (the expected result being a "good" paper (7)).
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper addressed the topic of identifying the integration window of hidden layers within ASRs. Specifically, the work applies temporal context invariance proposed in “Multiscale integration organizes hierarchical computation in human auditory cortex“ to study the integration window of each layer in ASRs.
The integration window is defined as the latency for which neurons provide saturated responses to a certain signal. In other words, the time a unit needs for identifying the underlying temporal signals is its integration window.
The approach infers integration window through comparing the correlation between the response to the same signal segment within different utterances, where the window length is the time when the correlation exceeds a predefined threshold (0.75 in the paper).
The paper also studies the effect of stretching and shrinking the audio, and showed that upper layers’ integration windows are more proportional to time scaling while lower layers are more invariant.
The time-variance study provides an insight on whether the approach is effective on inferring the integration window of each layer: if the approach or the underlying unit simply reflect the time scale of the signal then it should scale proportionally with the input, while if the approach or the unit only reflect a fixed time response then they should be completely invariance. 
The time-variance study shows that all layers within an untrained ASR are invariant to the time scaling, where the trained ASR exhibits different variance at different layers.
main_review: This paper provides a mechanism to help understand the temporal window of underlying layers of ASRs, which allows gaining insights about ASRs’ behaviors and are quite meaningful to the speech community. The time-variance study further reveals each layer’s temporal behavior and proves the effectiveness of the approach. One small drawback is that the criterion for preparing the input signals makes it less flexible and harder to scale. On the other hand, given the access to the semantic of the input signal, a further analysis on how semantics affect integration windows can provide deeper understanding on the ASR behavior.
Related works for identifying the width of the receptive field in artificial neural networks should be discussed, for example like “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks”.
limitations_and_societal_impact: Authors have reasonably discussed their limitations.
Authors mentioned exploring how architecture can affect the analysis as a future work; it will be interesting to include self-attention-based components like Transformer/Conformer and compare how they differ from RNNs, in particular that Transformer/Conformer allows limiting the self-attention window and thus can exhibit distinct behaviors than RNNs.
review_title: null

Meta-review: In this paper the authors use temporal context invariance (TCI) to understand temporal integration in black-box deep neural networks. The study is carried out using the DeepSpeech 2 model. The authors find that integratoin windows are specialized to different time scales in different layers,  the higher the layers the more sensitive to longer time scales.  In addition, lower layers are more sensitive to static windows while higher layers are more adaptive to duration of structures.  All reviewers find the work interesting and novel in methodology.  The hierarchical structure and pattern of response revealed in the experiments, despite in line with some previously observations/speculations, are still quite interesting and intriguing.  The method investigated in the paper also can potentially provide a new tool to the community to gain insights into the black-box machine learning models.  I would recommend accept.   The authors should address the questions raised by the reviewers in the revision. If possible, please also include new experimental results on other model architectures to make the paper stronger. 
Decision: Accept (Poster)