Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 1. Motivation: The paper focuses on understanding the behavior of networks with attention and investigating how encoder-decoder networks solve different sequence-to-sequence tasks. To this end, the authors hypothesize that representations (i.e., hidden states) can be decomposed into "temporal" and "input" components.

2. Approach: In detail, the author presents a decomposition of the hidden states of encoder/decoder into temporal (independent of input) and input-driven (independent of sequence position) parts, where the former is used to explain the temporal behavior of the network and the latter is used to describe the input behavior. 

3. Experiments: This paper conducts extensive experimental studies on several common sequence-to-sequence frameworks, including vanilla RNN, RNN with attention, and simplified Transformer networks. They study the sequence-to-sequence networks on both synthetic and real-world tasks, i.e., a synthetic task, eSCAN and English-French translation tasks, and find that the temporal components contribute more when computing the attention.
main_review: Strengths:
1. The paper is well written. The studied problem is interesting and important. Considering the attention-based encoder-decoder networks have been widely used for sequence-to-sequence problems, I think this paper involves an interesting topic, aiming to understand the mechanism behind the attention-based encoder-decoder networks.

2. The paper gives a very detailed analysis of various aspects of the input and temporal components, and observes some interesting phenomena regarding how the attention patterns are formed by the hidden state dynamics under different tasks.

3. The experiments, consisting of both synthetic and real-world tasks, are extensive.


Weaknesses:
1. It's not clear what generalizable knowledge we should take away from the fact that in tasks where attention *can* be computed without looking at input content, attention *is* computed without looking at input content.

2. The three tasks used for experiments and analyses are relatively simple. Most real-world tasks are not like this.
- First, among the three tasks, the English-French phrase translation is the most complex, but most experimental analyses are concentrated on the eSCAN task. It's interesting to see the analysis of hidden state dynamics can discover that the attention for the one-to-one task is only driven by temporal components, and for eSCAN, some input components such as "twice" also contribute to the formation of the attention. But it is not surprising given the simplistic nature of these two tasks, and it is not clear what indication it might have over more realistic tasks. It would be better if more analysis is done on the phrase translation tasks (especially the sequence-to-sequence networks are applied to machine translation tasks mostly), and some discussion on how the analysis could generalize to other tasks would be nice. Therefore, I recommend the authors concentrate more on the analyses of more MT tasks in addition to the English-French phrase translation.
- Second, for the English-French phrase translation task, the delta component of the hidden states becomes relatively large. I am concerned that the validity and usefulness of the "input"/"temporal" analysis if the delta dominates the hidden states when applied to more realistic tasks. Besides, I suggest the authors report BLEU scores for the translation task so that readers can check and understand the performance of the trained translation models.
- More analyses on the complicated tasks would significantly add support to your claims.

3. Although extensive experimental results are provided, it is hard for me to conclude the main contribution of this paper. The authors do have summed up some conclusions, such as the temporal component contributes more to the attention, which, however, does not lead to some deeper insights. For example, I would expect that the findings of this paper could be utilized to improve the sequence-to-sequence networks and promote their performance. The paper only provides some intuitive conclusions to the sequence-to-sequence networks, without taking a step forward and proposing some novel improvements to them. Can we get any benefits from the observation, e.g., guiding us to design better models?

4. Some parts of the paper can be made clearer for easier reading. 
- The definition of temporal and input components is essential in understanding the paper, and it should be in the main paper, not in the supplementary material. 
- The axis in Figure 2 should be explained in the paper, even if the reader might be able to guess its meaning. 
- It would be better if there is an outline paraphrase at the end of the first section.
- The authors should report BLEU scores for the translation task so that readers can check and understand the performance of the trained translation models.

Overall:
I think this is a promising start, but the paper would benefit from both a clearer statement of the generalizable machine learning knowledge, fewer monotonic tasks, and a more detailed discussion of how to interpret findings with large \Delta. Considering the new insights and observations, I would like to give a positive suggestion for this paper.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a novel method for understanding how attention works in sequence alignment problems by decomposing the hidden states into temporal and input-driven contributions.  This is applied to three different network architectures, with AO only recurrent connection, AED with both recurrent connections and attention, and AO, with only attention.  They then test on some different synthetic dataset and link the attention matrix to the hidden state decomposition and to the temporal and input component dynamics.
main_review: Originality: The analysis and decomposition appear to be novel, the paper is cites relevant papers and contrast their contributions to their analyses.

Impact:  Understanding how neural networks attend could have a suggest methods for improving performance and increasing interpretability of neural networks with attention.  Some of the main insights seem to be that the AED attention can be approximated by looking at the AO attention and assuming an implicit attention is present on the previous local elements in the sequence.  Another insight is that AED behaves closely to AO as opposed to behaving more like the VED, but it is not explored in more depth why this is more efficient.  The results and analysis are thorough and confirm that the networks are solving the synthetic/translation problems in the potentially expected manner given the dataset setup.  However, the insights provided by the analysis don't suggest further enhancements to how to improve attention and how networks are designed, with would increase the papers impact.  

Clarity:  Paper is generally well written, but there are some minor issues with the figures.  In particular, Figures 3 and 4 would be easier to compare if the same example was used for both systems.  It needs detailed study by the reader to see the differences in attention in the systems due to needing to map the different attention system to different problems.  

The presentation of the results (in particular on the synthetic sections) could end up being easier to follow if the results on the simpler architecture were introduced first, namely AO not AED, as it is a simpler system.

Line 279 is missing "with"
Line 289 in supplementary has a typo

Quality: The submission appears to be technically sound, with sufficient technical detail to replicate the experiments, and the claims are well supported.  The paper does a good job of making the different architectures comparable, and the investigation into the hidden state is well visualized in various manners in the figures.  


limitations_and_societal_impact: Limitations and potential negative social impacts are addressed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a novel method to decompose hidden representations of encoder-decoder architectures into two kinds of components: temporal and input-driven. The temporal component is the average hidden representation at a given time step thus it only varies with time, while the input-driven component is defined as the representation of a specific input word removing temporal component thus it is input-dependent but time-independent. Based on the decomposition, the authors show how encoder-decoder architectures attend, providing some insights for future research.
main_review: Overall, I think the proposed decomposition method is very interesting and intuitive, which provides a new way for understanding model behaviors of (attention-based) sequence-to-sequence learning models. However, I think it still has the following weaknesses:

***1.	The studied architectures are unrepresentative.***
In this paper, the authors study the architectures of the recurrence-based encoder-decoder architecture without any attention (VED), the attention-based encoder-decoder architecture (AED), and the attention-based encoder-decoder architecture without any recurrence (AO). However, the most widely used Transformer architecture has not been studied. The authors claim that AO bears resemblance to Transformer, but I think both the analyses and conclusions will be significantly changed in Transformer due to its much stronger layer-to-layer abstraction. The analyses and conclusions on Transformer might make the proposed decomposition method more convincing and more widely used, bringing more benefits to the community. Therefore, I think that the experiments on Transformer should be indispensable. The audiences will be less interested in a paper on sequence-to-sequence learning without Transformer experiments.

***2.	The studied tasks are unrepresentative.***
This paper only studies some very simple sequence-to-sequence learning tasks, thus a question is raising that do the contributions listed in the introduction part can really inspire current research on sequence-to-sequence learning, like machine translation, question answering, and text summarization? Do the observations of the temporal and input components really exist in the more complicated task? Although the authors have conducted experiments on the English-French translation task, the uncommon settings (e.g., only training on sentences of length up to 15 tokens) make it less convincing. 

limitations_and_societal_impact: This paper should focus more on mainstream sequence-to-sequence learning architectures and tasks.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, the authors propose a framework to analyze how different encoder-decoder architectures attend in 3 simulated tasks and 2 real-world tasks. The framework relies on isolating the  following components in the network:
1. Temporal - Average hidden state of the encoder/decoder at a specific timestep, across all the test samples.
2. Input - Average deviation of a word's hidden state from the mean state at the timepoint it occurs in, across all its occurrences in the test set.
Using these components, the paper factorizes the attention alignment between eh encoder & decoder's hidden state into 9 different products of the temporal and input states.

The usefulness of the two components in understanding attention mechanisms is demonstrated with 3 tasks:
1. Simulated tasks: Matching 3 characters to their numbered IDs | The authors find that the input and read-out states are highly aligned such that the decoder only needs to find its corresponding temporal match in the encoder. This is apparent from the results in the paper. Further, the non-attention based network doesn't demonstrate similar behavior.
2. eSCAN: Extension of the SCAN dataset formed by concatenating commands to generate longer sequences
3. English $\rightarrow$ French translation

For tasks 2 & 3, the authors find that the networks learn mappings between the input tokens and corresponding actions/French words in the readout layer. Further, the networks attend through a combination of temporal information and input components unlike task 1. Since the output in several cases doesn't depend on just 1 timepjoint in the input, these networks also exhibit 'offset' behavior wherein they attend to information at different time points depending on the input token (thus violating their orthogonality)

Overall, by separating the input and temporal components of the network state, the paper reveals if networks use only one mechanism or a combination to attend and solve a task.
main_review: Post discussion: I have read the author responses to both my comments and those of the other reviewers. The authors have highlighted the complexity of explaining attention mechanisms in large architectures and other real-world tasks. Overall, the proposed decomposition seems like a promising approach to study attention in encoder-decoder architectures and I would be interested in seeing how it can be used to improve model performance. I stand by my score.

###################################

Overall, I find this direction of research interesting and enjoyed reading several analyses in the paper. However, the current version of the paper lacks clarity and organization that could/should highlight main findings for each task & architecture and also highlight the usefulness of the component characterization beyond existing work (visualizing attention, attribution methods etc.). My confidence in the paper's merits would increase if the authors could point at a clear takeaway on how the networks behave. In its current form, it is hard to gauge the framework's usefulness for more complicated tasks that don't necessarily exhibit local alignment or easy 1-1 correspondence between tokens.

- While the SM contains clear definitions, lines 94-100 are confusingly worded despite being central to the paper. I’d recommend moving equations from SM to main text for readability. For instance, just from the main text it was not evident to me that $\mu_t^E$ was averaged across the test samples.
- Fig. 2:
    - There is no mention of PCA being used or why 2 PCs explain enough variance in the data that we can rely on them. 
    - Why only 3 tokens?
    - What is the takeaway from the result that they traverse a circular path? It should be explicitly stated (in addition to highlighting it much later on Pg-5). I agree that they have _similar_ patterns across time, but it would be helpful to visualize in a 3D space (see Cai et al., ICLR 2021 for example) It is hard to discern if the patterns are similar because of the dimensionality or the learned structure.
    - What is the purple ‘X’ in the input component visualization?
    - There is a singular mention to ‘readout’ layers in Fig. 1 caption and no mention of it again, except directly in Fig. 2. From the footnote (and not main text) it is apparent that the readout vector is the weights of that layer on the context vector. If the readout is also a point in the space, depicting it as a line is confusing. In my opinion, it would be easier to understand the structure in the lower panels if everything was visualized as a dot/blob.
    - Line 137: It would be helpful to elaborate on why the temporal and input terms are orthogonal.
    - Overall, some of the explanation is missing/confusing, but the characterization of network behavior in Lines 140-147 is interesting!
    - Clarify the attention matrix headings for row 2 in Fig 1h.
- Fig. 3 & 4 & 5:
    - It would be helpful to demonstrate attentional differences between the two architectures in Fig. 3 by using the same input sequence.
    - Add discussion on Fig 4c to main text.
    - Overall, I think the visualizations are interesting but I’m afraid I don’t understand how they inform our understanding of the network beyond visualizing attention matrices. In other words, can the authors elaborate on what splitting the network representations into temporal and input components tells us in addition to attention matrices?
- Where are the details of this?
	> It is possible to determine the average theoretical offset of input and output words in eSCAN
- I agree with several inferences made in the real-world experiments regarding the nature of attention and the use of temporal information. But if we were looking at a task that didn’t have local alignment like in eSCAN or English$\rightarrow$French, it would be much harder to interpret under the proposed framework. As a simple example, it is not apparent to me what Fig. 4b vs. its inset indicate. One says that the temporal components are aligned in some fashion from start to end but the softmax shows that the network is relying on temporal components farther in the past as the translations get ‘offset’. What does this tell us about how the network solves the task? How is this building on our understanding beyond say, visualizing raw attention values as a heat map and averaging across samples in a test set? While these inferences fit into a clear possible explanation of the network in the simulated task, it seems a much harder ask in the English-to-French translation settings. I would appreciate if the authors can clarify the usefulness of the proposed temporal & input components framework given the intuitive results that in a non one-to-one task that relies on multi-word information like translation, the input plays a significant role and the network doesn’t rely on pure temporal alignment. Having said that, I found the characterization in Fig. 5 interesting and would be excited about future directions that can provide more insight into the model.
- I extend my comments on how interpretation may still be hard with this setup for more complex tasks. While the analysis on ‘twice’ in Section 4.2 throws light on how the input and temporal components both play a role to output required behavior (and do this differently between recurrent and position-encoding based architectures), I disagree that the analysis is equivalent to and as simple as the translation case. While _run twice_$\rightarrow$`RUN RUN` only needs to attend to the past word and copy it over, there could be examples of translation wherein there is a paraphrase/ difference in case based on meaning etc. wherein the one-to-one mapping is lost like in _we_$\rightarrow$_nous_ or _ils_$\rightarrow$_they_. To this end, what is the scope of this method?
limitations_and_societal_impact: The authors do not discuss any limitations or societal impact of their approach.
review_title: null

Meta-review: Three of four reviewers rated this paper as a 6, one reviewer still assesses this paper as being below threshold.
The concerns that remain relate to the general applicability of the method, but I think the work is in good enough shape for a poster.
The AC recommends accept as poster.

Decision: Accept (Poster)