Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Impact (HI): a submission which meets the reviewing criteria and is predicted to make an impact on the field
review_content: summary_of_contributions: This paper empirically investigates the transformer model's attention to out-of-the-distribution words. Specifically, the authors measure transformers' attention with the original token and a perturbed token with a lower log-likelihood at position i, Lˆsi ≪ Lsi. Then, they statistically compared the attention on both the original and perturbed tokens.
strengths_and_weaknesses: Strength:
Systematic investigation of the transformer's attention for the out-of-distribution words.

Weakness
Although the perturbed attention showed a stong correlation ρ, the correlation was not statistically significant as demonstrated by the p-value
suggested_changes: I enjoyed reading the paper.

The authors could investigate a follow-up study along this line: how does the model's size impact the out-of-distribution tokens' attention? 
Do LLMs with large parameter sizes handle the out-of-distribution tokens well than the smaller ones?
From the table, the RoBERTa large has the greatest correlation coefficient.
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions:  This paper analyzes how large language models represent a token and its attention. They find a strong correlation between token likelihood and attention values, indicating that the model relies on the token to understand its semantics. 
strengths_and_weaknesses: Strength
1. The paper is well written and the problem is well defined. 
2. The results are reproducible. 


suggested_changes: Suggested Changes:
1. It would be beneficial if the author could provide practical examples of how their research can be applied in practical tasks.
review_title: null

final_score: rating: Needs Clarification (NC): a submission which does not meet the reviewing criteria and needs clarification for its described problem or solution
review_content: summary_of_contributions: This paper explores the correlation between token likelihood and attention values in transformer-based language models and finds that model relies more on the context word to compute the representation of an unexpected token.
strengths_and_weaknesses: Strengths:

This paper extensively explores the correlation between token likelihood and attention values at different layers using Spearman correlation.

Weaknesses:

1. Some important details are missing:

(a) How to get “perturbed” token is unclear. 

(b) The sentence attention is a vector as it is the concatenation of the other attention weights, but how can you calculate the Spearman correlation between a vector variable and a number variable?

 (c) It is unclear which type of attention is used in the calculation of the Spearman correlation between attention values and loglikelihood. (attention, token attention or sentence attention?) 

2. It is unclear what the practical implications of this study are, and it appears to be a technically correct but perhaps not very useful exploration. It would be helpful if the authors could provide a more thorough explanation for the motivation and significance of their work.

suggested_changes: Typos:

1. Wrong quote in “it with a ”perturbed” token ...” in page2.

2. Tense error in appendix, such as “we retrieved all... ” -> “we retrieve all...” in page5.
review_title: null

Meta-review: metareview: This paper explores the correlation between token likelihood and attention values at different layers using Spearman correlation. 
The paper is well-written, the experiments support the hypothesis and the authors followed the submission guidelines. 
There are some clarifications needed for instance the authors mention that the sentence attention is a vector as it is the concatenation of the other attention weights, but it's not clear how they calculate the Spearman correlation between a vector variable and a number variable. 
Overall, this is a nice submission with valued findings and high impact to the field. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to present