Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper conducts detailed study on relaxed attention for transformer. The relaxed attention mechanism is applying smoothing to the attention weights. Previous work which proposed relaxed attention only applied it to cross-attention in the decoder. This work studies applying relaxed attention to self-attention layer in the encoder. The authors suggest two benefits: 1. the relaxed attention provides regularization when applied to the self-attention layer. 2. the relaxed attention applied to the cross-attention layer suppresses the intrinsic language model and allows better integration of an external language model. The experiments are conducted in automatic speech recognition (ASR), lip-reading, machine translation, and image classification. The work shows better or on par performance with state-of-the-art models on lip-reading LRS3 benchmark (-0.6% in word error rate) and machine translation IWSLT14(DE->EN) (+0.18 BLEU). 
strength_and_weaknesses: Strength:
1. The experiments are detailed and cover multiple tasks/domains (speech, translation, and image).

Weakness:
Minor sentence fluency:
1. page 2: "Very early once ..."
2. page 9 in Conclusions: "thereby regularizing already in the encoder"

Question:
  - Have you tried applying the relaxed attention in the pretraining stage?
clarity,_quality,_novelty_and_reproducibility: The paper is clear, with minor places where the fluency of the sentence could be improved.
The paper conducts experiments that introduces new understanding into a previously proposed method.
I believe the experiments can be reproduced.
summary_of_the_review: The thorough experiments confirm the usefulness of an previously proposed technique that smooths the attention weights.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper explores relaxed attention in transformer models. The relaxed attention interpolates the attention weights with a uniform distribution. This method was proposed by previous work, and this paper extends its application from cross attention to all attention use cases (cross, self, causal). The paper further proposes a variant that randomly draws the interpolating coefficients at training time.

The experiments cover a variety of tasks, including automatic speech recognition, lip reading, machine translation, and image classification. On some, the results are mixed; on others, the improvements are marginal. 


strength_and_weaknesses: Strength
- Clear presentation
- The method is straightforward
- Experiments cover a variety of tasks

Weaknesses
- Extending an existing technique to self and causal attention is thin in terms of technical contribution
- No evidence is provided to support the significance of the improvements (if any)
- Lacks in-depth analysis and discussion of the method
- An important baseline is missing: smoothing the attention weights using a larger temperature
- The “relaxed attention” naming is unfortunate—it leaves me the impression that it is lifting some constraints in conventional attention (e.g., simplex). I suggest changing it to something related to “smoothed” or “flat.”

Details
- The improvements in the MT experiments seem very marginal—the ~0.2 BLEU delta is definitely reachable with a lucky random seed; I suspect this might be the case in other tasks too (MT is the only one I’m familiar with). The paper can benefit from training multiple models with different random seeds, and/or presenting significance tests.
- It is great to cover a diverse set of tasks, but the paper favors width over depth. I would really appreciate it if the paper could include some in-depth analysis and discussion of the method. For example, how does the smoothing term affect the “flatness” of attention? Comparing the entropy of the attention weights w/ and w/o the uniform term is a good starting point. Performance change over different choices of gamma is another good one. Some discussion/analysis on why smoother attention is better is also interesting. 
clarity,_quality,_novelty_and_reproducibility: - The paper is clearly written. 
- It falls short in terms of technical novelty. 
- Without carefully checking the appendix, my educated guess is that the results are fairly easy to reproduce. 
summary_of_the_review: The paper did a good job in terms of covering a variety of tasks. But it lacks depth at the same time. Given that the technical contribution is thin compared to average ICLR papers, I suggest that the authors should dive deeper into the core assumption of the paper: smooth attention is better in some applications.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed to use relaxed attention in both self attention and cross attention part of transformer models. In details, the authors proposed to add a smoothing term on the cross attention module with a fuzzy realaxation coefficient drawn from a Gaussian distribution. Extensive experiments were done on automatic speech recognition, lip-reading, machine translation, and image classification.
strength_and_weaknesses: Strength:
1. Simple techniques apply to wide range of application.
2. Extensive experiments.

Weakness
1. Novelty. The method was first proposed in [1] as the authors cited in paper.
2. The experimental results are not strong enough as the improvement seems marginal. And also lack of variance on all results. For the results with less significant improvement, it would better to also show the variance among runs.

[1] TimoLohrenz,PatrickSchwarz,ZhengyangLi,andTimFingscheidt.RelaxedAttention:ASimple Method to Boost Performance of End-to-End Automatic Speech Recognition. InProc. of ASRU, pp. 177–184, Cartagena, Colombia, December 2021. 
clarity,_quality,_novelty_and_reproducibility: The paper is overall clear. 

It introduces a simple idea with a lot of experimental results. The significance of the proposed model is concerning as the improvement is marginal.

It is somewhat novel, although similar idea was proposed in literature, extending the application to wider range is still valuable.

The code is provided in supplementary. 
summary_of_the_review: Generally I think relaxed attention is an interesting idea, as it is simple and easy to apply in multiple applications. However, the efficiency of the technique is supported by experimental results, which is not significant enough as the improvements are marginal and metrics' variance between runs are not provided.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the relaxed attention method on the Transformer architecture, across a variety of tasks including automatic speech recognition, lip reading, machine translation, image classification. The technical contributions include exploring relaxed attention on the self-attention module, utilizing it on both training and inference, and introducing variation into image classification where the sequence length is fixed. 
strength_and_weaknesses: **Strengths:**

1. The paper studies various tasks including on text, speech and image, which guarantees that the conclusions are general and robust.
2. The paper is generally well-written and easy to follow.


**Weaknesses:**
1. The contribution is somewhat incremental. Extending relaxed attention from cross-attention to self-attention is straightforward, the paper shows that it will bring some improvements to the model performance, but lacks showing the novelty of doing so. For example,  what it brings to the training/inference of the model? Does it solve some problems that the original attention has? And how the relaxed attention achieves that? In my experience, the sparsity in self-attention is natural and provides clear local context information to the model, then what the smoothing regularization term brings to the model?
2. Another contribution, the matched inference, does not show consistent improvement across different tasks. The paper does not provide a detailed analysis of the results which also limits the novelty. If it cannot bring improvements and no in-depth explanation is conducted, then why take it as one of the contributions?
3. The method section is too short to provide the details and insights of the proposed method.
4. The experimental settings lack some details. For example, what is the difference between the clean and other settings in the ASR task? In the “Approach” column in each table, is the relaxed self-attention added on top of relaxed cross-attention or they are separately enabled? In the machine translation tasks, have you conducted experiments on larger benchmark datasets such as WMT which will make the conclusions more faithful?

clarity,_quality,_novelty_and_reproducibility: The formulation of the problem is clear and the paper is well-written. However, the contribution is quite limited. 
summary_of_the_review: I believe the novelty of the paper is straightforward, but the contribution is limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper extends relaxed attention to self and causal attention. Although relaxed attention was introduced earlier, its potential for translation, image classification and lip reading was not assessed before.
 
Strength:
- detailed experiments in multiple domains.
- clear paper, easy to follow. It provides supporting code.
 
Weaknesses:
- the reviewers expressed doubts about the significance of the technical contribution.
- the reviewers found the empirical benefit of the method limited.


justification_for_why_not_higher_score: - lack of novelty, limited empirical benefit.

justification_for_why_not_lower_score: - the paper is clear, the approach is simple and easy to incorporate.
Decision: Reject