Relaxed Attention for Transformer Models
Authors: Timo Lohrenz
Björn Möller
Zhengyang Li
Tim Fingscheidt
Venue: Submitted to ICLR 2023 (2023)
Keywords: transformer
attention
regularization
internal language model
relaxed attention

Abstract: The powerful modeling capabilities of all-attention-based transformer architectures often cause overfitting and - for natural language processing tasks - lead to an implicitly learned internal language model in the autoregressive transformer decoder complicating the integration of external language models. In this paper, we explore relaxed attention, a simple and easy-to-implement smoothing of the attention weights, yielding a two-fold improvement to the general transformer architecture: First, relaxed attention provides regularization when applied to the self-attention layers in the encoder. Second, we show that it naturally supports the integration of an external language model as it suppresses the implicitly learned internal language model by relaxing the cross attention in the decoder. We demonstrate the benefit of relaxed attention across several tasks with clear improvement in combination with recent benchmark approaches. Specifically, we exceed the former state-of-the-art performance of 26.90% word error rate on the largest public lip-reading LRS3 benchmark with a word error rate of 26.31%, as well as we achieve a top-performing BLEU score of 37.67 on the IWSLT14 (DE$\rightarrow$EN) machine translation task without external language models and virtually no additional model parameters. Code and models will be made publicly available.

Under review as a conference paper at ICLR 2023
RELAXED ATTENTION FOR TRANSFORMER MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
The powerful modeling capabilities of all-attention-based transformer architee- tures often cause overfitting and-for natural language processing tasks-lead to an implicitly learned internal language model in the autoregressive transformer decoder complicating the integration of external language models. In this paper, we explore relaxed attention, a simple and sosy-to-implemen smoothing of the attention weights, yielding two-fold improvement the general transformer architecture: First, relaxed attention provides regularization when applied to the self-attention layers in the encoder. Second, we show that naturally supports the integration of an external language model as it suppresses the implicitly learned internal language model by relaxing the cross attention in the decoder. We demon- strate the benefit of relaxed attention across several tasks with clear improvement in combination with recent benchmark approaches. Specifically, we exceed the former state-of-the of-the-art performance of 26.90% word error rate on the largest public lip-reading LRS3 benchmark with word error rate of 26.31 %, as well as we achieve pop-perormnnn BLEU score of 37.67 the IWSLT14 14 (DE + EN) ma- chine translation task without external language models and virtually no additional model parameters. Code and models will be made publicly available.
INTRODUCTION
Early encoder-decoder models emerged from machine translation, where the encoder compressed the entire source language sentence into a fixed-length embedding vector (Cho et al., (2014b). This is particularly difficult for very long sentences (Cho et al. (2014a), as the fixed-length embedding vector only limited-capacity representation. The use of attention, introduced in Bahdanau al. (2015). enabled the computation of variable-length weight distributions over the input sequence and soon turned out be advantageous for far more applications than just neural machine translation (NMT), e.g., automatic speech recognition (ASR) (Chorowski et al. 2015) Chan et al. 2016 Bahdanau ct al. 2016), language modeling and understanding (Devlin et al. 2019), object detection (Carion et al. 2020). and image classification (Dosovitskiy al.) [022 Liu al.. 2021b). Soon the most prominent attention- encoder-decodee (AED) model emerged, namely the transformer (Vaswani et al., 2017) model. Without the of any recurrency, it entirely relies self-attention in the encoder to model temporal dependencies the input and cross attention in the decoder to extract relevant timesteps thereof during the autoregressive decoding process. While transformers in language modeling tasks are well-suited for upscaling the model size and depth without any saturation when large amounts of data are present (Devlin et al. 2019) Brown al. 2020 Kaplan et 2020 Fedus et al. 2022). they are also susceptible to overfit and require strong regularization to learn at all (Xu al. 2021 Popel Bojar 2018 Steiner et al. (02T). In study exclusively on ASR (Lohrenz et al. 2021). it was shown that regularization by smoothing the attention weights in the decoder's 's cross attention, dubbed relaxed attention, improves performance when the transformer model is combined with an external language model but, for reasons yet to be explored, does not help without language model.
In this work, we take the idea of relaxed attention to expand it to the self-attention layers in the encoder, regularizing already the encoder. Thereby, we increase the method's 's versatility as it becomes applicable to encoder-only transformers, which are common in several non-sequence tasks such as image classification or pre-trained bidirectional encoder representation by transformer (BERT, Devlin et al. (2019) models. Our main contributions are summarized follows:

---

Under review as a conference paper at ICLR 2023
We introduce relaxed self-attention in the transformer encoder to improve generalization and develop fuzzy relaxation as a variant thereof.
Beyond relaxed self-attention, we extensively investigate the capability of relaxed cross attention in the decoder of sequence-to-ssequenne transformer models and show that the improvement is due better external language model integration as it suppresses the influence of the internal language model.
We show improvements of the relaxed attention approaches on a variety of tasks including automatic speech recognition, lip-reading, machine translation, and image classification. On the lip-reading and machine translation task we report a new state of the art and top- performing result, respectively.
The paper is structured as follows:
After summary of related work in Section 2 we introduce the relaxed attention approach in Section 3 followed by the experimental evaluation including results and discussion in Section[4] [ Section 5 concludes the paper.
2 RELATED WORK
Regularization of transformer models In this work, we introduce regularization method the self-attention function (Vaswani et al., (017)) which is fundamental to transformer models. Several regularization approaches proposed for such networks in the past are related to the network output of transformer models by modifying the loss computation, either through label smoothing (Müller et al. 2020 or by introducing additional loss terms. This could be a CTC-llss computed on the encoder outputs (Karita et al.] [2019] Chen et al.. 2021) for monotonous tasks such as ASR, or a divergence term between output softmax distributions of two forward passes with different dropout masks (Liang et al. [021]] Shen et al. 2020). Related to the network input, several-mosthy application-dependenn data augmentation approaches such as spectral augmentation for ASR (Park et al.] 20T9b), or cutoff for machine translation (Shen et al. 2020 have proven to be effective. Another set of regularization methods, specific to transformer models, adds loss term to encourage attention heads to yield diverse outputs (Li et al. 2018; Audhkhasi et al., 2022) or is based the dropout technique (Srivastaval jet al. 2014) and randomly masks attention heads (Zhou et al.) 2020 [un et al.] 2020) or entire en-/decoder block layers (LayerDrop) (Fan et al. 2020) during training. It was also observed that only a subset of specialized attention heads contribute to model performance, while other heads remain useless and can be pruned (Voita et al. 2019.. Relaxed attention in Lohrenz al. (202I) was used to prevent too narrow attention weight distributions in the cross attention during training which only yielded improvements with an external language model in ASR. We, however, apply this approach to the self-attention function to reduce over-fitting already in the encoder and investigate if relaxed self-attention also helps when applied during both training and test (matched inference). In addition we include variety of the atoremmentioned-ppe -rroven to be rereciive-reuulriizaatioo methods as baselines and show that relaxed attention is able to further improve performance yielding complementarity to other regularization methods.
When attention based encoder- proposed modified softmax function to smooth the attention weights in the cross attention between encoder and decoder by replacing the exponential function in the standard softmax function with a sigmoid. Thereby, they compressed the probability-like outputs, but didn't 't take into account the input sequence length, despite the authors' observation that longer sentences require less smoothing of the attention weights. Even though this method dubbed smooth focus was so far only applied to recurrent neural network (RNN)-based AED models, we include it as reference method in our simulations as it is the closest the relaxed attention approach.
networks were first applied to ASR, Chorowski et al. (2015)
Internal language model handling For many equeence-t--squeeee tasks the integration of lan- guage models (LMs) to AED models is of dual use: First, LMs leverage the use of large amounts of additional text-only data to improve performance. Second, LMs can be utilized to adapt acoustic models to domains which differ from the original acoustic training data domain. Several techniques exist to combine language models with AED models, such shallow fusion (Gülçehre et al. 2015). deep fusion Gülççehre et al. 2015), and cold fusion (Sriram et al. 2018), whereas shallow fusion still is the most common solution due to its simplicity and flexibility. However, AED models tend to learn an internal language model in the autoregressive decoder (McDermott et al., 2019). which can either be suppressed by subtracting additional LM trained only text transcriptions from the
2

---

Under review as a conference paper at ICLR 2023
Z
Bxd
Multi-Head Attention
Fully Connected
1 B
Concatenation
2. 4Bx Lxd/44
Matrix Multiplication
X Lx
4Bx
G
Y
Relaxation C
4x
G; 4B LxT
Softmax
Attention Weights
Matrix Multiplication
B BxLxd/4
Bx
4x
Fully Connected
Fully Connected
Fully Connected
Q Bx Lxd
K Bxxxd
BxTxd
Figure I: Multi-head attention (MHA) as used encoder and decoder blocks transformer models
with N₁ = attention heads. The proposed relaxed attention (red block) is presented in Section
acoustic training data (c.g., density ratio fusion from McDermott et al. (2019) or as was shown
more recently-can be adapted to domain (Meng al. 2022) requiring additional retraining.
For the specific application of automatic speech recognition, in a small study, Lohrenz et al. (2021) ) have investigated relaxed cross attention, whereby performance improvements with external language models. In our work, we investigate the hypothesis that relaxed cross attention successfully suppresses the internal language model, which in contrast the aforementioned methods does not-apart from a single hyperparameter- any additional models (McDermott et al. 2019), parameters (Sriram et al. 2018), or adaptation trainings (Meng et 2022). but weakens the internal language model during training of the transformer thus supporting shallow fusion (Gülgehre ct al. 2015). In addition, we will introduce relaxed self-attention, which improves performance in many applications even without the use of an explicit L.M
only achieved
3 PROPOSED RELAXED ATTENTION
Scaled dot pot-product multi-head attention (MHA, see Figure 1 is typically used in two variants in the encoder-decode transformer model (Vaswani et al. 2017 First, it used in the encoder as self-attention to model positional (e.g., temporal) dependencies in the preprocessed input sequence indexed with f E [1..... T} of length T. Second, it is used in the decoder as cross attention (also often referred to encoder-decoder or source target attention), which draws the decoder's ' attention to relevant parts the encoded input sequence hf e R'xx for decoding at output sequence index Le (I.....) L} with model dimension d. In case of self-attention!) all MHA inputs (key K, value V. query Q) are the same, i.e., K=V= with query input Q E Lxd of length i. T. For cross attention, key and value inputs, K g''xd and V V respectively, stem from the encoder output of yielding K V=V=hā' while the query input Q comes from the previous decoder layer with Ī==1 during inference and L=L during training, where for the latter all L tokens of the target sequence are processed in parallel. Details of the entire typical encoder-decoder transformer architectures are recapitulated in Appendix A.1 The attention weights G. K) (K)1ll 1 for the scaled dot- fot-product MHA sum up one across the query input length L after the softmax activation function and thus
Note that masked self-attention is also used in the decoder to attend to prefix output tokens (ef. Fig. 2 Appendix A. while in this work we focus the multi-head attention variants that attend the input domain.
3

---

Under review as a conference paper at ICLR 2023
can be interpreted as a probabilistic weighting applied to the value input projection Y, E E Rxd/N, with N₁, being the number of attention heads each indexed with i N.
Relaxed attention follows the basic principle of regularization by introducing some stress into the training process. Shown as red box in Figure iit modifies the standard attention weights QQ K) that draw attention the encoded input sequence. Our relaxed attention weights for scaled dot- product attention are defined as simple as
G. 111-1111100 (1) 7 i E
1
gradually injecting a uniform distribution (with 1 here being an x ''' matrix of ones), thereby smoothing the distribution of attention weights and imposing some of the attention weight mass across the entire input sequence. Please note that the normalizing division by T in equation (T) is different for each file and minibatch for sequence tasks due to varying input sequence lengths. This provides natural and rich variation of the effective height of the injected uniform distribution and prevents compensation of the relaxation by the learning process. The injection of the uniform distribution is controlled by relaxation coefficient y e [0, 1J. which is constant single hyperparameter for all respective attention layers.
While Lohrenz al. (2021) only investigated relaxed cross attention, only for automatic speech recognition and only during training. in our work (i) we propose relaxed cross attention and self- attention, (ii) during training and during inference (matched inference), (iii) investigate their application to automatic speech recognition, lip-reading, machine translation, and image classification, and (iv) we introduce fuzzy relaxation for the image classification task, where we randomly draw the relaxation coefficient from a normal distribution N (æ; == o²), with the initially set "o being the mean μ.. For this specific task, the variable sequence length T in equation (T) is substituted by constant number of image patch tokens M (see equation (3) in Appendix A.2), thereby omitting the aforementioned natural variation. Fuzzy relaxation e-establishes this variation of the relaxation by randomizing during training. while for the matched inference case, the relaxation coefficient is kept fixed at 7=1=%0 during inference. Details for the encoder-only transformer used for the image classification task are given in Appendix A.2.
4 EXPERIMENTAL VALIDATION AND DISCUSSION
4.1 APPLICATION TO AUTOMATIC SPEECH RECOGNITION
Task and datasets Automatic speech recognition transforms recorded speech signals into a se- quence of text tokens. We investigate
relaxed attention method on the Librispeech dataset (Panay- otov et al. 2015) with the clean and other conditions of the dev and test subsets. We measure system performance in terms of word error rate WER= 1 N --D-J-S, depending on the number of words N, deletions D, insertions and substitutions S. All raw speech signals are sampled at kHz and analyzed with 25 ms window at a frame shift of 10 ms. As common in ASR, we also use an external language model trained on the text labels of the 960 h training set as well as on the text-only Librispeech language model training corpus, the latter containing sentences from a total amount of 14,500 books from project Gutenberg (Panayyoovv et al.) (2015) which are accessible under publie-domain. The Librispeech ASR corpus is available under the very permissive Creative Commons BY 4.0 license.
Models and training For training with 100h and 960h of training data, we trained standard encoder-decoder transformer models (Vaswani et al. 2017) from scratch in the small and base configuration, comprising 19.3M and 8M.8M parameters, respectively. As common for ASR, filterbank features are extracted for each time frame t and then preprocessed by layer convolutional neural network, each using 3 3 filter kernels (cf. preprocessing block in Figure [ Appendix IA. All hyperparameters were according the recipes available in the fairseq based espresso toolki (Wang et al., 2019) except the relaxation coefficients % which have been tuned on the joint clean and other portions of the dev set for both, relaxed cross attention and relaxed self-attention. As additional regularization we use SpecAugment (Park et al. 2019a), label smoothing (Müller et al. 2020 and dropout (Srivastava et al. 2014 during training. See Appendix A.3.1 for more details.
ASS training recipes
4

---

Under review as a conference paper at ICLR 2023
without LM
with LM
Training data
dev
test
dev
test
Approach
clean other clean other
clean other clean other
Baseline (Lohrenz et al. (2021), resim.)
13.98
28.71
14.82
29.31
10.62
24.19
12.06
25.56
smooth Tocus (Chorowski ct al. 2015
14.60
28.73
15,50
30.78
10.83
24.86
12.11
26.46
relaxed cross atlentron
13.91
28.70
14.70
30.10
9.33
22.16
10,62
23.04
100h
matched inference
14.30
29.03
15.15
30.09
11.04
25.19
12.16
26.36
relaxed self-attention
13.48
27.87
14.20
28.96
10.22
23.53
11.04
24.55
matched inference
13.43
28.00
14.46
29.23
10.01
23.96
11.19
25.32
Baseline (I (ohrrenz al. (2021). resim.)
3.92
9.00
4.47
9.23
3.73
8.52
4.40
8.95
smooth Tocus (Chorowski ct al. 2015)
4.11
9.42
4.35
9.63
3.70
9.18
4.31
9.33
relaxed cross attention
3.95
9.33
4.28
9.45
3.44
7.74
3.58
8.35
960h
matched inference
3.96
9.29
4.20
9.40
3.69
8.95
4.21
9,46
relaxed self-attention
3.82
8.50
4.05
8.71
3.52
8.03
4.17
8.51
matched inference
3.79
9.12
4.09
9.07
3.35
8.28
3.91
8.50
Table I: Automatic speech recognition results in terms of WER (%) on the Librispeech task using standard encoder-decoder transformer models. Attention relaxation is applied in training only, except for "matched inference" (attention relaxation in training and test). We separately use the 100 h and 960 h training datasets and highlight the respective best results for each size in bold font.
Results and discussion For both, the 100 and 960 training data cases in Table the resimulated baselines (using training scripts from Wang et al. (2019) yield similar results as Lohrenz et al. (202T) using a standard transformer approach. The smooth focus method (Chorowski et al. 2015) has a higher WER compared to the baseline on the small training data case, but yields small improvements on some clean settings for the 960 h training data case. Compared to smooth focus, relaxed self- and cross attention adapt to the length T of the input sequence, with the latter yielding solid WER reduction across all dev and test conditions when LM is used (right-hand side of Table D). thereby confirming the results Lohrenz et al. (202I) In Appendix [..4 we show that the strong improvement with LM using relaxed cross attention due improved internal language model suppression. Without an LM, both the resimulated baseline and relaxed cross attention approaches are outperformed by our new relaxed self-attention in all dev and test conditions for both training data cases. Specifically, the WER across the test conditions of the 960 h case for relaxed self-attention improved by relative 9% (clean) and 5% (other) compared to the resimulated baseline, yielding complementary regularization of our method to the other employed regularization methods. Note that in all aforementioned cases, relaxed attention best when used only in training. Only very specific case on the dev set, however, "matched inference", i.e., relaxed self-attention in training and test, is slightly ahead using it in training only. Please also see the Appendix, where we conduct deeper analysis on initialization seed robustness (Appendix[A.5 sensitivity of the relaxation coefficient (Appendix A.7 as well as an ablation on the related attention dropout (Appendix A.6)
[A.4]
4.2 APPLICATION TO LIP-READING
Task and datasets Automatic lip-reading strives to process an image sequence from recordings of talking faces. We evaluate lip-reading performance in terms of WER on the test partition of the Lip Reading Sentences 3 (LRS3) dataset consisting of a total of 1.321 recorded videos of English utterances sourced from TED talks (Afouras al. (018). To investigate the performance of the relaxed attention approach on recently successful self-supervised learning approaches, we closely follow the training setup from Shi et al. (2022) and use audio-visual hidden unit BERT HUBERT) encoder models pre-trained on the English subset of the Voxceleb2 dataset (Chung et al. 2018). containing a total amount of 1,326 hours of unlabeled video recordings. For some experiments we also use external language model trained the joint text data from LRS3 and the Librispeech language model training corpus. LRS3 is publicly available under the TED terms of use as well as the Creative Commons BY-NC-ND 4.0 license.
5

---

Under review as a conference paper at ICLR 2023
Unlabeled
Labeled data (fine-tuning)
without LM
with LM
data
Approach
(pre-training)
dev
test
dev
test
h34
433h
Afouras et al.
2020
59.80
1,362 +157h
Atouras et al.
2018
59,90
58.90
433 + 157h
ct al.
2020
57.80
h
Ma at
202
46.90
433 h + 157h
Ma et al.
2021
43.30
h
Makıno
2019)
33.60
Baseline Shi et al.
20022)
46.10
Baseline Shi et al. 20022) resimulated)
47.36
45,90
47.61
45.33
smooth Iocus (C Chorowski et al. (2015)
47.08
45.80
46.69
45.38
30
relaxed attention
45,92
44.00
45.11
42.68
matched inference
46.55
45.25
46,46
45.39
relaxed self-attention
46.90
45.47
46.95
44.64
matched inference
46.85
45.04
46.68
44.69
Baseline (Shi et al.
(2022)
28.60
Baseline (hh et al. (2022 resimulated)
21.90
29.52
21.61
28.97
+ smooth locus Chorowski et al. (2015)
21.87
29.25
21.29
28.86
1,326
4
relaxed cross attention
22.12
29.49
21.05
28.05
matched inference
22.11
29.20
21.55
28.55
relaxed self-attention
21.89
28.96
21,25
28.55
matched inference
21.86
28.84
21.24
28.48
Baseline Shi et al. (2022)
26.90
Baseline (hi al. (2022 resim.)
17.71
26.73
17.18
26.50
h
smooth locus (Chorowski et (2015)
17.42
26.78
17.22
26.29
+
1,326 h
relaxed cross attention
17.40
26.57
16.92
25.51
matched inference
17.71
26.43
17.48
25,95
relaxed self-attention
17.54
26.31
17.12
26.17
matched inference
17.65
26.40
17.16
26.06
Table 2: Automatic lip-reading results in terms of WER (%) on the LRS3 task using various topologies (top segment baselines) or AV-HuBERT encoders (lower three seg- ments) pre-trained on unlabeled English data from Voxceleb2 and fine-tuned with joint transformer decoder on the given amount fine-tuning training data. We also use self-training (bottom segment) by creating pseudo-labels for the 1,326 h of unlabeled data and using these for fine-tuning. Attention relaxation is applied training only, except for "matched inference" (attention relaxation training and test). Best results for each of the three fine-tuning setups are in bold font.
sequence-tee sequence-to-sequee
Models and training We use AU-HUBERT model introduced recently by Shi al. 2022). which receive image and acoustic frames for pre-training by unlabeled training data to iteratively learn contextualized feature representations h/_ For fine-tuning and inference, only the video input is used and preprocessed (cf. preprocessing layer in Figure Appendix A. with 3D convolutional layer and a subsequent ResNet 18 (He et al. 2016)) Stafylakis Tzimiropoulos, (017) architecture. The models fine-tuned on h0 of LRS3 training data use the base configuration of the downloaded AV-HuBERT encoder and have a total of 160M parameters. Models fine-tuned on 433 h LRS3 training data (with without self self-training) use the large AV-HuBERT encoder and comprise 477M parameters in total, As additional regularization methods we use label smoothing (Müller et al. 2020).
LayerDrop (Fan al.] 2020), well dropout (Srivastava al., 2014). For final experiments, use the self-training (Zoph et al. (020)) method, where an AV- HUERT model fine-tuned on 433 of LRS3 training data inferred to generate pseudo-labels for the 1,326 h of unlabeled Voxceleb2 data. These were then used together with the true labels from the LRS3 training data to fine-tune the pre-trained -uBBER model. Relaxed attention was only used during this final fine-tuning, and relaxation coefficient 7 of each relaxed attention approach was optimized on the development set for each corresponding amount of fine-tuning data. See Appendix A.3.2 for more details.
*Pre-trained -UHBBERT models and fine-tuning code downloaded from https://github.com facebbokreserrch/av.lbbbett

---

Under review as a conference paper at ICLR 2023
Results and discussion The upper segment of Table shows various baselines on LRS3, whereby Makino et al. (2019) reached 33.60% WER w/o LM, using 33,000 of YouTube training data, and Ma et al. (2021) achieved 43.30% with LM and 157 h of additional data from the Lip Reading in the Wild dataset (Chung & Zisserman 2016). By leveraging pre-training of AV-HuBERT models, (Shi et al. (2022) report state of the art so far on LRS3 in three cases with 1,326 h unlabeled pre-training data plus 30h, plus 433 h, plus 433 h+ 1,326 of fine-tuning data, respectively, the latter using self-training leverage the pre-training data using pseudo-labels. See also our resimulated numbers of that approach. Note that as it is common practice on the LRS3 task to not even report performance on dev condition, we also formulate performance claims on the test set. Smooth fo- cus (Chorowski et al., 2015) helps bit in 4 out of the 6 total test conditions. Without a language mondlladdiing virtually no parameters and only marginally more complexity during training -ur relaxed self-attention achieves WERs of 45.04% vs. 45.90% from [Shi et al. (2022), resimulated, and 28.84% vs. 29.52% from Shi et al. 2022), resimulated, in the 30h and 433 fine-tuning cases, respectively, with matched inference (relaxation in training and test). With self-training (433 h + 1,326 h), relaxed self-attention without matched inference even achieves 26.31 % WER compared to the best lip-reading WER of 26.90% from Shi et al. (2022) thus setting new state of the art for LRS3. With an additional LM, similar to the ASR task in Section 4.1 relaxed cross attention yields consistent improvemett on the test set compared the resimulated baseline in all three fine-tuning cases (i.e., 42.68 % vs. 45.33%, 28.05% % vs. 28. 97.%, and 25.51% % vs. 26.50%, respectively). We show that this is also caused by the improved internal language model handling for this task in Appendix A.4. We also investigate robustness towards different initialization seeds in Appendix A.5
4.3 APPLICATION TO MACHINE TRANSLATION
Task and datasets Neural machine translation (NMT) models use neural networks that translate an input text sequence from a source language to a different target language. For our particular experi- ments on relaxed attention we use data from the well-known IWSLT14 translation challenge (Cettolo jet al. 2014), choosing the Berman-to-Englin of BLEU scores (Papineni et al., 2002). For training of an external LM we either use the IWSLT14 target language transcripts (160k utterances) or the MuST-C dataset, the latter contains 47% addi- tional transcripts (236k utterances) from TED talks and is available under the Creative Commons BY-NC-ND 4.0 international license (Cattoni et al. 202T).
(DE~ EN) subtask and report performance in terms
Models and training For training we use the standard encoder-decoder transformer model from Vaswani et al. (2017) in the base configuration with 36.7M parameters and apply cutoff augmentation, which first randomly masks input positions and feature dimensions of the embedded input tokens and second uses a divergence loss to minimize the difference in predictions for different input maske") (Shen et al., 2020 The joint dictionary for source and target language comprises 10k tokens generated with SentencePiece (Kudo& & Richardson) 2018) and preprocessed with an embedding layer (cf. preprocessing layer in Figure Appendix A. As the previous tasks, to investigate relaxed attention with LM, we trained two transformer LMs of equal size: One LM trained with WWSLI14 training transcripts and an extended LM trained on the MuST-C dataset, respectively. For both, relaxed cross attention and relaxed self-attention, the relaxation coefficient yy has been tuned on the development set. See Appendix [.333 for more details.
Results and discussion In the upper segment of Table 3 we report BLEU scores for recent transformer-baat based approaches to NMT, whereof we choose the strong approach from [Shen al. (2020) using cutoff augmentation as a bascline and report a somewhat lower BLEU score of 37.42 in our resimulation. Smooth focus here achieves comparable performance that baseline with small gains when LMs are used. We observe that a LM trained only with the target language training transcripts of the translation model yields no additional information compared the internally learned language model and thus does not improve performance for most approaches, even the relaxed cross attention that has been strong (with LM) in previous tasks. However, in case of a strong extended LM trained with additional data, relaxed cross attention (only during training again) yields the best performance of 37.96 BLEU, it suppresses the internal LM. The best performance for the common case without LMM is achieved with our relaxed self-attention approach applied during training and test, slightly outperforming the previous sfate-of-the-art BLEU score without additional training data (37.60, Shen et al. (2020)), ), with a score of 37.67, exceeding the resimulated baseline
*Code available from https:: httpss//ggthheeco///cagggaaeeeee

---

Under review as a conference paper at ICLR 2023
with LM
with extended LM (additional data) test
without LM
Approach
(training transcripts only)
test
test
Vaswani
al. [2017
34.40
Fan et al.
2020
34.50
Wu al.
2019
35.20
Wu al.
36.88
Liang a
[202
37.25
Shen al.
2020
37.60
Baseline (Shen et al. 22200) resim.)
37,42
37.42
37.62
smooth Tocus (Chorowski et al. (2015)
37.42
37.52
37.67
relaxed cross attention
37.56
37.53
37.96
matched inference
37,60
37.64
37.57
relaxed self-attention
37.57
37.49
37.74
matched inference
37.67
37.67
37.71
Table 3: Neural machine translation results terms of BLEU scores on the IWSLT14 task (DE
+ EN) using encoder-decoder transformer models with cutoff augmentation (Shen et al.) 2020). Attention relaxation is applied training only, except for "matched inference" (attention relaxation in training and test). Best results across all approaches
bold font, second best underlined.
even by 0.25 BLEU. We note, that in lyer et al. (2021) (only available as preprint) the authors also chose the model of (Shen al. (2020) as baseline but were able reproduce the result of 37.60 BLEU. They report a BLEU score of 37.78 by simply applying modified learning rate schedule achieving a somewhat smaller improvement of 0.18 BLEU absolute their baseline. Without claiming new state of the art, we note that both, our and their method are top- pop-performing on the IWSLT14 task. In Appendix [.5] also show robustness of the relaxed self-attention method towards different initialization seeds. Ablations on attention dropout and the sensitiveness of are shown in Appendices A.6 A.6 and A.7
4.4 APPLICATION IMAGE CLASSIFICATION
Task and datasets Image classification is fundamental task in computer vision aiming recog- nizing the primary content of images and differs significantly from the previous sequence-fo-seqqennc tasks as it uses a type of an attention-based encoder-only (decoder-less) transformer model, which recently dominate vision benchmarks. To investigate if relaxed attention is also applicable to such tasks. we evaluate performance in terms of classification accuracy (%) on the computationally less demanding CIFAR-100 dataset (A. Krizhevsky (2009). For each of its 100 classes, it contains 500 and 100 images for training and test, respectively, and is publicly available without a specified license. As initialization, contains 1.28M training images from I 1,000 classes and also available for research purposes upon agreement of the terms of access.
model pre-trained on the ImageNet-IK 1k dataset (Deng et al., 2009), which
Models and training For our experiments we use the vanilla Swin-T transformer model (Liu et al. 2021b as baseline-a recently established vision transformer comprising 29M parameters using localized attention. Details on the architecture (including figures) are given in Appendix A.2 For training settings follow Liu et al. (2021b). For some experiments downloaded the official ImageNet-Ik pre-trained model" and report results after fine-tuning for 100 epochs on CIFAR-100 training data. With or without pre-training, relaxed self-attention is applied only during fine-tuning. We investigate the interaction of our relaxed self-attention approach with other regularization methods by omitting already employed (i.c., the well-known stochastic depth method (Huang et (0166) or adding recently proposed (i.c., the dense relative localization loss Carloe. (Liu et al. 2021a)
approaches. For fair comparison and following common practice in (Kwon et al. al.] [021]] Liu et al.
2021a Wang et al. 2017), we report results of our relaxed self-attention approaches after roughly optimizing test accuracy with small grid search over 7 values (and o² for fuzzy relaxation after the optimal To was found) separately for each batch size (1024 and 128) with pre-training, applying the found values to experiments without pre-training See Appendix (A.3.4 for more details.
IImageNe-1k pre-trained Swin transformer models and fine-tuning code downloaded from https:// pittubb.coo//iicrrofff////ii wtwp:t//SSwnnnTaaffoooooo Transformer
8

---

Under review as a conference paper at ICLR 2023
w/ pre-training batch size
w/o pre-training batch size
Approach
1024
128
1024
128
test
test
test
test
Dosovitskiy ct al.
(002), ViT-S-16
87.10
Other transformers
Yuan ct al. 2021
T2T-ViT-14
88.40
Guo et al.
2021
CMT-S
91.70
Liu ct al.
2021b;
Swin-T (vanilla) Swin-T (+ Larloc)
88.22
53.28
Liu cl al.
2021
88,40
66.23
Bascline
(Liu et al. (2021b) resimulated)
88.53
89.16
53.12
64.10
sttcchsttic depth Huang et al. 2016
87.62
88.42
57.62
68.08
Swin-T transformers
Larloc (Liu al. (2021a)
88.63
89.27
61.72
66.29
smooth Tocus (Chorowski
(2055)
88.44
89.53
57.02
64.15
relaxed self-artention
88. 64
89.21
52.89
63.72
+ matched inference
88.73
89.39
53.15
63.52
stochastic depth (Huang et al. 2016
87.49
88.42
56,99
67.91
+ Cartee (Liu al. (021a))
88.55
89.29
61.37
65.90
+ fuzzy relaxation
88.63
89.60
52.51
63.58
Table 4: Image classification results in terms of accuracy (%) on the CIFAR-100 task using encoder- only transformer models. Relaxed self-attention applied in training only, except for "matched inference" (relaxation in training and test). All reference methods have roughly the same model size and complexity. Best results across all Swin-T approaches are in bold font, second best underlined.
Results and discussion The first segment of Table shows results for reference vision transformer models ranging from 87.10% accuracy for the pure attention-based V 16 (Dosovitskiy et al.) 2021) to 91 91.70% accuracy for the convolution attention hybrid model CMT- S (Guo et al.) 2021). The second table segment presents baselines and experimental results for Swin-T transformer models where we chose the vanilla architecture (Liu et al. 20216) to resimulate a baseline for our experiments. Omitting stochastic depth (Huang al. 2016)) causes severe loss of performance with pre-training but clearly helps when training from scratch. For the dense relative localization loss Edrloe (Liu al., 2021a), we confirm performance gains with and especially without pre-training. Smooth focus helps for the small batch size using pre-training and performs remarkably good for a large batch size when training from scratch. Without pre-training we observe that relaxed self-attention doesn't 't help. This might be due to the limited number of training epochs and slower convergence caused by the additional relaxed atteeeteniin regularization, similar to the effect of stochastic depth in the resimulated baseline. When applying relaxed attention after pre- pre-tiaining, however, relaxed self-attenti attenntnnion alone slightly outperforms the baseline but achieves even higher accuracies when used with matched inference (88.73% 88.53%) and (89.39% sizes, respectively. Matched inference turned out to be advantageous on this task in most cases, thus we continue report based thereon. Also, we note that the combination with stochastic depth seems to be beneficial for relaxed self-attention. Our new fuzzy relaxation with matched inference turns out to be useful only on smaller batch sizes after pre-training, achieving a strong accuracy of 89.60% outperforming the baseline (Liu et al. (2021b), resimulated) at 89.16%. We also investigate robustness towards different initialization seeds Appendix A.3
89. 19%) for the large and small batch
5 CONCLUSIONS
In this work we broadly explored the idea of relaxed attention for transformer architectures, simple smoothing method of the attention weights in the attention layers. We confirmed the advantage of relaxed cross attention when combined with strong external language models and introduced relaxed self-attention, thereby providing regularization also in the transformer encoder and increasing the versatility of relaxed attention to different transformer variants. We show improvements when applying relaxed attention to automatic speech recognition, lip-reading, machine translation, and image classification. On the LRS3 lip-reading task in particular achieve a word error rate of 26.31 % (vs. the former state of the art of 26.90%) as well as top-performing BLEU score of 37.67 on the IWSLT14 machine translation task.

---

Under review as a conference paper at ICLR 2023
REFERENCES
A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. https://ww..co. cs...w....... toronto edu/-kriz/cifar..he 2009.
T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman. Deep Audio-Visual Speech Recognition. IEEE Trunsactions on Puttern Analysis and Machine Intelligence (Early Access), pp. 1-11, 2018.
T. Afouras, J. S. Chung, and A. Zisserman. LRS3-TED: A Large-Scale Dataset for Visual Speech Recognition, 1809: 00496, October 2018.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. ASR is All You Need: Cross-Modal Distillation for Lip Reading. In Proc. of ICASSP, pp. 2143-2147, Barcelona, Spain, May 2020.
Kartik Audhkhasi, Yinghui Huang, Bhuvana Ramabhadran, and Pedro J Moreno. Analysis of Self- Attention Head Diversity for sefffommr-bbseed Automatic Speech Recognition. arXiv:2209.06096 September 2022.
D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Yoshua Bengio. End-to-End Attention-Based Large Vocabulary Speech Recognition. In Proc. of ICASSP, pp. 4945-4949, Shanghai, China, March 2016.
Damitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In Proc. of ICLR, pp. 1-18, San Diego, CA, USA, May 20155
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child. Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. Proc. of NeurIPS, volume 33, pp. 1877-1901, virtual, December 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection With Transformers. In Proc. of ECCV, pp. 219-229, Glasgow, UK, August 2020.
Roldano Cattoni, Mattia Antonino Di Gangi, Luisa Bentivogli, Matteo Negri, and Marco Turchi. MuST-C: A Multilingual Corpus for End-to-End Speech Translation. Computer Speech and Language, 66, October 2021.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT Evaluation Campaign. In Proc. of the IWSLT, pp. 2-17, Lake Tahoe, CA, USA, December 2014.
W. Chan, N. Jaitly, Q. Le, and Vinyals. Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition. In Proc. of ICASSP, pp. 4960-4964, Shanghai, China, March 2016.
Nanxin Chen, Piotr Zelasko, Jesus Villalba, and Najim Dehak. Focus on the Present: A Regularization Method for the ASR Source-Target Attention Layer. In Proc. of ICASSP, pp. 5994-5998, Toronto, ON, Canada, June 2021.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In Proc. of SSST, pp. 103-111, Doha, Qatar, October 2014a.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. Proc. of EMNLP, pp. 1724-1734, Doha, Qatar, October 2014b.
Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk. Kyunghyun Cho, and Yoshua Bengio. Attention-Bas Based Models for Speech Recognition. In Proc. of NIPS, pp. 577-585, Montréal, Canada, December 2015.
10

---

Under review as a conference paper at ICLR 2023
J. S. Chung, A. Nagrani, and A. Zisserman. VoxCeleb2: Deep Speaker Recognition. In Proc. of Interspeech, pp. 1086-1090, Hyderabad, India, September 2018.
Joon Son Chung and Andrew Zisserman. Lip Reading in the Wild. In Proc. of ACCV, pp. 87-103,
Taipei, Taiwan, November 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li. and Li Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In Proc. of CVPR, pp. 248 255, Miami, FL, USA, June 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In Proc. of NAACI-HILT, pp. 4171-4186,
Minneapolis, MN, USA, June 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. In Proc. of ICLR, pp. 1-21, virtual, May 2021.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing Transformer Depth on Demand With Structured Dropout. In Proc. of ICLR, pp. 1-16, virtual, April 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling Trillion Parameter Models With Simple and Efficient Sparsity. Journal of Machine Learning Reseurch, 23:2-40, 2022.
Çaglar Gülçehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On Using Monolingual Corpora in Neural Machine Translation. arXiv:/503.03555 March 2015.
Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang, CMT: Convolutional Neural Networks Meet Vision Transformers. arXiv:2107.06263. July 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proc. of CVPR, pp. 770-778, Las Vegas, NV, USA, June 2016.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep Networks With Stochastic Depth. In Proc. of ECCV, pp. 646-661, Amsterdam, Netherlands, October 2016.
Nikhil Iyer, V. Thejas, Nipun Kwatra, Ramachandran Ramjee, and Muthian Sivathanu. Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule. arXiv:2003.03977, June 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv:2001.08..0 January 2020.
Shigeki Karita, Nelson Enrique Yalta Soplin, Shinji Watanabe, Marc Delcroix, Atsunori Ogawa, and Tomohiro Nakatani. Improving Transformer-Baut End-to-End Speech Recognition With Connectionist Temporal Classification and Language Model Integration. In Proc. of Interspeech, pp. 1408-1412, Graz, Austria, September 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR, pp. 1-15, San Diego, CA, USA, May 2015.
Taku Kudo and John Richardson. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. arXiv:1808.06220 August 2018.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive Sharpness- Aware Minimization for Scale-Invariant Learning Deep Neural Networks. In Proc. of ICMI. pp- 5905-5914, 14, virtual, July 2021.
Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, and Tong Zhang. Multi-Head Attention With Disagreement Regularization. In Proc. of EMNLP, pp. 2897-2903, Brussels, Belgium, October 2018.
11

---

Under review as a conference paper at ICLR 2023
Xiaobo* Liang, Lijun* Wu, Juntao Li, Yue Wang, Qi Meng. Tao Qin, Wei Chen, Min Zhang, and Tie- Yan Liu. R-Drop: Regularized Dropout for Neural Networks. In Proc. of NeurIPS, pp. 1-21, virtual, December 2021.
Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco De Nadai. Efficient Training of Visual Transformers With Small Datasets. In Proc. of NeurlPS, pp. 1-13, virtual, December 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows, In Proc. of ICCV, pp. 10012-10022, virtual, October 2021b.
Timo Lohrenz, Patrick Schwarz, Zhengyang Li, and Tim Fingscheidt, Relaxed Attention: A Simple Method to Boost Performance of End-to-Ena Automatic Speech Recognition. In Proc. of ASRU, pp. 177-184, Cartagena, Colombia, December 2021.
Pingchuan Ma, Stavros Petridis, and Maja Pantic. End-To To-End Audio- Visual Speech Recognition With Conformers. In Proc. of ICASSP, pp. 7613-7617, Toronto, ON, Canada, June 2021.
Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, and Olivier Siohan. Recurrent Neural Network Transducer for Audio Visual Speech Recognition. In Proc. of ASRU, pp. 905-912, Singapore, Singapore, December 2019.
Erik McDermott, Hasim Sak, and Ehsan Variani. A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition. In Proc. of ASRU, pp. 434-441, Singapore, Singapore, December 2019.
Zhong Meng, Yashesh Gaur, Naoyuki Kanda, Jinyu Li, Xie Chen, Yu Wu, and Yifan Gong, In- ternal Language Model Adaptation With Text-Only Data for End-to-End Speech Recognition. 10X0v:2110.05354, March 2022.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When Does Label Smoothing Help? 17X6vv/906.02000 June 2020.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: An ASR Corpus Based on Public Domain Audio Books. Proc. of ICASSP, pp. 5206-5210, South Brisbane, QLD, Australia, April 2015.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proc. of ACL, pp. 311-318, Philadelphia, PA, USA, July 2002.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recogni- tion. In Proc. of Interspeech, pp. 2613-2617, Graz, Austria, September 2019a.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational Knowledge Distillation. In Proc. of CVPR, pp. 3967-3976, Long Beach, CA, USA, June 2019b.
Martin Popel and Ondřej Bojar. Training Tips for the Transformer Model. The Prague Bulletin of Mathematical Linguisties, 110:43-70, 2018.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. A Simple But Tough- to-Beat Data Augmentation Approach for Natural Language Understanding and Generation. arXiv:2009.1338 October 2020.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning Audio-Visii Visual Speech Representation by Masked Multimodal Cluster Prediction. In Proc. of ICLR, pp. 1-24, virtual, April 2022.
Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. Cold Fusion: Training Seq2Seq Models Together With Language Models. In Proc. of Interspeech, pp. 387-391, Hyderabad, India, September 2018.
12

---

Under review as a conference paper at ICLR 2023
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15: 1929-1958, June 2014.
Themos Stafylakis and Georgios Tzimiropoulos. Combining Residual Networks With LSTMs for Lipreading. In Proc. of Interspeech, pp. 3652-3656, Stockholm, Sweden, August 2017.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to Train Your ViT? Data, Augmentation, and Regularization Vision Transformers.
vrXiv:2106.10271 June 2021.
Zewei Sun, Shujian Huang, Xinyu Dai, and Jiajun Chen. Alleviating the Inequality of Attention Heads for Neural Machine Translation. arXiv:2009.09672, September 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. arXiv:1706.00 03762, December 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing Multi- Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proc. of ACL, pp. 5797-5808, Florence, Italy, July 2019.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang. Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual Attention Network for Image Classification. In Proc. of CVPR, pp. 3156-3164, Honulu, HI, USA, July 2017.
Yiming Wang, Tongfei Chen, Hainan Xu, Shuoyang Ding, Hang Lv, Yiwen Shao, Nanyun Peng, Lei Xie. Shinji Watanabe, and Sanjeev Khudanpur. Espresso: Fast End-to-End Neural Speech Recognition Toolkit. In Proc. of ASRU, pp. 333,143, Singapore, Singapore, December 2019,
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay Less Attention With Lightweight and Dynamic Convolutions. Proc. of ICLR, pp. 1-16, New Orleans, LA, USA, March 2019.
Zhen Wu, Lijun Wu, Qi Meng, Yingce Xia, Shufang Xie, Tao Qin, Xinyu Dai, and Tie- Yan Liu. UniDrop: A Simple Yet Effective Technique to Improve Transformer Without Extra Cost. In Proc. of NAACL-HILT, pp. 3865-3878, virtual, June 2021.
Bo Xu, Cheng Lu, Yandong Guo, and Jacob Wang. Diseriminative ModaitMy Speech Recogni- tion. In Proc. of CVPR, pp. 14421-14436 Los Alamitos, CA, USA, June 2020.
Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi Keyi Tang, Chenyang Huang. Jackie Chi Kit Cheung, Simon J.D. Prince, and Yanshuai Cao. Optimizing Deeper Transformers on Small Datasets. In Bangkok, Thailand, August 2021.
Proc. of ACL., pp.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-Toke ViT: Training Vision Transformers from Scratch on ImageNet. In Proc. of ICCV, pp. 538-547, virtual, October 2021.
Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, and Ke Xu. Scheduled DropHead: Regular- ization Method for Transformer Models. In Proc. of EMNLP, pp. 1971-1980, virtual, November 2020.
Barret Zoph, Golnaz Ghiasi, Tsung Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking Pre- Pre-Training and Self-Traini In Proc. of NeurlPS, pp. 3833-3845, virtual, December 2020.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
output token probs
Py PxfXxIX
Network
Fully Connected
D()
Softmax
8 x x
Layer Norm
Encoder
h
Decoder
BxT xxxx
Encoder Block
Decader Block
total Na decnder blocks
emooder blocks B x d
8x x
Encoder Block
Decoder Block
positional encoding
positional encoding
HxTxd
8xixd
Preprocessing
Embedding
B x x
B x
input sequence
x
previous CX-1
output token
Figure 2: Standard encoder-decoder transformer during inference used for sequence-to-ssquenee
tasks (i.e., automatic speech recognition, lip-reading, and machine translation).
A.1 MODEL SPECIFICS FOR THE SFQURNCEE-T--SQQUENC TRANSFORMER
In this section we briefly review the original transformer architecture from Vaswani et al. (007)) consisting of encoder and decoder as shown in Figure 2 Please note that here describe the transformer architecture exactly as used for the investigated sequence-to-sequee matic speech recognition, lip-reading, and machine translation), employing task-dependent individual preprocessing steps, while the encoder-only Swin transformer, used for the image classification task, separately described and shown in Appendix A.2
tasks (i.e., auto-
A.1.1 ENCODER-DECODES TRANSFORMER
The imput sequence xf of length I' (and more optional dimensions, c.g., for lip-reading: video channel, height, and width, or for ASR: acoustic feature dimension F) is entirely fed the transformer's encoder and auto-regressively transformed (by the decoder model D )) into an output token sequence ef = (c1, .2..... CL) with EC= (c(1), _!?)...., c being single output token (i.e., grapheme- based characters or (sub-) word units (Kudo & Richardson 2018) at output sequence index (1....,I) [}} from a vocabulary of size D. Specifically, the original input sequence x? first subject task-dependent preprocessing that outputs a feature sequence of E (I,..., T'} frames, optionally sub-sampled with T T. For each decoding step (starting at f= 1), the transformer decoder uses the entire encoded input sequence h and the previous output token Cz-1 to finally output vector P. = D( c comprising probabilities of all D possible output tokens. These probabilities are then subject to a beam search algorithm which, step-by-step, invokes the decoder until an end- of-sentence (EOS) threshold is exceeded and the final set of hypotheses is emitted. Considering regularization, the standard encoder-decoder transformer model employs three different variants of dropout SSriaastava et al. 2014): Residual dropout applied to sub-layer outputs before the residual connection added, activation dropout applied after the rectified linear unit (ReLU) activation, and attention dropout which is applied to the attention weights inside the MHA function (all layers, where dropout might be applied to the respective outputs, shown dashed boxes in Figures) I and 3).
14

---

Under review as a conference paper at ICLR 2023
Decoder Block
Fully Connected
Fully Connected
ReLU
1xixd
BXTx d
Layer Norm
Encoder Block
Bxlxd
Fully Connected
Z
Multi-Has Headd-Head
Fully Connected
from encoder
Attention
ReLU
x
8x1xd
Layer Norm
Layer Norm
D xx
Bxlxd
Masked Multi-
Attention
Head Attention fQ fK"
0 fK
Layer Norm
Layer Norm
B X Xxxd
Bxixd
(a) Single encoder block with self-attention.
(b) Single decoder block with cross attention.
Figure 3: Encoder and decoder blocks used in the transformer model (Figure 2p during inference. Multi-head attention blocks which may exhibil relaxed attention are colored yellow. Details thereof shown in Figure []] Layers, where dropout (Srivastava al. (014] might be applied to the outputs. are depicted as dashed-line boxes.
A.1.2 SCALED DOT-PRODUCT ATTENTION
Besides other variants of the original attention function introduced in Bahdanau et al. (2015). in this work we focus on scaled dot product multi-head attention (MHA), introduced together with the orignal encoder-decoder transformer model (Vaswani et al. 2017). As shown in Figure (without the red block), the standard MHA employs multiple (i.e., N.) attention heads
QW
(KW(K)
VW E L
Z, K, V) softmax
Vd
value projections =Y.(V)
attention weights
= G.(Q.K)
with (!Q) (kk) w(V) dx being linear projection weight matrices for the query Q, key K, and value V inputs, E =1 Nu) being the index of the in total N₁, attention heads, and d is the feature vector size being used in most layers of the transformer model often referred to as the model dimension. Considering cross attention, key and value inputs stem from the encoder's last layer, yielding K=V=hī and the entries in each of the
L rows of the attention weight matrix (Q, EI -xT with T= [o, 1), sum up to one and are treated as probabilities that correspond to the relevance of a time frame to the L-th f-th position in the query input for cross attention or self-attention, respectively. The outputs Z, of all N₂, separate attention heads are concatenated and subject fully connected output layer, yielding the MHA output Z R.xdd Note that for brevity of notation the attention dropout commonly applied to the attention weights in transformer models is not shown in equation
15

---

Under review as a conference paper at ICLR 2023
P output posterior vector
B x
Fully Connected
+ Softmax
x sC
Layer Norm
+ Avg. Pooling
8C
Swin Transformer
2x
Block
W
Patch Merging
B 16
4C 16
Swin Transformer Block
Nx
# Patch Merging
B H 20
Swin Transformer Block
B
20
Patch Merging
x
Swin Transformer Block
2x
B H C
conv (4 C),(4)
B x H x W x
x input image
Figure 4: Swin transformer as used for the image classification tasks.
A.2 MODEL SPECIFICS FOR THE VISION TRANSFORMER
As attention based model for the image classification task we employ the recently successful encoder- only transformer, dubbed the Swin transformer (Liu et al. 2021b), shown in Figure The Swin transformer is hierarchical vision transformer, which uses shifting window scheme to compute its feature representations h and can be used general purpose backbone for various vision tasks. In contrast to the sequence-based tasks, where a whole decoder is employed to yield sequential output, here, single fully connected layer with softmax activation (and preceding layer normalization and adaptive average pooling) is used after the Swin transformer blocks to assign probabilities P to the D classes for each image inside batch B.
As input, the Swin transformer receives an RGB input image x xW x3 II = [0, 1] of height H and width W. which is divided into non-overlapping image patches of size x 4 by convolutional
16

---

Under review as a conference paper at ICLR 2023
Bx kyxe
(Shifted) Window-based
Multi-Hean Attention
Reshape Window Merge"
hus B BXMMXx ay
Fully Connected
My x M x
Concatenation
Swin Transformer Block
Bx M xe/1
Matrix Multiplication
Fully Connected
G lys Bx M
Aay B Xxxxxxxff
Relaxation (3)
(h.x de
Fully Connected
G, K)
+ GELU
Softmax
Attention Weights
Layer Norm
Rpos
Mey Bx At x Ad
x (6 x ue) X
Matrix Multiplication
Aul B x xe/d
(Shifted) Handow-Based Multi- Head Attention
Vela
Ma Bxxxxxxxx xf/4
Fully Connected
Fully Connected
Fully Connected
Layer Norm
Aay B M
(/
Reshape "Window Partitioning
(a) Single Swin transformer block as used in the Swin transformer model (ef. Figure 4 dur- ing training and inference. Instead of dropout in the encoder-dccoder transformer, here we apply stochastic-deple (Huang el al. 2016) to randomly drop layers parallel to the residual connections dur- ing training. (Shifted) window-based multi-head attention blocks which may exhibit relaxed attention are colored yellow. Details thereof are shown in Figure 5b
B (w x 8c
(b) (Shifted) window-based multi-head attention (MHA) as used in the Swin transformer block Fig- ure 5a with No = 4 attention heads. The proposed relaxed attention (red block) is presented in Sec- tion A.2
Figure Details of a Swin transformer block and the (shifted) window-based meadi-head attention (MHA), where relaxed attention (red block) is applied for the image classification task.
layer with stride of (4, 4) and is thereby embedded into feature representation of dimension C. The hicrarchical structure of the Swin transformer consists then of four stages each depicted as a dashed box in Figure 4 In each stage, the patch merging modules first reduce the spatial resolution and double the feature dimensionality (n C + 2n C),, while dimensions remain constant for the subsequent processing of that specific stage through the specified number of Swin transformer blocks for that specific stage.
The Swin n transformer block, shown in Figure 5al is based on the original standard transformer block (Vaswani et al. 2017), see also Figure 3a). but replaces the ReLU activation with a Gaussian error linear unit (GELU) activation function after the first fully connected layer and, more importantly, uses (shifted) window-basee multi-head attention module, shown in Figure (bb There, the window partitioning limits the self-attention computation to non-overlapping local M M windows (M = 7), which are shifted in position every other Swin transformer block. Once the features are split into windows, they are treated as separate batch instances yielding a temporary batch size of kw B with M²Z
B being the original batch size. Different to the standard multi-head attention, a relative position bias Riss M XM² is added before softmax activation. The attention weights G. K) E T x
17

---

Under review as a conference paper at ICLR 2023
inside the shifted window-based MHA contain probabilities for relevant entries in these windows and are then subject to the herein investigated relaxation (see red box in Figure 5b), yielding
G, K) - (1-
1 M2
i No.
with M² being the fixed amount of features in single window. See Section 3 for the sequence-basee relaxed attention approach as well as for the fuzzy relaxation which randomly varies the relaxation coefficient to compensate for the now constant M term in equation
A.3 EXPERIMENTAL DETAILS
In this section we will list additional experimental details for all of the investigated tasks, thereby starting with general settings that apply to multiple tasks and then providing details for the experiments of cach individual task that has been investigated in this work. Please note that for all tasks used publicly available code as baselines and did not change any syper-parameter: unless explicitly mentioned (e.g., for ablation studies).
In experiments where an additional language model was included, we used the common shallow fusion method (Gülçehre al., 2015) for language model fusion. Specifically, shallow fusion combines the output token probability vector Pa at the output of the transformer model (cf. Figure 2 for each
decoding timestep E with the same D-length output token probabilities P (LM) in the logarithmic
domain to gather joint output token probability log P, = log Pa + À log P ((LM) The language model weight is used to steer the influence of the language model during decoding and is gathered individually for each task.
In all investigated tasks, the smooth focus method from Chorowski et (015)) is applied as reference method that smoothes the attention weights in the cross attention layer by modyifing the softmax function. Defining the scaled dot-product of query and key projections, which is input to
the softmax function in equation as E, 1
va Qw! (Q) (KW(K) = (cs,e,)) E LLXT with ex
being elements thereof, the single elements gi,e, of the attention weights G, K) with smooth focus are computed
Osig (ei.e.)
gr (Q, K)
(4)
Σn Ssig (Ci,e.)
with asig being the sigmoid function, which for smooth focus replaces the unbounded exponential function from the standard softmax function. Please note that for the Swin transformer the softmax
input can be defined analogously as E, Rpos + ve/4
Qw(
(KW!K)
MMM
A.3.1 AUTOMATIC SPEECH RECOGNITION
The specific model architecture for the trainings with 100 and 960 of training data, stan- dard encoder-decoder transformer models with a small =N.=N,=6, N₁, 44, d15)2) and a large (N.= 12, =6, Ni, =4, d=512) configuration, respectively, with Ne. Na. and Nn being the number of encoder blocks, decoder blocks, and attention heads, respectively, and d is the model dimension (i.e., the amount of nodes used most layers of the model). The external RNN language model con- sists of shared input and output embedding layers with four LSTM layer in between each comprising 800 nodes yielding total of 24.5M parameters.
Training of ASR models was done using the espresso toolkit, which is an extension of the PyTorch- based fai rseq toolkit. We performed a small grid search on the joint dev clean and dev other datasets among values {0.0001, 0.001, 0.01, 0.05, 0.1} and (0.1, (0.1,0.15 0.15. 0.2, 0.25, 0.3) for the relax- ation coefficient and found = 0.01 and 7 = 0.25 to be optimal for relaxed self-attention and relaxed cross attention, respectively. Optimal values were used for both, 100h h and 960 training data configurations. All remaining training hyper-parameters parameters were adopted from the recipes available train all
at h/thttp::/// /ttt::://gituutet cwm/ f htw////eeweeeeeeeee hettym/mmeppssssooo with commit id 390ad6f. Specifically, models for 100 epochs using the Adam optimizer (Kingma & Ba (015)) with a learning rate of 0.001. All dropout layers (i.e., residual, activation, and attention dropout) used the dropout rate p=0.2 and
18

---

Under review as a conference paper at ICLR 2023
the label smoothing coefficient set to 0.1. Models for 100 of training data were trained using a single RTX2080ti GPU, while larger models on 960 h of training data were trained on a single A100 GPU.
Inference was done using beam search with beam size of 60 and the language model weight À was fixed at 0.4, following recipes from Wang et al. (2019) for all experiments with LM, without further optimization.
A.3.2 LIP-READING
The specific model architecture for fine-tuning with 30 of labeled data, is pre-trained base AV AV-HUBERT encoder model with appended standard transformer decoder model (N,=12, Na =6, N =1, 768) while for the 433 and 433 h + 1,326 setups a large AV-HuBERT encoder with larger decoder was used 24, 99 N₂. 16, d= 1024).) The external transformer language model comprises 16 decoder blocks with = 512 (cf. Figure 3b. but without the cross attention layer) and uses shared input/output embedding of the in total D 1000 subword units, resulting in a language model size of 51M parameters.
Training of lip-reading models done using the PyTorch-based based fairseq toolkit. We performed a small grid search on the development dataset among values (0.005, 0.01, 0.02, 0.05, 0.1} and (0.1, 0.150, 0.2, 0.25, 0.33 for the relaxation coefficient of relaxed self-attention and relaxed cross attention, respectively. For the 30 fine-tuning case we found 7 0.001 and 7: = 0.25, for 433h we found y 0.005 and y = 0.25, and for the 433 1,366 case we found 7 = 0.005 and y to be optimal. All remaining training parameeass were adopted from the recipes available ltttps://githbb com/ facebookresearchhccccc av hubert hubert with commit id cd1fd24. Residual, activation, and attention dropout layers were using dropout rate of 0.1, 0.1, and 0.0, respectively. The label smoothing coefficient was set to a 0.1. Models for the smaller 3 fine-tuning data setup were trained using single RTX3080 GPU, while for all other settings single A100 GPU was used for training.
Inference was done using beam search with beam size of 50 and the language model weight À was optimzied for each approach by searching optimal values on the development data among values of (0.05, 0.1, 0.15, 0.2).
A.3.3 MACHINE TRANSLATION
The standard meoder-decoder transformer from Vaswani et al. (2017) was used in the base config- =4, input and output embedding layers of the in total D = 10000 subword units with 6 decoder blocks (cf. Figure 3b but without the cross attention layer) in between and comprises 24. 1M parameters.
uration =N, =4, d12))). The external transtormer language model consists of shared
Training of the machine translation transformer models was done using the PyTorch-based fairseq toolkit. We performed a small grid search on the development dataset among val- ues {0.005, 0.01, 0.02, 0.05, 0.1} and {0.1, 0.15, 0.2, 0.25, 0.3} for the relaxation coefficient and found 7 = 0.05 and y = 0.1 optimal for relaxed self-attention and relaxed cross attention, re- spectively. For approaches with LM the language model weight À was tuned among values (0. 0.05, 0.1, 0.15, 0.2). All remaining training hyper-parameters were adopted from the recipes available at /tttts:://gthub.. cwm/dit diinhanshen/Cutoff.. with commit id 4978563. Residual, ac- tivation, and attention dropout layers were set to 0.3, 0.1, and 0.1, respectively. All models were trained using a single RTX2080ti GPU.
Inference was done using beam search with beam size of 10 and the language model weight À was optimized for each approach by searching optimal values on the development dataset among values of (0.05, 0.1. 0.15, 0.2
A.3.4 IMAGE CLASSIFICATION
We chose the Swin transformer as the specific model architecture for trainings with and without pre-training. It is multi purpose backbone for various vision tasks and can be configured in terms
of size and complexity. Specifically, we use the tiny configuration of the model dubbed Swin-T. T, which is defined by an initial feature embedding dimensionality C = 96 and comprises N 6
19

---

Under review as a conference paper at ICLR 2023
Swin transformer blocks in the third stage, resulting in total of Ne- 12 Swin transformer blocks. The number of attention heads Nr doubles with each consecutive stage, yielding an amount of {3, 6, 12,11,2,20 24) attention heads for the respective stages. In total, the model comprises 29M parameters.
Training of image classification models was done using the PyTorch toolkit. We performed a small grid search among values {0.005, 0.01, 0.05, 0.1, 0.15, 0.2) and {0.01, 0.02, 0.03) for the relaxation coefficient of relaxed self-attention and o² for fuzzy relaxation, respectively. Following common practice on the CIFAR-100 task (see, e.g., Kwon et al. (0221): Liu al. (2021a): Wang et al. (0177), parameter search was conducted on the test dataset. For the training with pre- training we found ? 0.1 and 0.03 to be optimal. Both found relaxation hyper-parameters were also applied for experiments without pre- tre-training All remaining training parameeersseeerss Transformer
were adopted from the recipes available at https:: /ttp://gghhhbb.../ com/micmmmcccmc httm//mccooooof/tt///c/ with commit id 5d2aede. For some trainings we use an auxiliary dense relative localization loss ferloo- which encourages vision transformers to learn spatial information between image patches and thereby boosts convergence, especially for small datasets (Liu et al. 202 For the Ldrloc loss, adopted the official Swin-based code from ht : /ttgg/gttuubbbb. com/ yh Leo/ Doc with commit id b69adb6. Specifically we train all models for 100 epochs using the Adam optimizer (Kingma & Ba 2013) with a learning rate of 0.000125 for a batch size of 128 and 0.001 for batch size of 1024. Stochastic depth (Huang et al., 2016), which randomly drops layers in the transformer block, is standard method for training the baseline model and was used with a drop probability of 0.2. Label smoothing was used with a value of 0.1. All Swin transformer models were trained using a single RTX2080ti GPU.
A.4 INTERNAL LANGUAGE MODEL SUPPRESSION
As shown in Table I for automatic speech recognition, we achieved superior results with relaxed cross attention only when the transformer was combined with an external language model that trained with large amounts of additional text-only data. This finding is in line with Lohrenz et al. (202T). but Lohrenz et al. (202I) does not provide a sound reason for such behavior. Different to hybrid ASR approaches, the output token posterior Py of trained transformer model cannot technically be decomposed into an acoustic model P(x] [cf)) and language model P(cf).. since the latter is also implicitly learned on the training transcripts by the transformer decoder that in addition to the encoder output autoregressively receives previous output tokens as it is the case for language models.
absolute WER
LM-induced WER reduction
dev
test
dev
test
Approach
clean other
clean other
clean other
clean other
Baseline (Lohrenz et al. (2021). resim.), no LM
3.92
9.00
4.47
9.23
LM (training transcripts only)
3.92
8.90
4.44
9.20
0.00
0.10
0.03
0.03
LM (additional data, from Tab.
3.73
8.52
4.40
8.95
0.19
0.48
0.07
0.28
Relaxed cross attention, no LM
3.95
9.33
4.28
9.45
LM (training transcripts only)
3.91
9.26
4.23
9.30
0.04
0.07
0.05
0.15
LM (additional data, from Tab.
3.44
7.74
3.58
8.35
0.51
1.59
0.70
1.10
Table 5: Automatic speech recognition results in terms of WER (%) on the Librispeech task using standard encoder-decoder transformer models. The 960 h training dataset is used, also Table
Here, we investigate whether the improvement by relaxed cross attention might be due to a suppression of the internal language model. To accomplish this, in Table[5] we measure the WER improvement achieved by using an LM when the transformer was trained with and without relaxed cross attention, respectively. Both trained transformer models are combined with two language models, one trained only from the text transcripts of the acoustic training data, and one trained with additional text-only data. Note that both, resimulated baseline results and results for the LM with additional data, are taken from Table [ We observe that for both, the baseline and the relaxed cross attention model, the improvements with the training transcript only LM (rows and 5) vs. the no LM methods are about equally small. In contrast, when combined with the LM model trained model trained with relaxed cross attention yields far more WER reduction as if this strong LM would be used with the baseline. In any case it exceeds absolute reduction of 0.5% (nowhere reached
additional data, the
20

---

Under review as a conference paper at ICLR 2023
with the baseline), and for the (noisy) other condition it is more than 1% absolute WER reduction if relaxed cross attention is employed.
LM-induced WER reduction
absolute WER
Approach
dev
test
dev
test
Baseline (Shi et al. (2022), resimulated), no LM
17.71
26.73
LM (training transcripts only)
17.83
27.22
-0.12
-0.49
+ LM (additional data, from Tab.
17.18
26.50
0.53
0.23
Relaxed cross attention, no LM
17.40
26.57
+ LM (training transcripts only)
17.48
26.52
-0.08
0.05
+ LM (additional data, from Tab.
16.92
25.51
0.48
1.01
Table 6: Automatic lip-reading results in terms of WER (%) on the LRS3 task using standard encoder-decoder transformer models with pre-trained AV-HuBERT encoders. For fine-tuning 433 h of labeled data are used, also Table
For the automatic lip-reading task we observe similar behavior in Table 6 Here the integration of the training transcripts only LM is even harmful for the baseline model (row 2), while for the relaxed cross attention approach, WERs remain roughly the same compared to the relaxed cross autention-traine model without LM (row vs. 5). combination with the strong LM, both baseline and relaxed cross attention models take profit the dev set, while on the test set, relaxed cross attention yields a more than four-fold WER reduction by LM fusion (1.01% % absolute) compared to the baseline approach (0.23% absolute).
Overall, we observe that relaxed cross attention does not yet help when the LM was trained only with the text transcript data that was already exposed to the ASR transformer model during training of acoustic data. We conclude, however, that relaxed cross attention particularly helps when the LM has been trained with additional text data and seems suppress the internal model bias, thus suppressing the influence of the internally learned (usually poor) language model. Note that the same behavior is observed in for neural machine translation.
A.5 ROBUSTNESS TO DIFFERENT INITIALIZATION SEEDS
In Table investigate the influence of different initialization seeds our experiments. While for the main experiments in Section 4 we experimented on an unchanged and non-optimized seed for random number generation, here since both of our SOTA contributions are based the novel self- attention- weeettonn-wewwoee analyze the best relaxed self-attention schemes of each task w.r.t statistical significance when using different random seeds.
Automatic
Automatic Lip-Reading (LRS3)
Machine Trunslation (IWSLT14)
Image Classification (CIFAR-100)
Task
Speech Recognition (Librispeech)
100h training data
433 h 1,326 h labeled data w/o LM
w/o LM, matched inference
w/ pre-training, batchsize 128 fuzzy relaxation
Setting
w/o LM
Metric
WER (%)
WER (%)
BLEU
Acc. (%)
Data subset
test clean
test other
test
test
test
Baseline (resimulated)
14.89
29.66 - 0.34
26.9210.21
37.19-1.100 37.66+0.02
19.19
Relaxed self-attention 1..29
2..5
0.22
89.45
Table 7: Sensitivity to different initialization of the various tasks. Training of the models for the baseline and best relaxed self-attention approach was repeated 5 times. Results are shown in terms of average and standard deviation values of the respective metrics.
We note that these experiments, we achieve significant improvement for all three sequence-base based tasks including those where we claim Mate-of-theean and top performance (ie.. lip-reading and
21

---

Under review as a conference paper at ICLR 2023
machine translation). In addition, not shown here, the relaxed cross attention method yielded even better performance on all three sequence-base based tasks, outperforming relaxed self-attention, but we do not formulate performance claims in this particular analysis it implies extra computational complexity due to the requirement of a language model as well as additional unpaired text training data. For the image classification task, note that reach clear improvement using the non- optimized standard seed for initialization of our main experiments (see Table 4 Here, however, with additional seeds for initialization, we observe the baseline and the fuzzy relaxation approach to differ without statistical significance. We suspect this is due to non-deterministie operations in the original baseline code from [iu (202Tb), which might have flawed the tuning process for the relaxation coefficients for fuzzy relaxation. However, as the average accuracy with fuzzy relaxation still bigher (89.45 65%.5%% vs. 89. 89.29%), we feel encouraged further expand the relaxed self-attention approach to attention-based approaches for computer vision tasks.
A.6 ABLATION STUDY ON ATTENTION DROPOUT
Depicted as dashed boxes in Figures [ and the well-known dropout method (Srivastava al. 2014) is employed to the standard encoder-decoder transformer in three different variations: Residual dropout, activation dropout, and-most relevant for our study-attention dropout, where the latter is either applied to the attention weights G, after the softmax layer (baseline) or to the modified attention weights G, after the relaxation operation (relaxed attention approaches, equation (.). In Table 8 we investigate how these regularization operations interfere with each other for two different tasks that incorporate attention dropout during training. Therefore, in this ablation, removed attention dropout throughout the encoder and the decoder of the transformer model for both approaches with and without specified types of relaxed attention. Note that the employed models for the lip-reading and image recognition tasks did not use attention dropout (following the respective baseline recipes from Shi et al. (2022) and Liu al. (202Tb), see experimental details in Appendices A.3.2 and A.3.4) and are thus omitted for this ablation. Specific values for attention dropout are given for each task Appendix A.3
Automatic Speech Recognitton (Librispeech)
Machine Translation (IWSLT14)
Task
w/ LM 100 h training data
Setting
w/o LM,
self-attention matched inference
Relaxation type (*)
cross attention
Metric
WER (%)
BLEU f
dev clean
dev other
test clean
test
Data subset
other
test
Baseline (resimulated)
10.62
24.19
12.06
25.56
37.42
attention dropout
11.02
25.24
11.89
26.88
37.51
relaxed (*) attention
9.33
22.16
10.62
23.04
37.67
alttetion dropout
9.68
21.38
10.91
23.16
37.47
Table 8: Ablation study on attention dropout (Srivastava et al. recognition and neural machine translation tasks. Best results across approaches are in bold font and arrows (L1) point to the direction of better metric values for each task.
2014) for exemplary automatic speech
We note that relaxed attention nicely combines with attention dropout (Srivastava et al., 2014) as in the test conditions of both tasks the combination of relaxed self-/cross attention with attention dropout yields the best results, which are also reported in the main experiments for both specific tasks in Section 4 Interestingly, attention dropout did even harm the baseline performance for machine translation, as omitting it yields an 0.09 absolute increase in BLEU score, while improves the advantageous relaxed self-attention even further. In summary, observe that both proposed relaxed attention approaches seem to go along with other regularization approaches, such as attention dropout, providing complementary regularization the attention layers.
22

---

Under review as a conference paper at ICLR 2023
A.7 SENSITIVENESS OF RELAXATION COEFFICIENT Y
In Figures 6 and 7 we investigate the sensitiveness of the relaxation coefficient for the automatic speech recognition and for the neural machine translation task respectively. As introduced in Section 3 the constant relaxation coefficient uniform distribution to the unmodified attention weights over the temporal dimension of the input sequence. For both exemplary tasks we investigate the influence of y for relaxing either the encoder self-attention layers (Figures 6a and 7ab or the decoder cross attention layers (Figures 66b and 7b
[0, 1] is single hyperparameter to control the addition of an
22
17.5
Word crror rate (%)
Word emorr rate (%)
21.5
17
21
16.5
20.5
16
20
15.5
.0001 .001
.01
.05
.1
.2
.3
0
.05
.15
.2
.25
.3
Relaxation coefficient
Relaxation coefficient
(a) Relaxed self-attention, without LMM
(b) Relaxed cross attention, with LM
Figure 6: Sensitiveness of the automatic speech recognition results with respect the relaxation coefficient y in terms of combined WER (%) on the joint clean and other portions of the dev dataset of the Librispeech task.
38.7
39.5
BLEU score
38.6
BLEU score
39.4
38.5
39.3
38.4
39.2
38.3
39.1
0
.05
.1
.15
.25
Relaxation coefficient 7
Relaxation coefficient y
(a) Relaxed self-attention, w/o LM,, matched in-
(b) Relaxed cross attention, with LM
fer.
Figure Sensitiveness of the neural machine translation results with respect to the relaxation coefficient in terms of BLEU scores the development dataset of the IWSLT14 task (DE-) EN).
Both, Figures 6 and 7 show the task-specific performance on the respective development sets, that were used for optimization of the 7 hyperparameter. In all shown cases, we make the following observations: (i) While relaxed self-attention performs best with smaller y values, relaxed cross attention reaches best performance with somewhat higher values, (ii) there a smooth and substantial range where relaxed self- and cross attention improves over the resimulated baselines with 0 thus showing that the contribution our method is insensitive with respect to the choice of y in these ranges.
A.8 STATEMENT ON POTENTIAL NEGATIVE SOCIETAL IMPACTS
Our method itself applies to the general transformer model and is is-as we have demonstrated- applicable to a variety applications. Out these applications, we identify that automatic lip- reading can be used for malicious purposes such cavesdropping on private civilian conversation in video surveillance footage. The dataset we use for automatic lip-reading consists of professionally recorded speakers that are of being recorded,
close distance, have well illuminated faces
23

---

Under review as a conference paper at ICLR 2023
while speaking, and are mostly facing towards the camera. These conditions very unlikely in malicious surveillance scenario where it is unlikely that the methods and models we developed in our work are of large benefit. In addition, we believe that the positive impact of lip-reading applications clearly outweighs the possible negative applications. Examples of such applications are (i) improving speech recognition in case audio is corrupted, (ii) helping in crime investigations, (iii) enabling people suffering from aphonia to communicate, (iv) silence dictations, and (v) uttering silent alarms or passphrases for security.
24


Reviews:
[{"review_id": "b4mptTl3He", "reviewer": "Reviewer_2CCe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed to use relaxed attention in both self attention and cross attention part of transformer models. In details, the authors proposed to add a smoothing term on the cross attention module with a fuzzy realaxation coefficient drawn from a Gaussian distribution. Extensive experiments were done on automatic speech recognition, lip-reading, machine translation, and image classification.", "summary_of_the_review": "Generally I think relaxed attention is an interesting idea, as it is simple and easy to apply in multiple applications. However, the efficiency of the technique is supported by experimental results, which is not significant enough as the improvements are marginal and metrics' variance between runs are not provided.", "strength_and_weaknesses": "Strength:\n1. Simple techniques apply to wide range of application.\n2. Extensive experiments.\n\nWeakness\n1. Novelty. The method was first proposed in [1] as the authors cited in paper.\n2. The experimental results are not strong enough as the improvement seems marginal. And also lack of variance on all results. For the results with less significant improvement, it would better to also show the variance among runs.\n\n[1] TimoLohrenz,PatrickSchwarz,ZhengyangLi,andTimFingscheidt.RelaxedAttention:ASimple Method to Boost Performance of End-to-End Automatic Speech Recognition. InProc. of ASRU, pp. 177–184, Cartagena, Colombia, December 2021. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G6K07Fs1rE", "reviewer": "Reviewer_WiHX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores relaxed attention in transformer models. The relaxed attention interpolates the attention weights with a uniform distribution. This method was proposed by previous work, and this paper extends its application from cross attention to all attention use cases (cross, self, causal). The paper further proposes a variant that randomly draws the interpolating coefficients at training time.\n\nThe experiments cover a variety of tasks, including automatic speech recognition, lip reading, machine translation, and image classification. On some, the results are mixed; on others, the improvements are marginal. \n\n", "summary_of_the_review": "The paper did a good job in terms of covering a variety of tasks. But it lacks depth at the same time. Given that the technical contribution is thin compared to average ICLR papers, I suggest that the authors should dive deeper into the core assumption of the paper: smooth attention is better in some applications.", "strength_and_weaknesses": "Strength\n- Clear presentation\n- The method is straightforward\n- Experiments cover a variety of tasks\n\nWeaknesses\n- Extending an existing technique to self and causal attention is thin in terms of technical contribution\n- No evidence is provided to support the significance of the improvements (if any)\n- Lacks in-depth analysis and discussion of the method\n- An important baseline is missing: smoothing the attention weights using a larger temperature\n- The “relaxed attention” naming is unfortunate—it leaves me the impression that it is lifting some constraints in conventional attention (e.g., simplex). I suggest changing it to something related to “smoothed” or “flat.”\n\nDetails\n- The improvements in the MT experiments seem very marginal—the ~0.2 BLEU delta is definitely reachable with a lucky random seed; I suspect this might be the case in other tasks too (MT is the only one I’m familiar with). The paper can benefit from training multiple models with different random seeds, and/or presenting significance tests.\n- It is great to cover a diverse set of tasks, but the paper favors width over depth. I would really appreciate it if the paper could include some in-depth analysis and discussion of the method. For example, how does the smoothing term affect the “flatness” of attention? Comparing the entropy of the attention weights w/ and w/o the uniform term is a good starting point. Performance change over different choices of gamma is another good one. Some discussion/analysis on why smoother attention is better is also interesting. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G-7jhQDIaXa", "reviewer": "Reviewer_ztHw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper conducts detailed study on relaxed attention for transformer. The relaxed attention mechanism is applying smoothing to the attention weights. Previous work which proposed relaxed attention only applied it to cross-attention in the decoder. This work studies applying relaxed attention to self-attention layer in the encoder. The authors suggest two benefits: 1. the relaxed attention provides regularization when applied to the self-attention layer. 2. the relaxed attention applied to the cross-attention layer suppresses the intrinsic language model and allows better integration of an external language model. The experiments are conducted in automatic speech recognition (ASR), lip-reading, machine translation, and image classification. The work shows better or on par performance with state-of-the-art models on lip-reading LRS3 benchmark (-0.6% in word error rate) and machine translation IWSLT14(DE->EN) (+0.18 BLEU). ", "summary_of_the_review": "The thorough experiments confirm the usefulness of an previously proposed technique that smooths the attention weights.", "strength_and_weaknesses": "Strength:\n1. The experiments are detailed and cover multiple tasks/domains (speech, translation, and image).\n\nWeakness:\nMinor sentence fluency:\n1. page 2: \"Very early once ...\"\n2. page 9 in Conclusions: \"thereby regularizing already in the encoder\"\n\nQuestion:\n  - Have you tried applying the relaxed attention in the pretraining stage?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FC4YMc5hNTN", "reviewer": "Reviewer_4P4f", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the relaxed attention method on the Transformer architecture, across a variety of tasks including automatic speech recognition, lip reading, machine translation, image classification. The technical contributions include exploring relaxed attention on the self-attention module, utilizing it on both training and inference, and introducing variation into image classification where the sequence length is fixed. ", "summary_of_the_review": "I believe the novelty of the paper is straightforward, but the contribution is limited.", "strength_and_weaknesses": "**Strengths:**\n\n1. The paper studies various tasks including on text, speech and image, which guarantees that the conclusions are general and robust.\n2. The paper is generally well-written and easy to follow.\n\n\n**Weaknesses:**\n1. The contribution is somewhat incremental. Extending relaxed attention from cross-attention to self-attention is straightforward, the paper shows that it will bring some improvements to the model performance, but lacks showing the novelty of doing so. For example,  what it brings to the training/inference of the model? Does it solve some problems that the original attention has? And how the relaxed attention achieves that? In my experience, the sparsity in self-attention is natural and provides clear local context information to the model, then what the smoothing regularization term brings to the model?\n2. Another contribution, the matched inference, does not show consistent improvement across different tasks. The paper does not provide a detailed analysis of the results which also limits the novelty. If it cannot bring improvements and no in-depth explanation is conducted, then why take it as one of the contributions?\n3. The method section is too short to provide the details and insights of the proposed method.\n4. The experimental settings lack some details. For example, what is the difference between the clean and other settings in the ASR task? In the “Approach” column in each table, is the relaxed self-attention added on top of relaxed cross-attention or they are separately enabled? In the machine translation tasks, have you conducted experiments on larger benchmark datasets such as WMT which will make the conclusions more faithful?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject