Cross-Layer Retrospective Retrieving via Layer Attention
Authors: Yanwen Fang
Yuxi CAI
Jintai Chen
Jingyu Zhao
Guangjian Tian
Guodong Li
Venue: ICLR 2023 poster (2023)
Keywords: Layer Attention
Recurrent Layer Attention
Layer Interaction
CNNs
Vision Transformers
Vision Networks

Abstract: More and more evidence has shown that strengthening layer interactions can enhance the representation power of a deep neural network, while self-attention excels at learning interdependencies by retrieving query-activated information. Motivated by this, we devise a cross-layer attention mechanism, called multi-head recurrent layer attention (MRLA), that sends a query representation of the current layer to all previous layers to retrieve query-related information from different levels of receptive fields. A light-weighted version of MRLA is also proposed to reduce the quadratic computation cost. The proposed layer attention mechanism can enrich the representation power of many state-of-the-art vision networks, including CNNs and vision transformers. Its effectiveness has been extensively evaluated in image classification, object detection and instance segmentation tasks, where improvements can be consistently observed. For example, our MRLA can improve 1.6% Top-1 accuracy on ResNet-50, while only introducing 0.16M parameters and 0.07B FLOPs. Surprisingly, it can boost the performances by a large margin of 3-4% box AP and mask AP in dense prediction tasks. Our code is available at https://github.com/joyfang1106/MRLA.

Published as a conference paper at ICLR 2023
CROSS-LAYER RETROSPECTIVE RETRIEVING VIA LAYER ATTENTION
Yanwen Fang"), Yuxi Cai Jintai Chen', Jingyu Zhao', Guangjian Tian', Guodong Lit
Department of Statisties & Actuarial Science, The University of Hong Kong
College of Computer Science and Technology, Zhejiang University
'Huawei Noah's Ark Lab
fu3545683, caiyuxi, g jtigerchenezju. edu.. cn, Tian .aaanGuun Guangj Lanhuaawe..n com gdli@hku.hr hk
aku.hk
ABSTRACT
More and more evidence has shown that strengthening layer interactions can enhance the representation power of deep neural network, while self-attention excels at learning interdependencies by retrieving query-activated information. Motivated by this, we devise cross-layer attention mechanism, called multi-head recurrent layer attention (MRLA), that sends a query representation of the current layer to all previous layers to retrieve query-related information from different levels of receptive fields. A weiih-we version of MRLA is also proposed to reduce the quadratic computation cost. The proposed layer attention mechanism can enrich the representation power of many state-of-the-art vision networks, including CNNs and vision transformers. Its effectiveness has been extensively evaluated in image classification, object detection and instance segmentation tasks, where improvements can consistently observed. For example, our MRLA can improve 1.6% Top-1 I accuracy on ResNet-50, while only introducing 0. 16M parameters and 0.07B FLOPs. Surprisingly, it can boost the performances by a large margin 3-4% box AP and mask AP in dense prediction tasks. Our code is available at https:/githubb.oo/joyfaaag 106/MRLA.
INTRODUCTION
Growing evidence indicates that strengthening layer interactions can encourage the information flow of deep neural network (He et al. 2016 (Hanng et al. 2017 Zhao et al. (02T). For example, in vision networks, the receptive fields are usually enlarged as layers are stacked. These hierarchical receptive fields play different roles in extracting features: local texture features are captured by small receptive fields, while global semantic features are captured by large receptive fields. Hence encouraging layer interactions can enhance the representation power of networks by combining different levels of features. Previous empirical studies also support the necessity of building interdependencies across layers. ResNet (He et al. 2016) proposed to add skip connection between two consecutive layers. DenseNet (Huang et al., 2017) further reinforced layer interactions by making layers accessible to all subsequent layers within a stage. Recently, GLOM (Hinton] (02TT) adopted an intensely interacted architecture that includes bottom-up. top-down, and same-level interactions, attempting represent whol- hierarchies in a neural network.
In the meantime, the attention mechanism has proven itself in learning interdependencies by retrieving query-activated information in deep neural networks. Current works about attention lay much emphasis on amplifying interactions within a layer (Hu et al. 2018 Woo et al. 2018 Dosovitskiy al. 202T). They implement attention channels, spatial locations, and patches; however, none of them consider attention on layers, which are actually the higher-level features a network.
is then natural to ask: "Can attention replicate its success in strengthening layer interactions?" This paper gives positive answer. Specifically, starting from the vanilla attention, we first give formal
Authors contributed equally. Correspondence gdli hkk.hk.

---

Published as a conference paper at ICLR 2023
stage
1.
(K pr-
2
0.6
Quay Inddr
Viak
0.2
Kay Index
16)
Kay Irdex
180
Figure 1: Visualiizniii of the layer attention scores from a randomly chosen head of MRLA in each stage of 50sMeL-50-MRRA model; (b) Schematic diagram of two consecutive layers with RLA.
definition of layer attention. Under this definition, a query representation of the current layer is sent to all previous layers to retrieve related information from hierarchical receptive fields. The resulting attention scores concretely depict the cross-layer dependencies, which also quantify the importance of hierarchical information to the query layer. Furthermore, utilizing the sequential structure of networks, we suggest a way to perform layer attention recurrently in Section 3.3 and call it recurrent layer attention (RLA).. A multi-head design is naturally introduced diversify representation subspaces, and hence comes multi-head RLA (MRLA). Figure aa) visualizes the layer attention scores yielded by MRLA Eq. (6). Interestingly, most layers pay more attention to the first layer within the stage, verifying our motivation for retrospectively retrieving information.
Inheriting from the vanilla attention, MRLA has a quadratic complexity of O(T"), where T is the depth of network. When applied to very deep networks, this will ineur high computation cost and possibly the out-of-memory problem. To mitigate the issues, this paper makes an attempt to devise light-weighted version of MRLA with linear complexity of (T)) After imposing linearized approximation, MRLA becomes more efficient and has a broader sphere of applications.
To our best knowledge, our work is the first attempt to systematically study tross-layer dependencies via attention. It different from the information aggregation in DenseNet because the latter aggregates all previous layers' features in a channel-wise way regardless of which layer a feature comes from. OmniNet (Tay et al. 2021) and ACLA (Wang al., 2022b) follow the same spirit as DensetNet. They allow each token from each layer to attend to all tokens from all previous layers. Essentially, both of them neglect the layer identity of each token. By contrast, we stand upon the layer attention to retrospectively retrieve query-related features from previous layers. Besides, to bypass the high computation cost, OmniNet divides the network into several partitions and inserts the omnidirectional attention block only after the last layer of each partition; and ACLA samples tokens with gates from each layer. Instead, our light-weighted version MRLA can be easily applied to each layer.
The two versions of MRLA can improve many state-of-the-an (SOTA) vision networks, such as convolutional neural networks (CNNs) and vision transformers. We have conducted extensive experi- ments across various tasks, including image classification, object detection and instance segmentation. The experiment results show that our MRLA performs favorably against its counterparts. Especially in dense prediction tasks, it can outperform other SOTA networks by a large margin. The visualiza- tions (see Appendix B.5) show that our MRLA can retrieve local texture features with positional information from previous layers, which may account for its remarkable success in dense prediction.
The main contributions of this paper are summarized below: (1) A novel layer attention, MRLA, is proposed to strengthen cross-layer interactions by retrieving query-related information from previous layers. (2) light-weighted version of MRLA with linear complexity is further devised to make cross-layer attention feasible to more deep networks. (3) We show that MRLA is compatible with many networks, and validate its effectiveness across a broad range of tasks on benchmark datasets. (4) We investigate the important design elements of our MRLA block through ablation study and provide guidelines for its applications on convolutional and transformer bansformer-based vision models.
2 RELATED WORK
Layer Interaction Apart from the works mentioned above, other CNN-based and transformer- based models also put much effort into strengthening layer interactions. DIANet (Huang et al. 2020) utilized parameter-shate with the help of previous layers' information. CN-CNN (Guo et al.) 2022) combined DIANet's LSTM and spatial and channel attention for feature fusion across layers. A similar RNN module was applied
LSTM along the network depth to model the cross-channel relationships
2

---

Published as a conference paper at ICLR 2023
along the network depth to recurrently aggregate layer-wise information in RLANet(Zhao al. 2021). To distinguish it from our RLA, we rename the former as 2 in the following. RealFormer (He et al., (021)) and EA- Tr--rrnsfrre (Wang et al.] 202T) both added attention scores in the previous layer to the current one, connecting the layers by residual attention. [Bapna et al. (2018) modified the
encoder-decoder layers by letting the decoders attend to all encoder layers. However, maintaining the
features from all encoders suffers from a high memory cost, especially for high-dimenniioall features.
Attention Mechanism in CNNs CNNs have dominated vision tasks by serving as backbone networks in the past decades. Recently, attention mechanisms have been incorporated into CNNs with large receptive fields to capture long-range dependencies. SENet (Hu et al. 2018) and ECANet (Wang) 2020b) are two typical channel attention modules, which adaptively recalibrated channel- features by modelling the cross-channel dependencies. The Squeeze-ande axxeiitatioonnn (SE) block in
et al.
SENet was later employed by the architectures of MobileNetV3 (Howard al., 2019) and EfficientNet
(Tan & Le 2019). CBAM (Woo et al. 2018) first combined channel and spatial attention to emphasize meaningful features along the two principal dimensions. Pixel-level pairwise interactions across all spatial positions were captured by the non-local (NL) block in NLNet (Wang et al., (018)) GCNet (Cao et 2019) simplified the query-specific operation in the NL block to a query-independent one while maintaining the performances. CANet (Li et al.) 2021) also extended the NI. block for small object detection and integrated different layers' features by resizing and averaging. TDAM (Jaiswal itt al. 2022) and BANet (Zhao et al. 2022) both perform joint attention on low- and high-level feature maps within a convolution block, which are considered inner-layer attention in our paper.
Transformer-based Vision Networks Motivated by the success of Transformer (Vaswani et al. 2017) in Natural Language Processing (NIP),, many researchers have applied transformer- architectures to vision domains. The first is ViT (Dosovitskiy et al. (00T)) which adapts a standard convolution- Transformer to image classification by embedding an image into a sequence of patches. However, it relies large-scale pre-training to perform comparably with SOTA CNNs. This issue can be mitigated by introducing an inductive bias that the original Transformer lacks. DeiT (Touvron et al. 2021 adopted the knowledge distillation procedure to learn the inductive bias from CNN teacher model. As a result, it only needs to be trained on a middle-size dataset, ImageNet- IK, and achieves competitive results CNNs. CeiT (Yuan et al. 2021) introduced convolutions the patch embedding process and the feed-forward network of VTT. Swin Transformer (Liu et al. 2021 employed a hicrarchical design and shifted- window strategy to imitate a CNN-based model.
LAYER ATTENTION AND RECURRENT LAYER ATTENTION
This section first recalls the mathematical formulation of self-attention. Then, it gives the definition of layer attention, and formulates the recurrent layer attention as well as its multi-head version.
3.1 REVISITING ATTENTION
Let X E RTxx... be an input matrix consisting of T tokens with Din dimensions each, and we consider a self-attention with an output matrix O RTxx While in NLP each token corresponds to word in a sentence, the same formalism can be applied to any sequence of T discrete objects, e.g. pixels and feature maps.
The self-attention mechanism first derives the query, key and value matrices Q, K and by projecting X with linear transformations, i.e., Q = XWQ with We RDnxDD k, K = XWK with Wx E IR Din Di, and V XWr with Wv E IR Din Dout, Then, the output is given by:
QKT
O Self-Attention(X == softmax(
DD V AV,
where A = (ar,) is T T matrix. Here adopt NumPy-like notations: for a matrix Y R Y,.. Y.J. and Vi. are its i-th row, j-th column, and (i,j)-th -th element, respectively. Morcover, [
refers to the set of indices to T. Then, self-attention mechanism mapping any query token i [ from Din to Dast dimensions can be formulated in an additive form:
T
O = A V
Gr Va..,
(1)
3

---

Published as a conference paper at ICLR 2023
3.2 LAYER ATTENTION
For deep neural network, let X E RIID be the output feature of its t-th layer, where t E [T) and T's is the number of layers. We consider an attention mechanism with X attending to all previous layers and itself, i.e., the input matrix is (X ,.... X²) E RtxD, and each X is treated as token. Assuming Din = Dout= = D, we first derive the query, key and value for the t-th layer attention below,
Q' S E R!xDe K = =Cocaatfff (Xx fk (x')] and V (X
(3)
where K E RtxD2 and V RIXD. Here, fá denotes a function to extract current layer's information: fk and fV are functions that extract information from the 1st t-th layers' features. Denote the output of the 1-th layer attention by O E IRIxD and then the layer attention with the I-th layer as the query is defined as follows:
O = Q' V' - [K V
which has a similar additive form as in Eq. ( The softmax and the scale factor vD, are omitted here for clarity since the normalization can be performed easily in practice. The proposed layer attention first depicts the dependencies between the t-th and s-th layers with the attention score Q'(K!.)) (. and then use it to reweight the transformed layer feature Ve..
3.3 RECURRENT LAYER ATTENTION
In Eq. (3. the computation cost associated with fkr and ff increases with t. Taking advantage of the sequential structure of layers, we make a natural simplification (see Appendix A. for details):
K' [oncat[sk
and V == Concat( (Xx) (x.)..., (x.,)))(
(5)
The simplification allows the key and value matrices of the I-th layer to inherit from the preceding
ones, i.e., K = Concat| K .f (X')) and V = Concat V'-1 .fv (X*)), which avoids the redundancy induced by repeatedly deriving the keys and values for the same layer with different transformation functions. Based on this simplification, we can rewrite Eq. 44 into
0' (K IV. + (K()
= =Σ Q'K ( +' =' (
QK(,.) VV
(6)
where the first term corresponds to the attention with the t-th layer sending the query to all previous layers. Compared to Eq. (4). K" and V are now constructed in recurrent way. i.e., K" and Vi-l are reused. We therefore call Eq. (6) the recurrent layer attention (RLA), which significantly saves the computation cost. A schematic diagram is provided in Figure (b).
Nonetheless, RLA suffers from quadratic complexity of the network depth since the first term in Eq. (6) still needs be recomputed for each layer. Besides, for an image. the feature dimension D of V is equal to H x W x C. For a very deep network with large f, the expanding size of V will result in the out-of-memory problem. Both of them limit the layer attention to a shallow or narrow network. Luckily, several techniques are available for linearizing the complexity of self-attention for the tokens within a layer. If workable at the layer level, they can be the tool that facilitates the layer attention to adapt for very deep networks under training time and memory constraints. We provide an example of linear RLA with the method proposed by Katharopoulos et al. (2020) See A.c Appendix.
Here suggest another tailor-made method that utilizes an approximation of Q'KK' v " to linearize RLA. Denote the element-wist product by O, and then there exists a X E I xx such that
Q' = X1 3 Q We speculate that query vectors at two consecutive layers have similar pattern, i.c. Q' is roughly proportional to Qt-1 or the elements of X₂ have similar values (see Figure b) for empirical support). Consequently,
Q'(K' V T
(X
[K V Q (K(-1) v - =0000 0
(7)
4

---

Published as a conference paper at ICLR 2023
30002
SOUA orpnunnc
20002
10000
0.3 value cos similarity between OF and of-1 (9)
Figure 2: (a) Detailed operations in MRLA-light block with feature dimensions; (b) Absolute cosine similarity between queries from MRLA bbas blocks of two consecutive layers.
where X E RIID intrinsically depends on Q', Q' and (K*-1) ( and we set X as learnable vector since its computation is complicated (see Appendix A.2 for the detailed explanation on the approximation above). The learnable vector can adaptively bridge the gap between Q' and Q'-1. Note that the approximation in Eq. becomes equivalency when the elements of X₂ are the same, i.e., X c1 1R and then X cl RIxD, Moreover, this approximation can be alleviated by the multi-head design in the next subsection.
With Eq.
7 an efficient and hight-weighter RLA version with complexity O(T) is suggested below:
='= o (K) VV. B ( V
(8)
where Ag = 1, and B X .... + X for 1 2 1. From the second equality, the extended RLA version is indeed weighted average of past layers' features (V:'.)) This is consistent with the broad definition of attention that many other attention mechanisms in computer vision use (Hu et al.) 2018; Woo et al. 2018 Wang et al., 2020b). It is also interesting to observe that the first equality admits the form of generalized residual connection, leading to easier implementation in practice.
3.4 MULTI-HEAD RECURRENT LAYER ATTENTION
Motivated by the multi-head self-attention (MHSA) in Transformer, this section comes up with multi-head recurrent layer attention (MRLA) to allow information from diverse representation subspaces. We split the terms in Eq. (6 and Eq. (8) into H heads. Then, for head E [H], RLA and its weightee version can be reformulated as
O (K%) V/ and O = N & O + Qi (Kij..)) Vv
respectively, where 0 E 1R fo is the h-th head's output of MRLA. The final outputs are obtained by concatenation, i.c. Of C
O. For convenience, we dub the MRLA and its weightee version as MRLA-base and MRLA-light, which are collectively referred to as MRLA.
In contrast Eq. (6 and Eq. (8 where 3D image feature Vt, is weighted by a scalar Q' Q((K( T the multi-head versions allow each layer's features to be adjusted by an enriched vector of length H, strengthening the representation power of the output features. In addition, for MRLA-light,
as now the approximation in Eq. ? is conducted within each head, i.e., Qi (K² V 2
Nr, 1 ( "v, it will become an equality as long as X CA 1 E RI* 2t which is a much more relaxed requirement. In particular, when there are D₂ heads, i.e., H = Dx, the approximation in Eq. 7 always holds.
If layer attention is applied small networks the minimum computation, time and memory cost are not chased, we recommend both MRLA-base and MRI-A-light as they can equivalently retrieve useful information from previous layers. In case has to consider the training time and memory footprints, MRLA-light is preferred as an effecient version that we adapt for deeper networks.
APPLICATIONS OF MRLA IN VISION NETWORKS
Recent networks are formed by deep stacks of similar blocks (layers), and therefore MRLA- base/light block can be inserted right after a building layer in network.
CNNs Figure a illustrates the detailed block design of MRLA-light in CNNs (see that of MRLA-
base in Figure of Appendix). Given the output of the t-th CNN block X E RR I where D H W C. perform global average pooling (GAP) summarize the 1-th layer's
5

---

Published as a conference paper at ICLR 2023
information. Then two 1D convolutions (Conv are used extract the query Q' and the key K whose kernel sizes are adaptively determined by the strategy in Wang et al. (2020b). 3x3 depth-wise convolution (DWConv) is applied to directly get Vi,-- Here, the two Conv (together with GAP()) and DWConv correspond to the fqy fk and fV, respectively. We then divide the query, key and value into H heads along the channels. The output of previous MRLA block, O is partitioned similarly. We set X² of size C instead of H x W x C' in order to introduce fewer parameters, and it performs an glement-wiss product with O after being expanded. A sigmoid activation is added on the projection of query and key to scale the attention weights into [0, 1].
Vision Transformers Most of the block design is the same as in Figure 2[a) except for the following changes. The output of the t-th block of vision transformer is X' E ER N is the number of patches and C' is the embedding dimension. We first split X into patch tokens X E R-NxC' and class token X! E R1xC Then the patch tokens that preserve spatial information are reshaped into X E RVNxVNNC as the input of our MRLA. Only previous patch tokens of the last MRLA output 0, are brought into the MRLA block. Lastly, the patch tokens are reshaped to the initial dimension and concatenated with the class token the next layer's input.
where D (N+1)) x C,
EXPERIMENTS
This section first evaluates our MRLAs by conducting experiments in image classification, object detection and instance segmentation. Then MRLA-light block is taken as an example to ablate its important design elements. All models are implemented by PyTorch toolkit on V100 GPUs. More implementation details, results, comparisons and visualizations are provided in Appendix B.
5.1 IMAGENET CLASSIFICATION
We use the middle-sized ImageNet- IK dataset (Deng et al. 2009) directly. Our MRLAs are applied to the widely used ResNet (He et al. 2016) and the current SOTA EfficientNet (Tan & Le 2019). which are two general ConvNet families. For vision transformers, DeiT (Touvron al. [021] CeiT (Yaan et al. 202T) and PVTv2 (Wang et al. 2022a) are considered. We compare our MRLAs with baselines and several SOTA attention methods using ResNet a baseline model.
Settings A hyperparameter da is introduced to control the number of channels per MRLA head. We set d, = 32, and 16 for ResNet, EfficientNet and vision transformers, respectively. To train these vision networks with our MRLAs, follow the same data augmentation and training strategies
as in their original papers (He et al. 2016 Tan & Le 2019 Touvron et al. 2021 Wang et al. 2022a).
The performances of the different-sized SOTA networks with our MRLAs are reported in Table T For a fair comparison, the results ResNets from torchvision are replicated. We first observe that MRLA-bass and MRIA--lght have comparable performances when added to relatively small networks, verifying that the approximation in MRLA-light does not sacrifice too much accuracy, The oft-of-memory problem occurs when MRLA-base is applied to ResNet- 101 with the same batch size. Therefore, MRLA--lght resources are taken into account, and it can perform slightly better probably because of the additional flexible learning vector. We next compare our MRLAs with other attention methods using ResNets as baselines. Results show that our MRLAs are superior to SENet, CBAM, Ae-Net, one NL, and ECA-Net. Especially among layer-interaction-rllle and RLA,--et, all of which beat the DenseNet of similar model size. MRLAs are also as competitive as AA-Net and GCNet with lower model complexity or fewer parameters. For EfficientNets, MRLAs introduce about 0.0IM and 0.02M more parameters, leading to 1.3% and 1.1% increases in Top-1 accuracy for EfficientNet- BOrrrennt respectively. It is worth noting that the architecture of EfficientNets obtained via a thorough neural architecture search which is hard to be further improved. Consistent improvements are also observed in the transformer-based models. Specifically, our MRLAs can achieve 1.2% and 1.0% gains in terms of Top-1 accuracy on DeiT-T and CeiT-T, while both introduce +0.03M parameters, and MRLA-light only increases +0.04B FLOPs. With CeiT-T, also validate that the FLOPs induced by our MRLA are nearly linear to the input resolution (See Appendix BB.12). We additionally supplement fair comparison with BANet (Zhao al. 2022) in Appendix B.
Results
light is recommended for deeper networks if the efficiency and computational
networks, our MRLAs outperform the DIA Net

---

Published as conference paper at ICLR 2023
Table 1: Comparisons of single-crop accuracy on the ImageNet- validation set. f means the results from torchvision toolkit. The bold fonts denote the best performances.
Model Type
Model
Params
FLOPs
Input
Top-1
Top-5
ResNct-50' (Hc ct al.
2016
25.6 M
4.1
224
76.1
92.9
SE (Hu el al. 2018
28.1 M
4.1 B
224
76.7
93.4
CBAM (Woo
2018
28.1 M
4.2 B
224
77.3
93.7
+ _1a
CCenn el al.
2018)
34.6 M
7.0
224
77.0
93.5
+ AA [Bello ct al
2019
27.1 M
4.5 B
224
77.7
93.8
NL (Wang ct
2018
29.0 M
4.4 B
224
77.2
93.5
GC Cao
2019
26.9 M
4.1
224
77.3
93.5
all G (Cao
2019
29.4
4.2
224
93.7
+ ECA (Wang et al.
2020b
25.6 M
4.1 B
224
77.5
93.7
DIA Huang et al.
20201
28.4 M
224
77.2
RLA,, Zhao
2021
25.9 M
B.5
224
77.2
93.4
CNNs
MRLA base (Ours)
25.7
4.6
224
93.9
MRLA-light (Ours)
25.7 M
4.2
224
77.7
93.8
ResNet-1101 (He
2016
44.5 M
7.8
224
77.4
93.5
(Hu 2018
49.3
7
224
77.6
93.9
+ CBAM (Woo et al.
2018
49.3 M
7.9
224
78.5
94.3
+ AA (Bello ct al
20T9)
47.6 M
8.6 B
224
78.7
94.4
ECA (Wang
2020b
44.5 M
7.8 B
224
78.7
94.3
RL.A, (Zhao
2021
4
8.4
224
78.5
94.2
MRLA-light (Ours)
44.9
7.9
94.4
ResNet-152 (He et al.
2016)
60.2 M
11.6
224
78.3
94.0
SE (Hu et al.) 2018
66.8 M
11.6
224
78.4
94.3
ECA (Wang al.
2020b
60.2
11.6
224
+ RLA, (Zhao et al.
2021
60.8 M
12.3B
224
78.8
94.4
+ M.RLA--ight (Ours)
60.7 M
11.7
224
79.1
94.6
EfficientNet-BB BO (Tan & Le
2019)
5.3 M
0.4
224
77.1
93.3
MRLA-bbse (Ours)
5.3 M
0.6
224
78.3
94.1
MRLA-light (Ours)
0.5
94.1
EfficientNet-B1 Tan & Le
2019)
7.8 M
0.7 B
240
79.1
94.4
MRLAAbbse (Ours)
7.8 M
0.9 B
240
80.2
95.3
MRLA--ight (Ours)
7.8 M
0.8 B
240
80.2
95.2
DeiT-T Toovvron et al.
2021
5.7 M
1.2
224
72.2
91.1
+
MRL-A-base (Ours)
5.7 M
1.4 B
224
73.5
92.0
MRIAA-lght (Ours)
5.7 M
1.2
224
73.4
91.9
DeïT-S Touvron et al.
2021
22.1 M
4.5
224
79.9
95.0
MRLL--lght (Ours)
22.1
4.6
224
95.9
DeiT-B Touvron et al.
2021
86.4 M
16.8
224
81.8
95.6
AMRLA-lihtt (O Ours)
86.5 M
16,9
224
82.9
96.3
CeiT-T Yuan et al.
2021
6.4 M
1,4
224
76.4
93.4
Vision Transformers
+MRL
6.4 M
1.4
224
77.4
94.1
CeiT-T Yuan et al.
M
5.1
384
94.7
-igLA- (0 (Ours)
6.4 M
5.1
384
79.6
95.1
CeiT-S Yuan et al. 2021
24.2 M
4.8
224
82.0
95.9
-MRLA-light (Ours)
24.3 M
4.9
224
83.2
96.6
PVTv2-BO (Wang
2022a
3.4 M
0.6
224
+ MRLLA-base (Ours)
3,4 M
0.9 B
224
71.4
90.7
+ -RRLA-light (Ours)
3.4 M
0.7 B
224
71.5
90.7
PVTv2-B1 (Wang al.
2022a
13.1 M
2.3 B
224
78.7
MRLA-light (Ours)
13.2
2.4
224
79.4
94.9
Assumption Validation To validate the assumption we make in Eq. 7 that Qi is roughly proportional to Ø within each head, compute the absolute value of the cosine similarity between them and visualize the histogram in Figure Note that if the absolute cosine similarity approaches 1, the desire that the elements of X h. have similar values reached. The validation is conducted by randomly sampling 5 images from each class of the ImageNet validation set and then classifying these images with the trained each head of all MRLA-base blocks extracted except for those belonging the first layer within each stage, as the first layer only attends to itself.
model. The query vectors from
7

---

Published as conference paper at ICLR 2023
Table 2: Object detection results of different methods on COCO val2017. FLOPs are calculated on 1280 x 800 input. The bold fonts denote the best performances.
Methods
Detectors
Params
Ap
AP
A/
APS
APM
APL
ResNet-50
41.53 M
36.4
58.2
39.2
21.8
40.0
46.2
+ (Hu et al. 2018
44.02 M
60.1
40.9
22.9
41.9
48.2
+ ECA Wang et al. 2020h)
41.53 M
38.0
60.6
40.9
23.4
42.1
48.0
RLA., (Zhao et al.
2021
41.79 M
38.8
59.6
42,0
22.5
42.9
49.5
BA (Zhao et al. 2022)
44.66 M
39.5
61.3
43.0
24.5
43.2
50.6
MRLA-base (Ours)
41.70 M
61.3
43.8
43.9
52.4
MRLA-light (Ours)
Faster R-CNN
41.70 M
61.5
44.0
24.2
44.1
52.7
ResNet-101
60.52 M
38.7
60.6
41.9
22.7
43.2
50.4
SE (Hu et al. 2018
65.24 M
39.6
62.0
43.1
23.7
44.0
51,4
ECA (Wan
2020b
60.52 M
40.3
62.9
44.0
24.5
44.7
51.3
+ RLA,, (Zhao 2022)
2021
60.92 M
44.9
45.7
53.8
+ BA (Zhao et al.
66,44 M
41.7
63.4
45.1
24.9
45.8
54.0
MRLA-light (Ours)
60.90 M
42.0
63.1
45.7
25.0
45.8
55,4
ResNet-50
37.74 M
35.6
55.5
38.2
20.0
39.6
46.8
+ (Hu et al. 2018
40.23 M
57.2
39.9
40.7
49.3
+ ECA Wang et al.
2020b
37,74 M
37.3
57.7
39.6
21.9
41.3
48.9
RLA,, Zhao et al
2021
38,00 M
37.9
57.0
40.8
22.0
41.7
49.2
MRLA- (Ours)
37.92 M
39.3
59.3
42.1
24.0
43.3
50.8
MRLA-light (Ours)
37.92 M
39.6
59.7
42.4
24.1
43.6
51.2
RetinaNet
ResNet-101
56.74 M
37.7
57.5
40.4
21.1
42.2
49.5
+ SE (Hu et al. 2018
61,45 M
38.7
59.1
41.6
22.1
43.1
50.9
ECA
Wang et al.
2020b
56.74 M
39.1
59.9
41.8
22.8
43.4
50.6
RLA,
Zhao
202T
57.13 M
40.3
59.8
43.5
24.2
43.8
52.7
MRLA Alihh (Ours)
57.12 M
61.4
44.2
45.6
53.8
Table 3: Object detection and instance segmentation results of different methods using Mask R-CNN framework COCO val2017. APM and APm denote AP of bounding box and mask.
Methods
Params
GFLOPs
A
AP
AP
Apm
AP
AP
ResNet-50
44.18 M
275.58
37.2
58.9
40.3
34.1
55.5
36.2
SE Hu et al. 2018
46.67 M
275,69
38.7
60.9
42.1
35.4
57.4
37.8
+ BCA Wang
2020b
44.18
275.69
39.0
61.3
35.6
58.1
+ NL Wane et al
2018
46.50 M
288.70
38.0
59.8
41.0
34.7
56.7
36.6
+ GC (r16) Cao et al
201
46,90 M
279.60
39.4
61.6
42.4
35.7
58.4
37.6
GC (r4) (Cao ct
2019)
54.40 M
279.60
39.9
62.2
42.9
36.2
58.7
38.3
RLA,, (Zhao
202T
44.43
283.06
39.5
60.1
43.4
35.6
56.9
38.0
+ BA (Zhao el al. 2022)
47.30
261.98
40.5
61.7
58.7
38.6
+ MRLA-base (Ours)
44.34 M
289.49
40.9
62.1
44.8
36.9
58.8
39.3
MRLA-light (Ours)
44.34 M
276.93
41.2
62.3
45.1
37.1
59.1
39.6
ResNet-101
63.17 M
351.65
39.4
60.9
43.3
35.9
S7.7
38.4
+ (Hu et al. 2018
67.89
351.84
40.7
62.5
$9.3
39.2
+ ECA Wang et al.
2020b
63.17 M
351.83
41.3
63.1
44.8
37.4
59.9
39.8
+ NL Wang ei al. 2018
65.49 M
364.77
40.8
63.1
44.5
37.1
59.9
39,2
GC (r16) Cao
2019)
68.10 M
354.30
41.1
63.6
45.0
37.4
60.1
39.6
+ GC (14) (Cao et al.
2019)
82.20
354.30
41.7
63.7
37.6
60.5
39.8
RLAA, Zhao el al.
202T
63.56
362.55
41.8
62.3
37.3
59.2
MRLA-light (Ours)
63.54 M
353.84
42.8
63.6
46.5
38.4
60.6
41.0
5.2 OBJЕCT DETECTION AND INSTANCE SEGMENTATION
This subsection validates the transferability and the generalization ability of our model in object detection and instance segmentation tasks using three typical object detection frameworks: Faster R-CNN (Ren et al. 2015), RetinaNet (Lin et al. 2017) and Mask R-CNN (He et al. 2017).
Settings All experiments are conducted on MS COCO 2017 (Lin et al. 2014), which contains 118K training. 5K validation and 20K test-dev images. All detectors are implemented by the open-source MMDetection toolkit (Chen et al. 2019). where the commonly used settings and Ix training schedule
(Hu et al. 2018 Wang et al. 2020b Zhao al. 202T are adopted.
Results on Object Detection Table rreports the results on COCO val2017 1 set by standard COCO metrics of Average Precision (AP). Surprisingly, MRLAs boost the AP of the ResNet-50 and
8

---

Published as a conference paper at ICLR 2023
Table 4: Ablation study on different variants using ResNet-50 as the baseline model.
Model
Params
FLOPs
Top-1
Model
Params
HLOPs
Top-1
(a) MLA (Eq.(4)
25.7 M
4.8
77.6
MiLA-light
25.7 M
4.2
77.7
MRLA-base
25.7 M
4.6
77.7
(c) X
25.7 M
4.2 B
77.1
MRLA-light
25.7 M
B.2
77.7
d1=16
25.7 M
4.2
77.5
(b) w/ X5
25.7 M
4.2
77.0
(f) (f)- d,=64
25.7 M
4.2 B
77.3
(c) w/o DWConv2d
25.6 M
4.1 B
77.4
d==1 (CRLA)
25.7 M
4.2 B
77.2
(d) w/ FC
28.2 M
4.2 B
77.5
(g) DWCon22d
25.7 M
4.2
76.6
ResNet-101 101 by 4.0% and 3.5%, respectively. The improvements on other metrics are also significant,
e.g., 3-4% APso and 4-6% on AP,. In particular, the stricter criterion AP75 can be boosted by 4-5%, suggesting a stronger localization performance. More excitingly, ResNet-50 with our MRLAs outperforms ResNet-101 by 2% these detectors. Even when employing stronger backbones and detectors, the gains of our MRLAs are still substantial, demonstrating that our layer-level context modeling are complementary
the capacity of current models. Remarkably, they surpass all other models with large margins. Though RLAg and our MRLAs both strengthen layer interactions and thus bring the most performance gains, our layer attention outperforms layer aggregation in RLAg-
Comparisons Using Mask R-CNN Table[]shows MRLAs stand out with remarkable improve- ments on all the metrics. Especially, MRLA-light strikes a good balance between computational cost and notable gains. For example, it is superior to RLAg module and GC block, while using much lower model complexity. Even though BA-Net ('+ BA') utilizes better pre-trained weights obtained from more advanced ImageNet training settings, our approach still outperforms in these tasks.
In summary, Tables Soblls@aand 3 demonstrate that MRLAs can be well generalized to various tasks, among which they bring extraordinary benefits to dense prediction tasks. We make a reasonable conjecture that low-level features with positional information from local receptive fields are better preserved through layer interactions, leading to these notable improvements (see Figures 6 and 7 in Appendix).
5.3 ABLATION STUDY
Different Variants of MRLA Due to the limited resources, we experiment with ResNet-50 model on ImageNet. We first compare the multi-head layer attention (MLA) Eq. (4) and MRLA-base in (a). Then we ablate the main components of MRLA-light to further identify their effects: (b) entirely without previous information; (c) without 3x3 DWConv; (d) replacing the 1D convolutions with fully-connected (FC) layers; (e) with the identity connection, i.e., A,=1; (g) adding a 3x3 DWConv to each layer of the baseline model. We also compare different hyperparameters: (f) different de. including the special case of channel-wise recurrent layer attention (CRLA), i.e., dę 1.
Comparing (a) and MRLA- validates our approximation in Eq. 77 Since the 3x3 DWConv mainly controls the additional computation cost, we compare with inserting a 3x3 DWConv layer after each layer of the original ResNet-50. Comparing (c) and (g) with ours shows the improvement in accuracy is not fully caused by increasing model complexity. Then the necessities of retrieving previous layers' information and introducing X are investigated. (b) indicates strengthening layers' interdependencies can boost the performance notably; while (e) shows MRLA plays a much more influential role than simple residual connection. Using the FC layer in (d) may be unnecessary because it is comparable to Cony ID but introduces more parameters. (f) shows that our MRLA with different hyperparameters is still superior to the original network.
Results from Table 4
CONCLUSION AND FUTURE WORK
This paper focuses on strengthening layer interactions by retrospectively retrieving information via attention in deep neural networks. To this end, we propose a multi-head recurrent layer attention mechanism and its light- weighted version. Its potential has been well demonstrated by the applications on mainstream CNN and transformer models with benchmark datasets. Remarkably, MRLA exhibits a good generalization ability on object detection and instance segmentation tasks. Our first future work is to consider comprehensive hyperparameter tuning and more designs that are still worth trying on CNNs and vision transformers, such as attempting other settings for the number of layer attention heads of each stage and using other transformation functions. Moreover, in terms of MRLA design at a higher level, it is also an interesting direction to adapt some other linearized attention mechanisms stack of layers.

---

Published as a conference paper at ICLR 2023
Reproducibility Statement To supplement more explanations and experiments and ensure the reproducibility, we include the schematic diagram of multi-head layer attention, detailed inner structure design and pseudo codes of MRLA-base and MRLA- in CNNs and vision transformers in Appendix A Besides, more implementation details about baseline models with our MRLAs are documented in Appendix B
REFERENCES
Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. Training deeper neural machine translation models with transparent attention. In EMNLP, 2018.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3286-3295, 2019.
Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. GCNet: Non-local networks meet squeere-excitation networks and beyond. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision Workshops, pp. 0 0-0, 2019.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 00950.0565,..... 2019.
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. -e-Nst Double attention networks. In Advances in Neural Information Processing Systems, 2018.
Cheng Chi, Fangyun Wei, and Han Hu. Relationnet++:: Bridging visual representations for object detection via transformer decoder. In Advances in Neural Information Processing Systems, 2020.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with Performers. ICLRR 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2021.
Chenyu Guo, Jiyang Xie, Kongming Liang, Xian Sun, and Zhanyu Ma. Cross-layer navigation convolutional neural network for fine-grained visual classification. In ACM Multimedia Asia, MMAsia '21, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
10

---

Published as a conference paper at ICLR 2023
Ruining He, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. Realformer: Transformer likes residual attention. In Findings of the Association for Computational Linguistics: ACL-IJCNLP, pp. 929 993,, 2021.
Geoffrey Hinton. How to represent part-whole hierarchies in neural network. arXiv preprint 1.NV:2200..12677, 2021.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8129-8138, 2020.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chinh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314 1324, 2019.
Jie Hu, Li Shen, and Gang Sun. Squeeze- Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132-7141, 2018.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. European Conference on Computer Vision, pp. 646-661. Springer, 2016.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Zhongzhan Huang. Senwei Liang, Mingfu Liang, and Haizhao Yang. DIANet: Dense-and-implice attention network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4206-42144 2020.
Shantanu Jaiswal, Basura Fernando, and Cheston Tan. TDAM: Top-down attention module for contextually guided feature selection in cnns. In European Conference on Computer Vision, pp. 259-276, 2022.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp. 5156-5165. PMLR, 2020.
Yangyang Li, Qin Huang, Xuan Pei, Yanqiao Chen, Licheng Jiao, and Ronghua Shang. Cross- layer attention network for small object detection in remote sensing imagery. IEEE Jourmal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:2148-2161, 2021. doi: 10.1109//SARSS.0000 3046482..
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988. 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. International Conference Computer Vision (ICCV), 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 91-99, Cambridge, MA, USA. 2015. MIT Press.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958, 2014.
11

---

Published as a conference paper at ICLR 2023
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826, 2016.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-6 61 14. PMLR, 2019.
Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip M Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10193-10202. PMLR, 18-24 Jul 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 24-25, 2020a.
Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. ECA-Net: Efficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. PVTv2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):1-10, 2022a.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018
Yancheng Wang. Ning Xu, Chong Chen, and Yingzhen Yang. Adaptive cross-layer attention for image restoration. arXiv preprint arXiv:2203.03619 2022b.
Yujing Wang, Yaming Yang, Jiangang Bai, Mingliang Zhang, Jing Bai, Jing Yu, Ce Zhang, Gao Huang, and Yunhai Tong. Evolving attention with residual convolutions. In International Conference on Machine Learning. PMLR, 2021.
Ross Wightman. Pytorch image models. 2019.
wyttorch-image-eee
Sanghyun Woo, Jongchan Park, Joon-Young Young Lee, and In So Kweon. CBAM: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492-1500, 2017
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. International Conference on Computer Vision (ICCV), 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 6023-6032, 2019.
12

---

Published as a conference paper at ICLR 2023
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paze Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 2017.
Jingyu Zhao, Yanwen Fang, and Guodong Li. Recurrence along depth: Deep convolutional neural networks with recurrent layer aggregation. In Advances in Neural Information Processing Systems, 2021.
Yue Zhao, Junzhou Chen, Zirui Zhang, and Ronghui Zhang. BA-Net: Bridge attention for deep convolutional neural networks. In European Conference on Computer Vision pp. 297-312, 2022.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data aug- mentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 13001-13008, 2020.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
This appendix contains supplementary explanations and experiments support the proposed layer attention and multi-head recurrent layer attention (MRLA). Appendix supplements the arguments made in Section 3 including clarification on layer attention in Eq. ( recurrent layer attention in Eq. (6) and (8) and another linearization of recurrent layer attention with the design in Katharopoulos al. 2020p. The pseudo codes of the modules are also attached. Appendix provides more experiment details, results, explanations for ablation study and visualizations of MRLA base and MRLA-light in CNNs and vision transformers. Besides, we also attempt support our motivation through experiment results.
A MULTI-HEAD RECURRENT LAYER ATTENTION
A.1 discusses the differences between layer attention and other layer interaction related work. A.2 elaborates more on the simplification and approximation made in MRLA-base and MRLA-light Besides, our attempt to linearize RLA with the recurrent method proposed in (Katharopoulos et al. 2020) included A.3 which briefly mentioned in Section 3 eA.lexplains more about broad definition of attention. A.5 illustrates the detailed block design of MRLA-base and provides the pseudo codes of the both modules.
A.1 LAYER INTERACTION RELATED WORK
Apart from the OmniNet (Tay al. 2021 that elaborate in Section we would like further compare our layer attention with other layer interaction related work, including DenseNet (Huang et al. (2017), DIA-Net (Huang al. 2020) and RLA, Net (Zhao al. 202T). The empirical studies
comparing them are included in Appendix B.1.3
Here residual layer refers to a CNN block or a Transformer block (including the multi-head attention layers and the feed-forward layer). For each layer, the feature maps of all preceding layers are used as its layer attention's 's inputs; and its own feature maps are also reused in subsequent layer attention, which intrinsically has quadratic complexity.
Layer Attention Figure 3
illustrates the layer attention proposed in Section 3.2
(*)
Mutt-head Altention
ULxx) ((X)))
UL00" ")
Layer
Rezi
Laver
Resiiall
Layer'
Pesid
Figure 3: Schematic diagram of layer attention Eq.
Comparison with DenseNet, DIANet, and Net Though DenseNet also extracts all previous layers' features and concatenates them before the Conv 1x1, not attention-based method. Moreover, it aggregates all previous layers' features in channel-wise way regardless of which layer feature comes from. can be viewed as channel attention across layers. Instead, our layer attention models the features from the same layer in the same way via attention. RLA, Net doesn't 't use attention, either. Finally, DIANet emphasizes channel-wise recalibration of the current layer output, which is intrinsically channel attention instead of layer attention.
A.2 DETAILS ABOUT SIMPLIFICATION IN EQ. 55) AND APPROXIMATION IN EQ.
Simplification in Eq. 5 This simplification is weak and reasonable. the vanilla self- attentitnnin,
suppose we have X E Txx where T stands for the number of tokens and D.n is the input
14

---

Published as a conference paper at ICLR 2023
feature dimension. The vanilla self-attention derives the key and value matrix K Tx and V Doot by applying the transformation functions FK and Fy respectively:
K = (X) = FK (Concat X 1.:5 2,.. XT.I)s
and Fv(X)= Fy (Concat L 2,:...,, xT.I). Importantly, both the transformation func-
tions linear projections, i.e., FR (X) XWK and Fv XWy. They only operate on the feature dimension Din and thus
= Concat [F (X1.), (x:)) and V Concat Fv (X1.), ( (Xr,))
(10)
Recall that in layer attention, we assume Din Dout D and denote the output feature of the t-th layer as X E RIID In the 1-th layer attention, we derive the key and value K e Rex and V E R D by applying the transformation functions fk and fV in Eq
Kt Concat(sk (X"), (x*) and
V Concat| I (x ,..,, (i())
In other words, we directly borrow the formulation in Eq 10 from the vanilla attention. Compared with the (5
K' Concat f'x
and =²= "C (x",.,, (x)),,
Eq. 33 has larger computation complexity. It is then natural to use the same transformation functions avoid redundancy that caused by repeatedly deriving the keys and values for the same layer with different transformation functions.
Approximation in Eq. Denote the resulting matrix of (K ( B R D&XD with b) being its j-th row vector. If the con-lincar softmax function on the projection of query and key is omitted, then the first approximation in Eq. 7 as follows:
Q'(K-11) Vi-l = v
(XA o Q Q(-*)((K v--1
(11)
= (Ag 3 Q(
(12)
Dr
(13)
j=1
> 9j
b/-1
(14)
1=1
= =qQ
(K*-1) y
(15)
where X and q3 in 13) We approximate 13) with by assuming all the elements of X are the same. As mentioned in Section 4.1, 1 the proposed meat--head design relaxes this condition by only requiring all the elements of Nr q.h E 1x f are the same. We then generalize (55 to A (K'-)
the j-th element of X E RIIDx and Q RIxDs, respectively.
and set X as learnable. This injects more flexibilities along the dimension D and alleviates the effects of the simplification in (14).
A.3 ANOTHER LINEARIZATION TECHNIQUE ON RLA
In addition to using learnable parameter to bridge the previous layer attention output with the current one in Eq. (8). we have tried another technique linearize the computation of layer attention. We can first rewrite the Eq. (6) with the softmax and the scale factor VD, as follows:
O' Σt K(Q', K. V
Σ' M(Q K!.)
where the kernel function k(x,v) exp( rTy Assuming that k(x, y) can be approximated by (DD)
another kernel with feature representation Φ(-), that is, k( ) E(d(x)) d(y)] [(v))) (Choromanski et al.
15

---

Published as a conference paper at ICLR 2023
2021
Then, the t-th layer attention output can be represented as:
O Ex 1 (QQ))(K))) VVV.
E ((Q))(K))
0 (Q)) E @( Vvi. o " o
The last equality holds because of the associative property of matrix multiplication. Adopting the linearization technique proposed by Katharopoulos et al. (2020), we introduce two variables:
U
Vi
and Z
o K.
"iQ' "(Q')2=-
and simplify the computation as O from U and z by
Is worth noting that U and Z' can be computed
-'-U' + (KK)) Vi: Z' - o (
This version of recurrent layer attention also results in linear computation complexity with respect to the network depth. However, compared with Eq. (8. this attempt suffers from higher memory costs and lower inference speed. Its multi-head implementation also performs poorer than our proposed BRRLA-bas/lighh in experiments. Therefore, we prefer weighting the previous layer attention output with learnable parameter, which more efficient way strengthen layer interactions in practice.
A.4
MRLA-LIGHT IS CONSISTENT WITH THE BROAD DEFINITION OF ATTENTION
In Eq. 8 we have proven that
[-1 O' O'= 1=0
3 3
Q (K-))) [v
which weighted average of past layers' information V.-L.- This is consistent with the broad definition of attention that many other attention mechanisms computer vision use.
More concretely, the existing channel attention spatial attention mechanisms (SE(Hu et al. 2018 CBAM((Wo et al. 2018) and BCA(Wang et al. (0205)) obtain the weights (similarities) of channels FC layers and a 1D convolution, while
pixels via learnable parameters (SE and ECA CBAM adopts 2D convolution), and then scale the channels or pixels by the weights. This broader attention definition relaxes the requirement on the weights, allowing them not to fully depend on X but to be freely lcarnable.
Motivated by the above, the extended version of layer attention (MRLA-light) also scales different layers and takes the weighted average of them. It's worth mentioning that MRLA-light is different from SE in terms of SE recalibrating channels within a layer while MRLA-light makes adjustments across layers.
16

---

Published as a conference paper at ICLR 2023
A.5 MRLA BLOCKS AND PSEUDO CODES
Figure 4 illustrates the detailed operations in MRLA-base block with feature dimensions. Most of the inner structure designs are similar to those of MRLA-light block, except for concatenating previous keys and values instead of adding the output of previous MRLA block.
H
ConviD
Grobal Averago Pooing
MK
Q'
Dot-Product Attentoon
tok ConviD
Kf
Concat
O/ N=WxO
K
X
1x1x0
INERC
SK3
Concel
V
V
K
Figure 4: Detailed operations in MRLA-base block with feature dimensions.
Pseudo Code Pseudo codes of MRLA-base's and MRLA-light's implementations in CNNs and vision transformers are given below.
Algorithm MRLA-base in CNNs
Input: Output the t-th CNN block X BIxHxWxC t 1-th MRLA-base block's key and value K [Ixx(=1)xCC and Vt-1 E RIx(-I)xHxW'x Number of heads H
2: Output: O E R HIx xC, K' XxXIXC Vt BlxxxHxWxC
3: // Summarize X spatial information
4: Y + GAP(X*) glxixC
5: // Derive the current layer's query, key and value via convolutions
6: Q' + (Cov11D( E Rlxixx"
7: K/ + ConvID(Y') E RRxixx
8: VA < (XC E BIxIIHxWXC
9: If then
10: // First MRLA-base block
K' K
y V.
13: else
14: Concatenate with the previous key and value
15: K' +- Concat kt-1 K[ E E.JiE RIxxxC
V Concat + VIl BlxIXHxXX
17: End if'
O Multi-head Attention Q',K' K', Vt) where the number of heads H
B EXPERIMENTS
Due to the limited space in the main paper, we provide more experimental settings and discussions in this section which are organized follows. We first provide the implementation details of ImageNet classification and more comparisons with other SOTA networks in Appendix B. We also compare the model complexity and memory cost of MLA and MRLAs in Appendix [.2 Next, some implementation details and results object detection and instance segmentation on COCO included in Appendix B.3 Then the ablation study of vision transformer (DeiT) is given in
17

---

Published as a conference paper at ICLR 2023
Algorithm 2 MRLA-base in Vision Transformers
1: Input: Output of the f-th transformer block X' E RIINN+IJCC 1-th MRLA-base block's ss
key and value Kt-1 E Rix((-I)xC and Vi-1 E RII--I//NXVAAC Number of heads H
2: Output: Of E Rix(N-1)xC K' RixxxC V BlxXx/NXvNXx
3: // Split into the class token and patch tokens
4: X IR IxXx X IR xxxxx t- Split(X*)
5: //Reshape
6: X +- Reshapee (x+) IIXVVXVNXCC
7: // Summarize X 's spatial information
8: Y <- -GAP(X)) R lxxx
9: // Derive the current layer's query, key and value via convolutions
10: Q' + ConviD( E Rixixc'
11: KG +- ConviD((;) p) RRIII
12: V. < +DWCom22D(X) E RIIII/NNVVXC
Ift then
14: // First MRLA-base block
K
K
16:
V
+++ V.
17: else
18: // Concatenate with the previous key and value
19: Kt +- Concat k KK Rlxtxc
V (o V BRxxxVNXVNXX
20:
21: End
22: O Multi-head Attention (Q', 'Q', K*, Vt) where the number of heads is H
23: // Reshape to the original dimension
0, Reshape( (O,) R |XXXxC
0 Concat[X 0
p
Algorithm MRLA-light in CNNs
1: Input: Output of the t-th CNN block X IIIHXWXC, -- 1-th MRLA-light block's output
r E IR X Number of heads H, Learnable parameter X E R R
2: Output: O R I
3: // Summarize X spatial information
4: Y' GAP(X') RixixC
5: // Derive the current layer's query, key and value via convolutions
6: Q' Conv (D(') glxxxc
7: K; + ConvID(Y")) E lxxxx
8: V <- 2WC) BIXIXHXWXC
9: 0 + Multi-head Attention Q', K. to V..) where the number of heads H
10: 0 Expand(X{) 00-1 O'
18

---

Published as a conference paper at ICLR 2023
Algorithm 4 MRLA-light in Vision Transformers
1: Input: Output of the t-th transformer block X E R RIx(N+1)xC, t 1-th MRLA-light block's
output O E R (X)xXC) Number of heads H, Learnable parameter X R 1xC
2: Output: O' E |
3: // Split into the class token and patch tokens
4: X E ERR xx X R IxIxx + Split(X"*)
5: O E 1 EIINNC + Split(Or-1)
6: //Reshape
7: X + Reshape (X.)) E RRIVNNVVNXC
8: // Summarize X spatial information
9: Y -GA((X;)) R. IxIxC
10: // Derive the current layer's query, key and value via convolutions
Q' + Conv MonviD((Y) Y E RIxIxC
12: K + Conv ) E RIxIxe
13: VE. < GELU(DWCOn>DD(X) E RIXIIVNNVVNX
14: O Multi-head Attention(() Q' K Vt.) where the number of heads is H
15: // Reshape to the original dimension
16: O + Reshape O p E ER' |XXXC
17: O Expand X @ O,- +
0 Concat X O
p)
Appendix B.4] Finally, visualizations of the feature maps/attention maps in ResNet50/DeiT and our MRLA counterparts are shown Appendix [B.5 All experiments are implemented on four Tesla V100 GPUs (32GB).
B. IMAGENET CLASSIFICATION
B.1.1 IMPLEMENTATION DETAILS
ResNet For training ResNets with our MRLA, we follow exactly the same data augmentation and prrerpparameter settings in original ResNet. Specifically, the input images are randomly cropped 224 224 with random horizontal flipping. The networks are trained from scratch using SGD with momentum of 0.9, weight decay of le-4, and mini-batch size of 256. The models are trained within 100 epochs by setting the initial learning rate 0.1, which is decreased by a factor of 10 per 30 epochs. Since the data augmentation and training settings used in ResNet are outdated, which are not as powerful as those used by other networks, strengthening layer interactions leads to overfitting on ResNet. Pretraining as most of our baseline models and the above attention models are not pretrained on larger datasets, these measures will result in unfair comparison. Hence we more efficient strategy: applying stochastic depth (Huang et al., 2016) with survival probability of 0.8 only on our MRLA, the effects of which will be discussed the ablation study later.
larger dataset and using extra training settings can be option; however,
EfficientNet For training EfficientNet with our MRLA, we follow the settings in EfficientNet. Specifically, networks are trained within 350 epochs using RMSProp optimizer with momentum of 0.9, decay of 0.9, batch norm momentum of 0.99, weight decay of 4e-5 and mini-batch size of 4096. The initial learning rate is set to 0.256, and decayed by 0.97 every 2.4 epochs. Since our computational resources hardly support the original batch size, we lincarly scale the initial learning rate and the batch size to 0.048 and 768, respectively. We also use Auto-Augment (Cubuk et al., 2211 stochastic depth (Huang al.) 2016) with survival probability 0.8 and dropout (Srivastava al.) 2014) ratio 0.2 for EfficientNee-Be and EfficientNet-Bl. Our MRLA shares the same stochastic depth with the building layer of EfficientNet since it is natural drop the MRLA block if the corresponding layer is dropped. We implement EfficientNers with these training tricks by pytorch-image-modde, (timm) toolkit (Wightman, 2019) on 2x V100 GPUs.
License: Apache License 2.0
19

---

Published as a conference paper at ICLR 2023
DeiT, CeiT and PVTv2 We adopt the same training and augmentation strategy that in DeiT. All models are trained for 300 epochs using the Adam optimizer with weight decay 0.05. We use the cosine learning rate schedule and set the initial learning rate 0.001 with batch size of 1024. Five epochs are used to gradually warm up the learning rate at the beginning of the training. We apply RandAugment (Cubuk al. 2020), repeated augmentation (Hoffer et al.. 2020), label smoothing (Szegedy et al., 2016) with e 0.1, Mixup (hang et al. (017)) with 0.8 probability, Cutmix (Yun a 2019) with 1.0 probability and random erasing (Zhogg al. 2020) with 0.25 probability. Similarly, our MRLA shares the same probability of the stochastic depth with the MHSA and FFN layers of DeiT/CeiT/PVTVV. xNote that since PVTv2 is a mtlti-stage architecture and the size of feature maps differs across the stage, we perform layer attention within each stage. DeiT and CeiT have 12 layers in total and we partition them into three stages, each with four layers, and apply the MRLA within the stage.
B.1.2 MODEL COMPLEXITY WITH RESPECT TO INPUT RESOLUTION
Figure visualizes the FLOPs induced by MRLA-light with respect the input resolution. We compute the FLOPs of the baseline CeiT-T and our MRLA-light counterpart and then derive their differences under various settings of input resolution. It can be observed that the complexity of MRLA-light is linear to the input resolution.
FLOPs induced by MRLA-light (B)
0.05
0.04
0.03
0.02
512 512
0.01
0.00
112x112
768 768
Input resolution
Figure 5: The FL.OPs induced by MRIA--lght with respect to input resolution.
B.1.3 COMPARISONS WITH RELEVANT NETWORKS
Layer-interation-rllted Networks We first compare our MRLAs with DenseNer(Huung et al. 2017). DIANet (Huang et al. 2020) and RLA,Net (Zhao et al. 202I) empirically. Their comparisons on the ImageNet- IK validation set given in Table Our MRLAs outperform the DIANet and Net, all of which beat the DenseNet of similar model size.
Other Relevant Networks TDAM JJiswal et al. 2022) and BA-Net (Zhao et al. 2022) adopted different implemental settings from ours in the training of ImageNet. For the baseline model, we used the results from the torchvision toolkit while TDAM utilized those from the shorch-image-modeee (timm). Note that the latter implementation includes advanced design settings (e.g., three 3x3 Convs instead of 7x7 Conv) and training tricks (e.g., cosine learning schedule and label smoothing) to improve the performance ResNets. And BA-Net applied cosine learning schedule and label smoothing in their training process. Therefore, it would be unfair directly evaluate the performances between TDAM, BA-Net and MRLAs using the current results. We reproduced the performance of BA-Net with our training settings and train model with the settings used in BA-Net. The results are given in Table 6 and we also compare the performances though we do not train our model by using the better pre-trained weights in object detection and instance segmentation tasks.
20

---

Published as a conference paper at ICLR 2023
Table 5: Performances of layer-interation--elaee networks the ImageNet- 1 K validation set.
Model
Params
FLOPs
Top-1
Top-5
ResNet-50
25.6 M
4.1 B
76.1
92.9
DIA (Huang al.
2020
28.4 M
77.2
RLA, Zhao al
25.9 M
4.5 B
77.2
93.4
MRLA base (Ours)
25.7 M
4.6 B
77.7
93.9
-MRLA-light (Ours)
25.7 M
4.2 B
77.7
93.8
ResNet-101
44.5 M
7.8
77.4
93.5
+RLA, Zhao et al.
2021
45.0 M
8,4 B
78.5
94.2
MRLA light (Ours)
44.9 M
7.9
78.7
94.4
DenseNet- 161 (k=48)
Huang et al.
2017
27 M
7.9 B
77.7
93.8
DenseNet-266 (k=32)
Huang
2017
31.8 M
5.9 B
77.9
93.8
Table 6: Comparisons of parameters, FLOPs and memory cost ResNet-50 and ResNet-101 by training on ImageNer-IK. denotes training with the implemental settings in BA-Net.
Model
Params
FLOPPs
Input
Top-1
Top-5
BA-Net-50 ZZao R50 MRLA-light (Ours)
2022
28.7 M
4.2 B
224
77.8
93.7
25.7 M
4.2 B
224
77.7
93.8
BA-Net-50' Zhao et al, 2022
28.7
4.2
224
78.9
94.3
R50 MRLA-light (Ours)
25.7 M
4.2 B
224
78.7
94.4
B.2 OUT-OF-MEMORY PROBLEM AND THE NECESSITY OF MRLA-LIGHT
Though it is possible to train naive quadratic version with MLA (Eq. and MRLA-base (Eq. (6) on ResNet-50 and obtain good results, it will lead to the out-of-memory (OOM) problem on ResNet- if we keep the same batch size of 256. Here, compare the parameters, FLOPs and memory cost per GPU (all models are trained on V100 GPUs) of MLA, MRLA-base and MRLA-light support claim.
Table 7: Comparisons of parameters, FLOPs and memory cost ResNet-50 and ResNet-101 by training on ImageNet-IK:
Model
Params
FLOPs
Memory (MiB)
ResNet-50 with MLA
25.7 M
4.8
18.2 K
ResNet-50 with MRLA-base
25.7
4.6 B
13.8 K
ResNet-50 with MRLA-light
25.7 M
4.2
12.2 K
ResNet- 101 with MLA
44.9 M
9.1
OOM
ResNet-101 with MRLA- Abbase
44.9 M
8.5
OOM
ResNet- with MRLA-light
44.9
7.9
17.7 K
Table 7 shows that MLA and MRLA--ase demand more extra FLOPs and memory than MRLA- light when the network becomes deeper. Moreover, ResNet-50 with MLA and MRLA-base cost 30% and 20% more time during the training period. Since ResNet- 101 has 23 building blocks in the third stage, the layer attention for the last layer should attend the features stacking all previous 22 layers' features. Unless we manually split them into several sub-stages for a deep network, MLA and MRLA-base for the last layer in the stage should attend the features stacking all previous layers' features, leading to the OOM problem. In general, stacking features layer by layer is unfriendly to deep network as it incurs sigmificant computation, memory and time costs.
B.3 OBJECT DETECTION AND INSTANCE SEGMENTATION ON COCO
We adopt the commonly used settings Hu et al. 2018 Wang et al. 2018 Cao al. 2019 Wang al. 2020b Zhao et al. 202T), which the default settings MeDetectin
Implementation details
21

---

Published as a conference paper at ICLR 2023
Table 8: Complete results of Mask R-CNN object detection using different methods. The bold fonts denote the best performances.
Methods
Params
GFLOPs
Apa
AP
50
AP76
APS
APM
API.
ResNet-50
44.18 M
275.58
37.2
58.9
40.3
22.2
40.7
48.0
SE (Hu et al. 2018
46.67
275.69
38.7
60.9
42.1
23.4
42.7
50.0
ECA Wang al.
2020b
44.18
275.69
61.3
42.8
49.9
+ NL Wang et al.
2018
46.50 M
288.70
38.0
59.8
41.0
GC (rl6) Cao a
2019
46,90 M
279.60
39.4
61.6
42.4
GC (r4) (C (Ca et
2019
54.40 M
279.60
39.9
62.2
42.9
+ RLA,, (Zhao al.
44.43
283.06
39.5
60.1
+ BA (Zhao et al. 2022)
47.3 OM
261.98
40.5
61.7
44.2
24.5
44.3
52.1
+ MRLA-light (Ours) 101
44.34 M
276.93
41.2
62.3
45.1
24.8
44.6
53.5
63.17 M
351.65
39.4
60.9
43.3
23,0
43.7
51.4
+ (Hu et al. 2018
67.89
351.84
40.7
62.5
23,9
45.2
52.8
+ ECA Wang ci al.
2020b
63.17
351.83
41.3
63.1
44.8
25.1
45.8
52.9
+ NL Wang ct al.
2018
65.49 M
364.77
40.8
63.1
44.5
GC (r16) Cao
2019
68.10 M
354.30
41.1
63.6
45.0
GC (r4) (C
2019
82.20 M
354.30
41.7
63.7
45.5
R ZZao et al.
63.56
362.55
41.8
62.3
+ MRLA-Iight (Ours)
63.54 M
353.84
42.8
63.6
46.5
25.5
46.7
55.2
toolkit (Chen al. 2019). Specifically, the shorter side input images are resized 800, then all detectors are optimized using SGD with weight decay of le-4, momentum of 0.9 and batch size of 16. The learning rate initialized 0.02 and is decreased by factor of 10 after and epochs, respectively, i.e., the Ix training schedule (12 epochs). For RetinaNet, rate to 0.01 to avoid training problems. Since the models no longer suffer from overfitting these transfer learning tasks, remove stochastic depth our MRLA used in ImageNet classification.
modify the initial learning
Results Complete results of Mask R-CNN on object detection and instance segmentation are shown in Tables 8 and for comprehensive comparisons. We observe that our method brings clear improvements over the original ResNet on all the evaluation metrics. Compared with the two channel attention methods SE and ECA, our MRLA achieves more gains for small objects, which are usually more difficult to be detected. Interestingly, even though BA-Net utilizes the better pre-trained weight (the model with 1 superscript 6 object detection and instance segmentation, our approach still outperforms the BA-Net counterpart on most of the metrics.
Besides the comparisons with various attention methods using the same baseline models, also compare our model with different types networks, as shown in Table I0
B.4 DISCUSSION ON ABLATION STUDY
both effective. Before introducing MRLA-light, also proposed the MLA (referring to multi-head version of Eq. (4) Section and the MRLA-base (referring to Eq.
MLA and MRLA-
99 in Section 3.4, both of which rigorously follow the definition of self-attention in Transformer Vaswani et 2017). From Tables and we can observe that MIA and MRLA-base perform better than most of other methods. Besides. there is only a negligible gap between their performance and that of MRLA-light in some cases, which can be attributed to the following reasons:
We directly applied MRLA-light's syper-paramed setting and design to the other two without further tuning.
Benefiting from simpler architecture and learnale X MRLA-light is easier to train and more flexible.
Therefore, if we do not chase the minimum computation, time and memory cost, MLA and MRLA- base are also good choices as they can equivalently enrich the representation power of a network as MRLA-light.
License: Apache License 2.0
22

---

Published as a conference paper at ICLR 2023
Table 9: Complete results of Mask R-CNN instance segmentation using different methods, The bold fonts denote the best performances.
Methods
Params
GFLOPs
AP
AP
AP
APS
AРM
APL
ResNet-50
44.18 M
275.58
34.1
55.5
36.2
16.1
36.7
50.0
SE (uu et al. 2018
46.67 M
275.69
35.4
57.4
37.8
17.1
38.6
51.8
ECA
Wang
2020b
44.18 M
275.69
35.6
58.1
37.7
17.6
39.0
51.8
NL Wane et al. 2018
46.50
288.70
34.7
56.7
36.6
+ GC (rl6) (Ca et a. 22019
46,90 M
279.60
35.7
58.4
37.6
GC (14) (Cao et al 2019
54.40 M
279.60
36.2
58.7
38,3
RI RLA, (Zhao
2021
44.43 M
283.06
35.6
56.9
38.0
BA Zhha al. (2022
36.6
58.7
38.6
18.2
39.6
52.3
+ MRLA-lIght (Ours)
44.34
276.93
37.1
59.1
39.6
19.5
40.3
52.0
ResNet-10 101
63.17 M
351.65
35.9
57.7
38,4
16.8
39.1
53.6
SE (Hu al. 2018
67.89 M
351.84
36.8
59.3
39.2
17.2
40.3
53.6
BCA Wang
2020b
63.17
351.83
37.4
59.9
39.8
18.8
41.1
54.1
+ NL Wang el al.
2018
65,49
364.77
37.1
59.9
39.2
+ GC (r16) Cao et al 2019
68.10 M
354.30
37.4
60.1
39.6
GC (14) (Cao et 2019)
82.20 M
354.30
37.6
60.5
39.8
RLA,, (Zhao
202T
63.56 M
362.55
37.3
59.2
40.1
BA (Zhao et al. 2022)
38.1
60.6
41.5
54.8
+ MRLA-light (Ours)
63.54 M
353.84
38.4
60.6
41.0
20.4
41.7
54.8
Table 10: Object detection results with different backbones using RetinaNet as a framework on COCO val2017. All models are trained in "Ix" schedule. FLOPs are calculated on 1280 x 800 input. The blue bold fonts denote the best performances, while the bold ones perform comparably.
Backbone Model
Params
GFLOPs
Ap
A
A
AP
AP
APM
APE
ResNet-101 (He
2016)
56.7
315
37.7
57.5
40.4
21.1
42.2
49.5
RelationNctt+ (Chi et al. 2020
39.0
266
39.4
58.2
42.5
ResNeXt- 1eexxt-1113222 -2x4d
Xic ct al. 2017
56.4 M
319
39.9
59.6
42.7
22.3
44.2
52.5
Swin-T (Liu et al. 202T)
38.5 M
245
41.5
62.1
44.2
25.1
44.9
55.5
MRLA-ResNNtt1000 (Ours)
57.1 M
318
41.3
61.4
44.2
24.8
45.6
53.8
Convolutions Transformers The improvement of our MRLA over transformers is not entirely caused by the convolution and the experiments Table II support this point:
We have inserted DWConv layer into DeiT which is a convolution frreee transformer. The result demonstrates that our MRLA outperforms adding the DWConv layer.
We have also applied our MRLA to some convolutional transformers, e.g., CeiT Yuan et al. 2021) and PVTv2 (Wang al., 2022a).. We can find that our MRLA can further boost the performances of these convolutional transformers.
Stochastic Depth Stochastic depth is not the fundamental component that helps MRLA outperform its counterparts. Instead, it is a tool to avoid overfitting (too-high training accuracy) on the middle-size dataset ImageNet-1K. We have found that MLA and MRLA-base also suffer from overfitting problem though they are less severe than that in MRLA-light. As stated in Section | strengthening layer interactions can improve model performances. We then conjecture that the information from previous layers brought by our MRLAs is too strong, leading to the overfitting. The detailed justifications as follows:
Applying stochastic depth with the same survival probability on the "+ DWConv2d" and ECA module does not bring significant improvements (see Table 12 (a) and (c)), proving that stochastic depth itself not the key boosting the model performance.
It is natural to share the same stochastic depth on EfficientNet and vision transformers since the layer attention should not be applied if that layer is dropped.
23

---

Published as a conference paper at ICLR 2023
Table 11: Performances of different transformers with our MRIAA-light and DeïT-T with additional convolutions.
Model
Top-1
Top-5
D (Touvron et al.
2021
72.2
91.1
+ DWConv
72.8
91.7
+ MRLA-light (Ours)
73.4
91.9
C Yuan et al.
202
76.4
93.4
+ -- Ours)
77.4
94.1
CeiT-S Yuan et al. 2021
82.0
95.9
+ MRI
MRI.A-lghtt A-gltght (Ours)
83.2
96.6
PVTv2-BO (Wang et al. + MRI-A-light (Ours)
2022a
70.5
71.5
90.7
PVTv2-B1 (Wang et al
2022a
78.7
MRLA-light (Ours)
79.4
94.9
For object detection and instance segmentation on COCO, we did not observe any overfitting problem when removing the stochastic depth trick. We speculate that the 12-epoch training leads to underfitting since we adopt the standard 1x training schedule. Therefore, there is no need to this trick for these two tasks.
There are indeed other solutions to address the overfi tting problem but they sub-optimal.
We prefer using stochastic depth over pretraining larger datasets because of limited computational resources and time. The ImageNet-22K (14M) and JFT300M (300M) datasets are significantly larger than the ImageNet-1k KK (1.28M). Besides, choosing this more efficient strategy allows fair comparison with current SOTA attention models, most of them not pretrained on these larger datasets.
Mixup augmentation (M) and label smoothing (LS) were also tried prevent overfitting. Using them simultaneously can achieve similar performance to the stochastic depth (see Table 12 (b)). However. these methods influence the entire network instead of our MRLA only, leading to unfair comparisons with other models.
-Manually applying MRLA to partial layers instead of stochastic depth is also feasible. However, it maycost much more time to decide MRLA at which layer should be dropped.
Table 12: Ablation study on the trick of stochastic depth.
Model
Params
FLOPs
Top-1
ResNet-50
25.6 M
4.1 B
76.1
DWConv2d
25.7 M
4.2
76.6
stochastic
25.7 M
4.2
76.9
(b) MRLA-light (M,LS)
25.7 M
4.2
77.9
(c) R50 ECA
25.6 M
4.1 B
77.5
stochastic
25.6 M
4.1
77.5
24

---

Published as a conference paper at ICLR 2023
B.5 VISUALIZATIONS
ResNet
ResNet 50 MRLA-base
ResNet 50 MRLA-light
Stage 3
Stage 4
Stage 3
Stage 4
Figure 6: Visualizations of the feature maps extracted from the end of Stage and 4 of ResNet-50 and our MRLA counterparts.
To investigate how MRLAs contribute to the representation learning in CNNs, we visualize the feature maps with the score-weighted visual explanations yielded by ScoreCAM (Wang et 2020a) in Figure 6 Specifically, we extract the feature maps from the end of each stage in ResNet-50 and MRLA counterparts. We display here the visualizations for Stage and 4 the feature maps from the first two stages are quite similar and focus on low-level features for models. The two example images randomly selected from the ImageNet validation set. In the visualizations, the area with the warmer color contributes more to the classification. We can observe that: (1) The models with MRLAs tend find the critical areas faster than the baseline model. Especially in stage 3, the MRLAs have already moved to emphasize the high-level features while the baseline model still focuses on the lower-level ones. (2)The areas with the red color in ResNet 50 MRLA bbbse/lighhtt models are larger than that in the baseline model, implying that the MRLA counterparts utilize more information for the final decision Docision-making. (3) The patterns of MRLA-base and MRLA-II -light are similar, validating that our approximation in MRLA-light does not sacrifice too much of its ability.
Figure 7 visualizes the attention maps of specified query (red box) from three randomly chosen heads in the last layer of DeiT-T and our MRLA counterparts. The first image randomly sampled from the ImageNet validation set, and the second image is downloaded from a website, In the
25

---

Published as a conference paper at ICLR 2023
visualization, the area with the warmer color has higher attention score. We observe that MRLA can help the network retrieve more task-related local details compared to the baseline model. In other words, the low-level features are better preserved with layer attention.
orig
Heed0
Head1
Head2
DEIF-T + MRLA-lghtt DeIT-T MRLA- b85e
DefF-t
Def-- + NRLA-lIght Deit-T + MRLA- HRLA-base
Figure 7: Visualizations of the attention maps in the last layer of DeiT-T and our MRLA counterpart given a query specified in the red box.
26


Reviews:
[{"review_id": "m5JVs2qRxXN", "reviewer": "Reviewer_Gvtx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-head recurrent layer attention (MRLA) mechanism, which uses the feature of the current layer to query all previous layers to retrieve query-related information of different levels. The paper further proposes a light-weighted MRLA to reduce the computational cost. The proposed method can be applied to both CNN and transformer-based networks. Experiments show its effectiveness in several vision tasks including image classification, object detection, and instance segmentation.", "summary_of_the_review": "I am on the negative side at this point mainly due to the out-of-date literature review, and some experiments that support key assumptions/claims are missing.", "strength_and_weaknesses": "Strong points.\n\n1. The paper proposes a new attention mechanism by formulating each layer as a token and querying all previous layers.\n2. The paper formulates extends the proposed layer attention to have multi-head and be light-weighted.\n3. Code is submitted.\n\n\nWeak points.\n\n1. It is not clear to me why Eq.5 can be derived from Eq.3. More explanation would be better. Besides, it seems there are some symbol ambiguities, for example, it is not clear what the differences are between f and F. in Eq.3 and Eq.5, and what is s in Eq. 1 and Eq. 4.\n\n2. Section 3.3 mentions an assumption that query vectors at two consecutive layers have a similar pattern, based on which the linearization is performed by using a learnable parameter. However, such an assumption is not justified. The paper should also compare the method proposed by (Katharopoulos et al. 2020) to demonstrate its own advantages.\n\n3. Almost all papers referred to are published before/in 2021. A more comprehensive literature review and experimental comparisons are essential in order to evaluate its novelties.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ouvi5a4ivf", "reviewer": "Reviewer_i4yM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel method for cross-layer interaction, which complements current mainstream networks emphasizing the interaction within a layer. Taking advantage of the attention mechanism, the proposed method enhances the layer interaction via attention. An efficient implementation is also introduced to avoid the vanilla quadratic complexity. Experimental results demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "I think the paper is quite good, and I recommend to accept the paper if the authors can address my two primary concerns above. Also, I would like to have further communication with other reviewers and the AC.", "strength_and_weaknesses": "[ Strength ]\n+ The motivation is very attractive. It is common that most of existing networks, including Transformers, only focus on the interaction within a certain layer. Even though ResNet and DenseNet, as analyzed by the authors, put some emphasis on layer interaction, the way they used (i.e., addition and/or concatenation) is a little bit naive and hard. The proposed method takes advantage of the attention mechanism and makes the cross-layer interaction more technically rational. \n+ The method is elegant and the presentation logic is clear. It is quite straightforward to understand each part of the method section. The proposed efficient implementation of layer attention, just like linear attention in efficient Transformers, is useful and necessary. \n+ Sufficient experiments demonstrate the effectiveness of the proposed method. The performance on image classification, object detection and instance segmentation indicates that the method is indeed effective and efficient. \n\n[ Weakness ]\n- On image classification, the input resolution is identically set as 224. What if enlarging the resolution? In fact, it is quite important to compare the accuracy and efficiency with various resolutions to reveal the robustness. Of course, it is necessary to check whether the complexity is linear (or nearly linear) to the input resolution.\n- Visual comparison is missing. The proposed cross-layer attention can enhance the interaction between layers, but there is no strong evidence specially for this interaction. For example, are local features from shallow layers transferred to top layers? How do high-level features facilitate the feature representation in shallow layers? These would need visual comparison.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mbz2HSd6an", "reviewer": "Reviewer_nNZw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a network family named Multi-head Recurrent Layer Attention (MRLA). It has two module variants: MRLA and MRLA-light. Both of the variants take a self-attention format. A major difference is that MRLA looks back at its previous module’s keys and values, while the light variant only uses the input features for the current layer. The self-attention takes a straightforward format where keys and queries are after global average pooling, and values are pixel features. The modules are tested with both convolutional networks and the vision transformers, and show consistent improvements over the baselines.", "summary_of_the_review": "The paper proposes a network family for cross-layer attention. The paper is clearly written and the results look solid. What bothers me is that the major topics and claims are not well supported by the network implementation and the experimental evidence.", "strength_and_weaknesses": "The paper presents a network family that combines the attention, densely connected networks, SENet into one unified framework. The overall performance is good, and it’s nice to see this method can work fine with the recent vision transformers given their basic building block is already self-attention. The idea seems simple and straightforward, yet providing performance improvements at little additional costs.\n\nThere are some problems about how this paper is formed and how the claims are supported by the experimental evidence.\n\nThe first concept I think is `recurrent`. The overall framework could be implemented as a recurrent module. Ignore this comment if I’m wrong but I think the codes show that each building block has its own MRLA instance meaning that the variables are not shared. This leaves a question: does stacking up building blocks make them `recurrent`?\n\nBy the way, I think the paper could do better to give us a clearer picture of the underlying implementation by providing some pseudo codes for the two variants in the main paper or appendix. Fortunately, the authors uploaded their codes. So understanding them hasn’t been a problem.\n\nThe second problem is the position of the MRLA. The MRLA-light is a self-attention module implemented in the SENet style using sigmoid to replace softmax. It does not look back explicitly, does not compute features recurrently, yet performs similarly as or better than MRLA. This put MRLA in an awkward position that the method following the main topic of the paper more does not show strong advantages over the light variant that follows less.\n\nFinally, I think it will strengthen the paper by showing performance improvements for larger models. The claims are mostly based on experimental results from models at small model scales. Larger model comparisons are necessary for methods that add complexity to the baselines.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "6rjbb-dhVz-", "reviewer": "Reviewer_XdGN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a module to perform layer attention for CNNs and Vision Transformers in linear-complexity. The core idea is to leverage the representations of previous layers in a recurrent form so that the quadratic-complexity self-attention is avoided. Experiments on image classification, object detection, and instance segmentation are performed to validate the effectiveness of the proposed module.", "summary_of_the_review": "Though the novelty and performance improvement over SOTA is minor, the work contains solid experiments and comes with good presentation, which provides insights into designing layer-wise connected attention. The idea is straightforward and easy to implement.  Overall, I think it reaches the acceptance bar. ", "strength_and_weaknesses": "Strength\n- The core idea is straightforward to implement.\n- A wide range of experiments (image classification, object detection, and instance segmentation) are performed on large-scale datasets including ImageNet-1K and COCO. Also, the same module has been tested on quite a few backbone networks including many common variants of ResNet and Vision Transformers.\n- The paper is clearly written.\n\nWeakness\n- Honestly, I reviewed this paper in the last NeurIPS 2022 and thought the work quality, e.g. solid experiments, well-written,  reached the weak acceptance bar at that time. Given the edited version, the authors have addressed my previous concerns about the experiments. However, my main concerns about this work are still reserved. The novelty of applying layer-wise connection to improve attention is limited, which has been proved by the existing works, e.g, denseNet. And the marginal improvement over existing methods makes it stay slightly above the borderline.\n- The ablation study of different layer connections impacting performance is missing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster