Siamese Attention Networks
Authors: Hongyang Gao
Yaochen Xie
Shuiwang Ji
Venue:  (2020)

Abstract: Attention operators have been widely applied on data of various orders and dimensions such as texts, images, and videos. One challenge of applying attention operators is the excessive usage of computational resources. This is due to the usage of dot product and softmax operator when computing similarity scores. In this work, we propose the Siamese similarity function that uses a feed-forward network to compute similarity scores. This results in the Siamese attention operator (SAO). In particular, SAO leads to a dramatic reduction in the requirement of computational resources. Experimental results show that our SAO can save 94% memory usage and speed up the computation by a factor of 58 compared to the regular attention operator. The computational advantage of SAO is even larger on higher-order and higher-dimensional data. Results on image classification and restoration tasks demonstrate that networks with SAOs are as effective as models with regular attention operator, while significantly outperform those without attention operators.

Under review as a conference paper at ICLR 2020
SIAMESE ATTENTION NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Attention operators have been widely applied on data of various orders and di- mensions such as texts, images, and videos. One challenge of applying attention operators is the excessive usage of computational resources. This is due the usage of dot product and softmax operator when computing similarity scores. In this work, propose the Siamese similarity function that a feed-forward network to compute similarity scores. This results in the Siamese attention opera- tor (SAO). In particular, SAO leads to a dramatic reduction in the requirement of computational resources. Experimental results show that our SAO can save 94% memory usage and speed up the computation by factor of 58 compared the regular attention operator. The computational advantage of SAO is even larger on higher-order and gigher-dimensionl data. Results on image classification and restoration tasks demonstrate that networks with SAOs are as effective as mod- els with regular attention operator, while significantly outperform those without attention operators.
INTRODUCTION
Deep learning networks with attention operators have demonstrated great capabilities of solving challenging problems in various tasks such as computer vision (Xu et al. 2015; Lu et al. (2016), natural language processing (Bahdanau et al. 2015: Vaswani al.) 2017 and network embed- ding (Veličković et al. (017). Attention operators are capable of capturing long-range relationships and brings significant performance boosts (Li et al. 2018 Malinowski al., 2018). The application scenarios of attention operators range from 1-D data like texts to high-order and high-dimensional data such as images and videos. However, attention operators suffer from the excessive usage of computational resources when applied on high-order or nigh-dimensional data. The memory and computational cost increases dramatically with the increase of input orders and dimensions. This prevents attention operators from being applied in broader scenarios. To address this limitation, some studies focus on reducing spatial sizes inputs such as down-sampling input data (Wang et al. (018) or attending selected part of data (Huang et al. 2018). However, such kind of methods inevitably results in information and performance loss.
In this work, we propose a novel and efficient attention operator known as Siamese attention opera- tor (SAO) to dramatically reduce the usage computational resources. We observe that the excessive computational resource usage is mainly caused by the similarity function and coefficients normaliza- tion function used in attention operators. To address this limitation, we propose the Siamese similarity function that employs feed-forward network compute similarity scores. By applying the same network to both input vectors, Siamese similarity function processes the symmetry property. By using Siamese similarity function compute similarity scores, we propose the Siamese attention operator, which results in a significant saving on computational resources. Based on the Siamese attention operator, we design family of efficient modules, which leads to our compact deep models known as Siamese attention networks (SANets). Our SANets significantly outperform other state-of- state-of-the-arr compact models on image classification tasks. Experiments on image restoration tasks demonstrate that our methods are efficient and effective in general application scenarios.
2 BACKGROUND AND RELATED WORK
In this section, we describe the attention operator that has been widely applied in various tasks and on various types of data including texts, images, and videos.

---

Under review as a conference paper at ICLR 2020
2.1 ATTENTION OPERATOR
The inputs of an attention operator include three matrices; those are a query matrix Q [qi. 92.... 9m] Rdxm with each q: R a key matrix K = (k,,,,.... ,k, Rdx= with each k, E R", and value matrix V [V1, V2,".. , Vnl E R with each V, E Rº. To compute the response of each query vector q;, the attention operator calculates similarity scores between q and each key vector k; using similarity function. Frequently used similarity functions include dot product (Luong et al., 2015). concatenation (Bahdanau al. (015), Gaussian function, and embedded Gaussian function. It has been shown that dot product is the most effective one (Wang) et al. 2018), which computes sim(q:, kj) = 5 q. After the normalization with a softmax operator, the response is computed by taking a weighted sum over value vectors دド= uysoftmax (kj q.). For all query vectors, the attention operator computes
O Vsoftmax (K" (T),,
(I)
where softmax(-) is the column- oolumn-wise softmax operator. The matrix multiplication between K" and Q computes a intermediate output matrix E that stores similarity scores between each query vector qi and each key vector k,.. The column-wise softmax operator normalizes E and makes every column sum to Multiplication between V and the normalized E gives the output O E R Self-attention operator (Vaswani et al.] 2017; Devlin et al.] 2018) is special case of the attention usually firstly perform linear transformations on input
operator with Q K V. In practice. matrices. For notation simplicity, we use original input matrices in following discussions. The computational cost of the operations in Eq. [ is (m n x (d + p)), and the memory required to store the intermediate output E is (m x n2). If m n and d = the time and space complexities of Eq. I1 are O(n² d) and ( respectively.
The matrix multiplication order in Eq. T is determined by the softmax operator, which acts as the normalization function. [Wang al. (0118) proposed to use scaling by 1/N as the normalization function on similarity scores. By this, the response of q: is calculated N Σ}= uzke qc. The attention operator using scaling by 1/N computes all responses as:
o = N (V N
(2)
By computing VKT first, the time and space complexities of Eq. O(Nd)) and (dd), respectively. When N > d, this saves computational resources compared to the attention operator in Eq. [ In practice, we usually have N in some parts of neural network, especially on high-order data.
2.2 ATTENTION OPERATORS ON HIGH-ORDER DATA
Non-local operators Wang et al. 2018) are essentially self-attention operators on high-order data like images and videos. Take 2-D data as an example, the input to a non-local operator is an image, which can be represented as a third-order tensor X E hxe Here, h, w, and c denote the height, width, and number of channels, respectively. The non-local operator converts the tensor into a matrix X(3) E Rexhw by unfolding along mode-3 (Kolda & Bader 2009). Then the matrix is fed into an attention operator by setting Q Q= K V= (3)) The output of the attention operator is converted back to a third-order tensor that is used the final output. A challenging problem of non-local operators is the excessive usage of computational resources. If h = w, the time and space complexity of the non-local operator is O x d) and O(h2), respectively. The computational cost becomes even bigger on higher-order data like videos. The excessive usage of computational resources limits the application of attention operators in broader scenarios.
SIAMESE ATTENTION NETWORKS
In this work, we propose learnable similarity function known as the Siamese similarity function. This function uses single-layer ferd-forward network compute similarity scores. Based on Siamese similarity function, we propose the Siamese attention operator, which dramatically reduces computational cost. We also describe how to build Siamese attention networks using this operator.
2

---

Under review as a conference paper at ICLR 2020
81
8
b
by
b4
b
b
by
(b)
Figure 1: Illustration of the similarity function using a single-layer feed-forware network (a) and the Siamese similarity function (b). In (a), different input elements use different weights. In (b), two input vectors share the same network. The lines with the same color indicate weights sharing.
3.1 STAMESE SIMILARITY FUNCTION
We analyze the problem of learning a similarity function given two vectors. To learn a similarity function, we employ single-layer feed-forward neural network. Given two vectors Rd and E TRd, the similarity score is computed using a trainable vector w E 1R2d as:
sim (a, b) = |aa b²
x w. + b; X whti = a
wa + buw,. wb.
(3)
where [w "f with w. Rd and w, Rd. Here, ignore the bias term for notation simplicity. We consider the similarity function defined in Eq. B as two feed-forward networks that process two vectors separately. The similarity score the sum of outputs from two different networks.
Unlike distance metrics, the non-negativity or triangle inequality do not need to hold from similarity functions. But we usually expect similarity measures to be symmetric, which means it outputs the same similarity score when two input arguments are swapped. Apparently, the similarity function defined by Eq. (3) does not have this property. To retain the symmetry property, we employ the same network while using both vectors to compute the similarity score. This leads to our proposed Siamese similarity function (Sia-sim), which follows the principle of Siamese networks (Bromley et al. 1994 Bertinetto et al. 2016). The Siamese similarity function computes the similarity score between a and b as:
Sia-sim (a, b)
Σ (a; b:) x wi; (a + b)= w. Sia-sii (b, (b, a)
where Rd is a trainable parameter vector. Although the time complexity of computing Sia-sim is the same as that of dot product, we show that Sia-sim leads to very efficient attention operator in Section 3.2 Figure provides an illustration of the similarity functions defined in Eq. and Eq.
3.2 SIAMESE ATTENTION OPERATOR
We describe the Siamese attention operator in the context of 1-D data, but it can be easily applied on high-order data by unfolding them into matrices. In this case, the inputs to an attention operator are Q E RdxN K E RdxN and V RdxN We replace the similarity function in the attention operator by our Siamese similarity function, leading to the Siamese attention operator (SAO). Given query vector q, in Q, SAO computes the response o,
N
Ug(g: + kTy
(1.g) u,kk w)
1=1
(5)
N
N Σ
1
=
Σ Bj
q. w
+
w
=ww- qi + N
VK"w, w.
N
N
j=l
j=l
where 8 =
1 N E
VgJ
Rdd SAO computes responses of all query vectors as:
= T Q
+
VK
w1
T
(6)
N
3

---

Under review as a conference paper at ICLR 2020
Table I: Comparisons among three attention operators in terms of time and space complexities. Atm and / denote the regular attention operators using softmax and scaling by 1/N for similarity scores normalization, respectively.
Operator
Computation
Time Complexity
Space Complexity
Attn
ssotmax K ("Q) N (VK")) Q
O(N x d)
O(N2)
Attn//N
O(N x d))
O
SAO
vw Q + VVK w1
O(N d)
O(N x d)
where I v is vector of ones of size N.
Note that we use 1N here to make it mathematically precise. In practice, the term N VKT w is the same all query vectors. This means we only need to compute it once and share it for the computation of all responses. By computing K" w first, the time complexity of this term is O(N x d). Similarly, the time complexity for computing the first term in Eq. 66) is O(N d). Thus, the overall time complexity of SAO is O(N x d). Notably, when Q K. we can save the computational cost by computing either w T Q or KT w. Table I provides the comparison of SAO and other attention operators. It can be seen from the comparison results that our SAO can significantly save computational resources compared to other attention operators.
In Eq. (5), the first response term Twr changes as the query vector 4. which we call a local response term. The second term & V K u is the same for all query vectors, which global response term. The local response term provides customized information to query vectors, while the global response term may include global information for SAO. In the experimental study part, we demonstrate the importance of the global response term to SAO.
3.3 SIAMESE ATTENTION MODULES
Attention models have been shown to be effective in various fields. However, they are not widely applied in compact neural networks to date primarily due to their excessive computational cost. Based on our efficient SAO, it is possible to integrate attention operators in compact convolutional neural networks (CNNs) such as MobileNetV2 (Sandler et al., 2018). In this section, we design family of efficient Siamese attention modules (SAMs) that integrate our SAO with convolutional operators. These modules can be used for designing compact CNNs.
MaseMedule: Bottleneck blocks with inverted residuals are main components in MobileNetV2. Each bottleneck consists of three layers; those are an 1x1 convolutional layer for expansion, a 3x3 depth-wise convolutional layer, and another 1 convolutional layer. The hyper-parameten of this module include the expansion factor and the stride s. Given the input tensor x(() E hx for the éth block, the first Ixl convolutional layer expands it into (r - 1)e feature maps. We then concatenate the output with the input, resulting in X E RRxxxxre The depth- wise convolutional layer with stride outputs re feature maps x E layer outputs d feature maps y()) E RR + x xd A skip connection is applied between x(1) and (()) when c = d and 1. The BaseModule is illustrated in (
Xre, Finally, the last 1 convolutional
We propose to integrate our SAO into the BaseModule, resulting in AttnModule as shown in ( Before the last 1x convolutional layer, add new parallel path by SAO, which outputs e feature maps. An average pooling layer with stride is followed when > 1, resulting in (x)) TR. #xe In the original path, both the first 1 convolutional layer and the depth-wise convolutional layer output (r 1)c feature maps x((") E R We concatenate and feed them into the last 1 Ix convolutional layer, which outputs feature maps y( E 3 x xd In SAOs, we only apply linear transformation and the number of trainable parameters. The original path extracts locality- locality-based features, while the new path using SAO computes global features. In this way, AttnModule is capable of capturing both local and global information. We add a skip connection between inputs and outputs of SAO to enable better feature reuse and gradient back-propagation (He et al. 2016a).
Module:
the value matrix V to limit the computational cost
4

---

Under review as a conference paper at ICLR 2020
xxwxc
XXWXC
Conv x stride BN-ReLU6
Cony 1 stride -1
BN+ReLU6
DWConv x stride BN-ReLU6
DWConv 3 stride BN-ReLUU6
Conv 1 stride
Conv 1 I stride = output
output d
W
W
xd
xd
(a)
(b)
Figure 2: Architectures of the BaseModule (a) and AttnModule (b) introduced in Section 3.3 The skip connection between inputs and outputs is used when and e = The one between inputs and outputs of SAO is used when s 1.
3.4 SIAMESE ATTENTION NETWORKS
In this section, we describe the model architecture of a compact convolutional neural network using our proposed SAOs and modules in Section 3.3 leading Siamese attention networks (SANets). We follow the basic architecture of MobileNetV2 (Sandler et al.) (2018) and replace inverted residual blocks with our SAMs. The details of network architecture are described in Table inble the appendix. Following the practices in Wang et al. (2018), we use AttnModules to replace the blocks with input spatial sizes of x 28, 14 14, and 7. For the rest of the blocks, we use BaseModule with the same expansion factors. The SANet built on SAMs saves about 0.03 million parameters and 12 million MAdd compared to MobileNetV2. We show that SANets with SAOs significantly outperform MobileNetV2 while using less computational resources.
4 EXPERIMENTAL STUDIES
In this section, we evaluate our methods using image classification and restoration tasks. We firstly evaluate our SAO in terms of computational resource usage by comparing with regular attention operators. We compare our designed SANets with compact CNNs on image classification tasks using the ImageNet ILSVRC 2012 dataset (Russakovsky et al., (015). We conduct ablation studies to investigate the benefits of our SAO and the trainable parameters it. We also evaluate methods in general application scenarios using image restoration tasks.
4.1 COMPARISON OF COMPUTATIONAL EFFICIENCY
The theoretical analysis in Section 3.2 shows that our SAO can achieve significant efficiency ad- vantages over regular attention operator especially on high-order and high- -ich-immensional data. In this section, we use simulated experiments to evaluate the efficiency advantage of SAO over other attention operators. We build networks using different attention operators. The network composed of single attention operator to eliminate the influence of other factors. We apply the TensorFlow profile tool (Abadi et al. (2016) and report the required memory and CPU time on 2-D simulated data. To fully evaluate the efficiency of attention operators, vary the number of channels and spatial sizes of the input data.
The comparison results is summarized in Table 2 On the simulated data with the size of 562 x 256, our SAO achieves 94.65% memory saving and 58.21 times speed-up compared to the regular attention operator. Our propose SAO can significantly save computational resources. The efficiency advantage
5

---

Under review as a conference paper at ICLR 2020
Table 2: Comparisons between the regular attention operator, the regular attention operator with pooling operation (Wang et al., 2018), and our proposed SAO in terms of the number of parameters, number of MAdd, memory usage, and CPU inference time on simulated data of different sizes. The input sizes are given in the format of "spatial sizes x number of input channels". "Attn" denotes the regular attention operator. "Attn+Pool"" denotes the regular attention operator which employs 2
pooling operations on K and V input matrices to reduce required computational resources.
Input
Operator
MAdd
Memory
Saving
Time
Speedup
Attn
4.92m
20.97MB
0.00%
20.47ms
1.00x
Attn+Pool
1.23m
8.39MB
59.99%
5.07ms
4.04x
14 64
Attn
1.61m
5.24MB
75.01% %
6.97ms
2.94x
SAO
0.05m
4.29MB
79.54%
3.09ms
6.62x
Attn
157.35m
301.99M1
0.00%
450.22ms
1.00x
Attn+Pool
39.34m
100.66MB
66.67%
119.55ms
3.77x
282 x 128
1tt
25.69m
37.75MB
87.50%
70.52ms
6.38x
SAO
0.40m
33,88MB
88.78% %
31.76ms
14.18x
Attn
5,035.26m
5,04GB
0.00%
12.92s
1,00x
Attn+Pool
1,258.82m
L.34GB
73.38%
4.82s
2.68
562
256
Attn /tN
411.04m
0,28GB
94.34%
1.02s
12.66x
SAO
3.21m
0.26GB
94.65% %
0.22s
58.21 x
of SAO over regular attention operators increases as the increase of spatial and dimension sizes. When comparing with other attention operators, our SAO is shown to be the most computationally efficient attention operator. The simulated results show that SAO is efficient attention operator that can operate high-dimensional and high-order data by consuming very few computational resources.
4.2 RESULTS ON IMAGE CLASSIFICATION
Based on efficient SAO, build family of compact CNNs in Section 3.4 for image clas- sification tasks. To evaluate the effectiveness of our SAO and SANets, we compare our models with other compact CNNs (Howard al. 2017
Table 3: Comparisons between SANet and other CNNs in terms of the top-1 accuracy, the number of trainable parameters, and MAdd the ImageNet
validation set.
Model
Top-1
Params
MAdd
Sandler et al. 2018 Zhang et al., 2017
Gao
et al.] on the ImageNet ILSVRC 2012
GoogleNet
0.698
6.8m
1550m
VGGT6
0.715
128m
15300m
age classification dataset. It has been serving as the benchmark dataset, especially for image classification tasks. The dataset contains 1.2 mil- lion images for training, 50 thousand images for validation, and 50 thousand images for testing. We provide the experimental setups on image classification tasks the appendix.
SqueezeNet
0.575
1.3m
833m
MobileNerV1
0.706
4.2m
569m
ChannelNet-vi
0.705
3.7m
407m
ShuffleNet 1.5x
0.715
3.4m
292m
MobileNetV2
0.720
3.47m
300m
SANet (ours)
0.730
3.44m
288m
We compare our SANets with other compact CNNs and report the top-1 accuracy, the number of parameters, and MAdd in Table 3 Mo- bileNetV2 (Sandler et al., 2018) the previous state-of-the-art model in terms of the computational cost and model performance. Compared to MobileNetV2, our SANet significantly outperforms MobileNetV2 by a margin of 1 % with smaller number of parameters. By using our SAO, SANet can achieve new state-of-the-ant performance with limited computational resources. Considering that we only make minor modifications from the architecture of MobileNetV2, the performance boost is significant. Compared to the module using regular convolutional layers, our proposed module uses SAO to obtain global features and avoids the excessive usage of computational resources. Our SAO successfully overcomes the limitations of regular attention operator and applies attention mechanism high-order and nigh-dimensional data with significant performance boost. Next, will show our SAO is as effective as regular attention operator but can dramatically reduce the usage of computational resources.

---

Under review as a conference paper at ICLR 2020
4.3 COMPARISON WITH REGULAR ATTENTION OPERATORS
We have shown that our SAO has significant ef- ficiency advantages over regular attention oper- ators in Section 4.1 In this section, we conduct experiments to compare our SAO with Squeeze- and-Excite block (Hu et al. 2018) and regular at- tention operators based on SANets using image classification tasks. Besides the regular atten- tion operator, we also consider the one defined in Eq. ( using scaling by I/N for coefficients normalization, and the one with a pooling oper- ation as in Wang al. (2018). In the attention operator with pooling operations, 2 x 2 pool- ing operations
Table 4: Comparisons between SANet, the net- work using the same architecture as SANet with Squeeze-and-Excite block (denoted as SENet) or with regular attention operators (denoted as At- tnNet), the AttnNet using regular attention opera- tors with a pooling operation (AttnNet+Pool), and the Attneet with regular attention operators using scaling by 1/N (denoted as AttnNet in terms of the top-1 accuracy, the number of total parameters,
and MAdd on the ImageNet validation set.
Model
Top-1
Params
MAdd
applied to key matrix K and value matrix V. leading to reduced spatial sizes. To ensure fair comparisons, we replace all SAOs in SANets with attention operators, attention op- erators with pooling operations, and attention operators defined in Eq. (2) denoted as AttnNet,
SENet
0.724
3.44m
294m
AttnNet
0.730
3.44m
365m
PttnnNetPPool
0.729
3.44m
300m
AttnNet1/N
0.730
3.44m
312m
SANet
0.730
3.44m
288m
PttnNet+Pool, and AttnNet,/N /N. respectively. We also compare the actual computational usage among MobileNetV2, SANet and AttnNet.
We summarize the comparison results in Table and Table 5 We can observe from the results that our SANet using SAOs achieves the same performance as AttnNet with regular attention operators but using significantly less computa- tional resources. Compared to MobileNetV2, our SANet uses slightly extra resources but achieves significantly better performances. The results demonstrate that our SAO is as effective as the regular attention operator while dramat- ically reduces computational costs. The better performance of SANet over pttoNet+pool indi-
Table 5:
Comparisons among MobileNetV2, SANet, and AttnNet in terms of top-1 accuracy, memory usage, and execution time. We report the actual memory usage using inputs of 128 x 224² x
3. The time reported here is the model execution
time of 100 iterations.
Model
Top-1
Memory
Time
MobileNetV2
0.720
7,4GB
14.4s
SANet AttnNet
0.730
7.5GB
14.9s
0.730
11.7GB
27.1s
cates that our SAO is more efficient and effective compared to the regular attention operator using pooling operations. Note that AttnNet1/N and SANet achieve the same performance as AttnNet, which demonstrates the feasibility of replacing the expensive softmax(-) function with scaling by 1 1/N. Notably, all networks with attention operators significantly outperform SENet which squeezes the spatial information. This demonstrates the effectiveness of attention operators that effectively build long-range relationships and lead to better performances.
4.4 ABLATION STUDIES
To fully demonstrate the effectiveness of our SAO, we conduct ablation studies based on Mo- bileNetV2 and SANets. We add SAOs to Mo- bileNetV2 to observe if it benefits other net- works like MobileNetV2. We replace some in- verted residual blocks with our AttnModules by referencing the structure of SANet, which we denote as MobileNetV2 w SAO. We remove SAOs from SANet by replacing AttnModule with BaseModule, resulting in SANet w/o SAO. We also explore the impact of the global item in SAO on the performance of SANets.
Table 6: Comparisons between Mob, MobileNetV2 with SAOS (MobileNalV2+SAO), SANet, SANet without SAO (SANet w/o SAO), and SANet without the global item in Eq. (5) in
terms of the top-1 accuracy, and the number of total parameters on the ImageNet validation set.
Model
Top-1
Params
SANet
0.730
3,44m
SANet w/o SAO
0.721
3.44m
SANet w/o global term
0.728
3.44m
MobileNerV24SAM
0.727
3.47m
MobileNetV2
0.720
3.47m
The comparison results are summarized in Ta- ble 6 By using SAOs, MobileNetV2 w SAO obtains performance boost of 0.7% over that of the original MobileNetV2, The performance of
7

---

Under review as a conference paper at ICLR 2020
Table 7: Comparison results between U-Net, U-Net with regular attention operator (U-Net+Attn), and U-Net with SAO (U-Net+SAO) in terms of peak signal to noise ratio (PSNR), structural similarity index (SSIM), and normalized root mean square error (NRMSE). On the metries followed by an t, the higher value indicates the better performance. Conversely, lower value indicates the better
performance on the metrics followed by
Dataset
Metric
U-Net
U-Net+Attn
U-Net+SAO
PSNRT
31,571
31,951
31.929
Planaria
SSIM
0.7707
0.7919
0.7898
NRMSEL
0.0268
0.0257
0.0257
PSNRT
32.433
32.571
32.549
Tribolium
SSIM
0.9171
0,9201
0,9196
NRMSEL
0.0241
0.0236
0.0239
PSNRT
21.958
22.361
22.329
Flywing
SSIM
0,5592
0.5903
0.5916
NRMSEJ
0.0798
0.0763
0,0767
SANet w/o SAO without using SAOs is 0.9% lower than that of SANet. Notably, SANet consumes the least computational costs while achieving the best performance. The results demonstrate that our SAO is consistently efficient and effective when being applied in different network architectures. Notably, it can be observed that the global item is important by comparing the performances between SANet and SANet w/o global term.
4.5 RESULTS ON 3-D IMAGE RESTORATION
In order to evaluate the effectiveness of our SAO in broader application scenarios, we conduct experiments on biological image restoration tasks, in particular, the 3D image denoising and the 3D image projection. The projection models map a noisy 3D image to 2D plane, i.c., mapping 3D images from A gxxwxx to 2D images in RAx We perform experiments on three different datasets collected by Weigert et al. (018);; those are Planaria, Tribolium and Flywing. Details of dataset are shown in Table i0 in the appendix. We adopt a general U-Net architecture (Ronneberger et al.) 2015; Çişek et al. 2016) as baseline model. We add attention operators and our SAOs in bottom and decoder blocks. We apply linear transformations on Q. K and V attention operators. The details of experimental setups provided in the appendix. The comparison results are summarized in Table 7 Clearly, U-Net models with attention operators outperform the baseline model, which demonstrates the effectiveness of attention mechanism. Compared U-Net with regular attention operators, SAOs result in similar results. This indicates our SAO is as effective as regular attention operators. Overall, these experimental results demonstrate that our SAO retains its effectiveness in broader application scenarios.
4.6 PARAMETER STUDY OF SAO
Since SAO involves extra trainable parameters our Siamese similarity function, we study the impact of these trainable parameters in SANets. We use an all-ones vector to replace the trainable parameters w in Eq. (6). We denote the resulting model as SANet w/o params. Suble&rreport the comparison results. We can observe that SANet outperforms SANet w/o params by a margin of 0.2% with regard to the top-1 accuracy using only 960 more parameters. This demonstrates that the trainable parameters SAOs are necessary since the importance of different features are weighted.
Table 8: Comparisons between SANet and the SANet removing trainable parameters in SAOs (SANet w/o params) in terms of the top-1 accuracy and number of parameters.
Model
Top-1
Params
SANet
0.730
3,449,408
SANet w/o params
0.728
3,448.448
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSIONS
In this work, we propose Siamese attention operators to overcome the excessive usage of com- putational resources of regular attention operators when being applied on high-order and/or high- dimensional data. We observe that the similarity score in the attention operator is computed using the dot product, which leads to high computational cost. To address this, we propose Siamese similarity function, which employs feed-forward network to compute the similarity score between two input vectors. By using the shared network on two vectors, Siamese similarity function is symmetric. We use Siamese similarity function to replace dot product in regular attention operator, leading to Siamese attention operator. Theoretical analysis and experimental studies show that our SAOs signifi- cantly reduce the usage of computational resources. Based on SAOS, design family of efficient modules, leading to SANets. The evaluation on image classification tasks shows that our SANet significantly outperforms previous state-of-the-art models while using fewer trainable parameters and computational resources. In addition, we conduct experiments on 3-D image restoration to demonstrate the effectiveness our SAOs in broader application scenarios. The parameter study shows that SAO brings great performance improvement with negligible extra trainable parameters.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large-scale machine learning. In OSDI, volume 16, pp. 265-283, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations, 2015.
Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully- convolutional siamese networks for object tracking. In European conference on computer vision, pp. 850-865. Springer, 2016.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. In Advances in neural information processing systems, pp. 737-744, 1994.
Özgüin Çięek, Ahmed Abdulkadir, Soeren S Lienkamp. Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 424- Springer, 2016.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1XX::18110004805 2018.
Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. Channelnets: Compact and efficient con- volutional neural networks via channel-wiss convolutions. In Advances in Neural Information Processing Systems, pp. 5203-5211, 2018
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp.
2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 770-778, 2016b.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04861, 2017.

---

Under review as a conference paper at ICLR 2020
Jie Hu, Li Shen, and Gang Sun. -aneezze-and-exiiatiioon networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet:
Criss-cross attention for semantic segmentation. arXiv preprint 1rXiv:1811.11110 2018.
Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Review, (1))::
455-500, 2009.
Guanbin Li, Xiang He, Wei Zhang, Huiyou Chang, Le Dong, and Liang Lin. Non-locally enhanced encoder-decoder network for single image de- de-raining. In 2018 ACM Multimedia Conference on Multimedia Conference. pp. 1056-1064. ACM, 2018
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pp. 289-297, 2016.
Minh-Thang Luong, Hieu Pham, and Christopher D) Manning. Effective approaches to attention- neural machine translation. Conference on Empirical Methods in Natural Language Processing, 2015.
Mateusz Malinowski, Carl Doersch, Adam Santoro, and Peter Battaglia. Learning visual question answering by bootstrapping hard attention. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 234-241. Springer, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, pp. 1-42, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510-4520. IEEE, 2018
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, (5():1992-11988 2014.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. International conference on machine learning, pp. 1139-1147, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. arXiv preprint 17Xiv:1710. 1710..000,... 2017.
Xiaolong Wang. Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pp. 4, 2018.
Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siăn Culley, et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nature methods, 15(12):1090, 2018.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048-2057, 2015.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv: 170700/88,, 2017.
10

---

Under review as a conference paper at ICLR 2020
Appendix
THE SANETS ARCHITECTURE
Table describes the SANets architecture. We use multiple heads with 4 heads in SAOs. The results of heads are concatenated as the output. We only perform linear transformations on value matrices to save computational resources.
Table 9: Details of SANet architecture. Each line represents a sequence of operators and arguments in the format of "input size / operator name / expansion rate output channels / number of operators / stride". We use "Conv2D", "AvgPool", and "FC" denote 2D convolutional layer, global average pooling layer, and fully-connected layer, respectively. We use kernel size of 3 x 3 in depth-wise convolutional layers. The first depth-wise convolutional layer sequence of operators applies stride of s, while the other layers use stride of 1. We use & to denote the class number in the task.
Input
Operator
c
71
234
Conv2D 3
32
2
x12
BaseModule
16
1
1
112*x16
BaseModule
6
24
2
2
x24
BaseModule
32
2
2
28*
AttnModule
32
x32
BaseModule
64
2
14
AttnModule
6
64
3
1
14 x64
AtinModule
6
96
3
14' x96
Masele
160
2
76x160
AutnModule
160
76x160
AttnModule
320
x2x320
Conv2D 1
1280
1280
AvgPool + FC
h
2
EXPERIMENTAL SETUP FOR IMAGE CLASSIFICATION
We use the ImageNet ILSVRC 2012 image classification dataset in our experiments. We follow the data argumentation methods used in He al. (2016b). During training, we scale the image into 256 x 256 and then randomly crop a 224 x 224 patch. Center cropping is used when performing inference. We use batch normalization to replace the scaling by 1/N in SAOs to enable a Iearnable scaling factor. All trainable parameters are initialized by Xavier initialization (Glorot & Bengio 2010). Standard stochastic gradient descent optimizer with momentum of 0.9 (Sutskever al. 2013) is used to train the model for total of 150 epochs. The learning rate is initially 0. and decays by 0.1 at the 80th, th5th, and 1 epoch. We apply dropout (Srivastava et al. 2014) with a keep rate of 0.8 after the global average pooling layer. We use 8 GeForce RTX 2080 Ti GPUs with batch size 640 for training. Performance results on the validation dataset is reported since testing dataset labels not available.
EXPERIMENTAL SETUP FOR IMAGE RESTORATION
The networks for the denoising models are the 3D U-Net with depth 3, which consists of two encoder blocks and two decoder blocks. The projection models consist of decoder-encoder architecture that reduces 3D images into 2D tensors and a 2D U-Net. The number of training batches is 16 and the learning rates are 0.0004 for all the three tasks. The denoising models are trained for 200 epochs while the projection model is trained for 100 epochs. The summary of the datasets used in this task is provided in Table 10
11

---

Under review as a conference paper at ICLR 2020
Table 10: Summary of datasets used in our image restoration experiments. Planaria and Tribolium are used for image denoising experiments and Flywing is used in image projection experiments.
Dataset
Task
Training
Patch Size
# Testing
Testing Size
Planaria
Denoising
17,005
16x64x64x1
20
95x1024x 1024
Tribolium
Denoising
14.725
16x64x64x
50x800 800
Flywing
Projection
16,891
50x64x64x1
26
50x520x692
REFERENCES
All citations refer to the references in the main paper.
12


Reviews:
[{"review_id": "SkxmBKIRtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a novel self-attention operator for neural networks. Their self-attention operator computes similarity between elements a and b as (a+b)^Tw where w is a learned parameter and does not use the softmax operator. This leads to improvements in space and time complexity compared to regular self-attention which uses the dot product (a^Tb).\nThey show that concatenating their operation with convolution brings improvements over the MobileNetv2 baseline on ImageNet classification and over U-Net on restoration tasks.\n\nAttention has been empirically shown to bring improvements in many visual tasks but certain methods (such as self-attention) can be quite expensive in computations and memory. Identifying cheaper attention mechanisms that obtain similar accuracy performance as expensive attention mechanisms is therefore an important direction for work.\n\nHowever, I take several crucial issues with this work and especially the evaluation/presentation of the methods:\n- Although this is the focus of the work, the authors do not report actual memory consumption and latency times for MobileNetv2, SANet and the other attention mechanisms. (There is no need for the simulated scenarios of Table 2 since we already know the theoretical complexities of the different methods).\n- The authors only compare their methods against regular self-attention (without or with pooling/softmax), and ignore a longstanding literature of other (potentially cheaper in terms of memory and computations) attention mechanisms in vision (see below). Without comparison to at least Squeeze-and-Excite, it is hard to evaluate the significance of the method presented in the draft.\n- The motivation for naming the method \"siamese\" is quite poor. Siamese networks typically are more complex than a single layer feed forward (which is just a dot-product). Furthermore, the siamese similarity (as introduced by the authors) does not respect the usual properties of similarity functions. For example siasim(a, 0) = a^tw = 1/2 siasim(a,a) can take arbitrary values including negative values (\"a can be dissimilar with itself\")\n- X vs Q, K, V? Self-attention is incorrectly described as \"a special case of the attention operator with Q = K = V\", instead of Q = XW^Q, K=XW^K, V = XW^V.\n- In Table 7, shouldn't SANet w/o params have less params than SANet?\n\nIn summary, the paper addresses an important challenge and proposes a technically sound method.  However, the current  draft has fundamental experimental flaws in its evaluation/presentation and lacks comparison against relevant cheap channelwise attention mechanisms (such as Squeeze-and-Excitation). I argue for rejection.\n\nRelevant literature:\n- channelwise attention: Squeeze-and-Excitation, Gather-Excite\n- Channelwise and spatial attention: Bottleneck Attention Module, Convolutional Block Attention Module\n- Relative Self-attention for vision: Attention Augmented Convolutional Networks, An Empirical Study of Spatial Attention Mechanisms in Deep Networks.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HylsrDYTtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY: A new similarity function replacing the dot product of key and query in attention modules - instead take a shared-weighted sum\n\nGood paper, sound theory, very clear explanations. Literature review was sufficient to explain the problem and underlying theory.\n\nResults look convincing, although they cannot be verified unless code is shared.\n\nReasonable direction of exploration - there are several possible similarity functions, this paper explores one of them that offers significantly less computational resources, which are essential for on-device applications. Thorough exploration of this idea was done and I am convinced this is a good alternative to regular attention.\n\nAll the rest of the paper was to incorporate this new method in different tasks in different architectures using or not using attention, and seeing the differences in terms of computational resources. Well thought experiments and results. Again, cannot be verified unless code is shared. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lCtyTlqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new mechanism to perform the attention operators. The similarity between a key and a query is performed as the dot product between a trainable weight and the addition of the key and query.  The proposed Siamese attention operator is much more efficient than prior attention methods in terms of speed. The evaluation on a few computer vision tasks shows the presented method performs as well as the typical attention methods, but it runs much faster. \n\nFirst of all, I think this attention method should be quite useful for various neural networks. It is faster and performs equally well as other attention operators.  However, I do have some concerns about this method.\n\n- It is not clear to me why this method works. (a+b)^T*w is a strange expression to compute the similarity between a and b. No much explanation or intuition is given in the paper, and I have no clue why this works.\n\n- It seems to me that the proposed method performs slightly worse than the regular attention, according to Figure 6. \n\nMy overall rating is borderline. It would be great if the authors can resolve my concerns.\n\nOther questions:\n- Why SANet (MobileNetv2+SAO) has fewer parameters than MobileNetv2 in Table 3?\n- Is attention an operator that significantly slows the speed of the whole network? If not, the speedup of attention is not that important.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents a Siamese attention operator that lowers the computational costs of attention operators for applications such as image recognition. The reviews are split. R1 posted significant concerns with the content of the submission. The concerns remain after the authors' responses and revision. One of the concerns is the apparent dual submission with "Kronecker Attention Networks". The AC agrees with these concerns and recommends rejecting the submission.
Decision: Reject