Title: SIAMESE ATTENTION NETWORKS

Abstract: Attention operators have been widely applied on data of various orders and dimensions such as texts, images, and videos. One challenge of applying attention operators is the excessive usage of computational resources. This is due to the usage of dot product and softmax operator when computing similarity scores. In this work, we propose the Siamese similarity function that uses a feed-forward network to compute similarity scores. This results in the Siamese attention operator (SAO). In particular, SAO leads to a dramatic reduction in the requirement of computational resources. Experimental results show that our SAO can save 94% memory usage and speed up the computation by a factor of 58 compared to the regular attention operator. The computational advantage of SAO is even larger on higher-order and higher-dimensional data. Results on image classification and restoration tasks demonstrate that networks with SAOs are as effective as models with regular attention operator, while significantly outperform those without attention operators.

1. INTRODUCTION:

Deep learning networks with attention operators have demonstrated great capabilities of solving challenging problems in various tasks such as computer vision (Xu et al., 2015; Lu et al., 2016) , natural language processing (Bahdanau et al., 2015; Vaswani et al., 2017) , and network embedding (Veličković et al., 2017) . Attention operators are capable of capturing long-range relationships and brings significant performance boosts (Li et al., 2018; Malinowski et al., 2018) . The application scenarios of attention operators range from 1-D data like texts to high-order and high-dimensional data such as images and videos. However, attention operators suffer from the excessive usage of computational resources when applied on high-order or high-dimensional data. The memory and computational cost increases dramatically with the increase of input orders and dimensions. This prevents attention operators from being applied in broader scenarios. To address this limitation, some studies focus on reducing spatial sizes of inputs such as down-sampling input data (Wang et al., 2018) or attending selected part of data (Huang et al., 2018) . However, such kind of methods inevitably results in information and performance loss.

In this work, we propose a novel and efficient attention operator known as Siamese attention operator (SAO) to dramatically reduce the usage of computational resources. We observe that the excessive computational resource usage is mainly caused by the similarity function and coefficients normalization function used in attention operators. To address this limitation, we propose the Siamese similarity function that employs a feed-forward network to compute similarity scores. By applying the same network to both input vectors, Siamese similarity function processes the symmetry property. By using Siamese similarity function to compute similarity scores, we propose the Siamese attention operator, which results in a significant saving on computational resources. Based on the Siamese attention operator, we design a family of efficient modules, which leads to our compact deep models known as Siamese attention networks (SANets). Our SANets significantly outperform other state-of-the-art compact models on image classification tasks. Experiments on image restoration tasks demonstrate that our methods are efficient and effective in general application scenarios.

2. BACKGROUND AND RELATED WORK:

In this section, we describe the attention operator that has been widely applied in various tasks and on various types of data including texts, images, and videos.

2.1. ATTENTION OPERATOR:

The inputs of an attention operator include three matrices; those are a query matrix Q =

[q 1 , q 2 , • • • , q m ] ∈ R d×m with each q i ∈ R d , a key matrix K = [k 1 , k 2 , • • • , k n ] ∈ R d×n with each k i ∈ R d , and a value matrix V = [v 1 , v 2 , • • • , v n ] ∈ R p×n with each v i ∈ R p .

To compute the response of each query vector q i , the attention operator calculates similarity scores between q i and each key vector k j using a similarity function. Frequently used similarity functions include dot product (Luong et al., 2015) , concatenation (Bahdanau et al., 2015) , Gaussian function, and embedded Gaussian function. It has been shown that dot product is the most effective one (Wang et al., 2018) , which computes sim(q i , k j ) = k T j q i . After the normalization with a softmax operator, the response is computed by taking a weighted sum over value vectors N j=1 v j softmax(k T j q i ). For all query vectors, the attention operator computes

EQUATION

where softmax(•) is the column-wise softmax operator. The matrix multiplication between K T and Q computes a intermediate output matrix E that stores similarity scores between each query vector q i and each key vector k j . The column-wise softmax operator normalizes E and makes every column sum to 1. Multiplication between V and the normalized E gives the output O ∈ R p×m . Self-attention operator (Vaswani et al., 2017; Devlin et al., 2018) is a special case of the attention operator with Q = K = V . In practice, we usually firstly perform linear transformations on input matrices. For notation simplicity, we use original input matrices in following discussions. The computational cost of the operations in Eq. ( 1 The matrix multiplication order in Eq. ( 1) is determined by the softmax operator, which acts as the normalization function. Wang et al. (2018) proposed to use scaling by 1/N as the normalization function on similarity scores. By this, the response of q i is calculated 1 N N j=1 v j k T j q i . The attention operator using scaling by 1/N computes all responses as:

EQUATION

)

By computing V K T first, the time and space complexities of Eq. ( 2) are O(N d 2 ) and O(d 2 ), respectively. When N > d, this saves computational resources compared to the attention operator in Eq. (1). In practice, we usually have N > d in some parts of a neural network, especially on high-order data.

2.2. ATTENTION OPERATORS ON HIGH-ORDER DATA:

Non-local operators (Wang et al., 2018) are essentially self-attention operators on high-order data like images and videos. Take 2-D data as an example, the input to a non-local operator is an image, which can be represented as a third-order tensor X ∈ R h×w×c . Here, h, w, and c denote the height, width, and number of channels, respectively. The non-local operator converts the tensor into a matrix X (3) ∈ R c×hw by unfolding along mode-3 (Kolda & Bader, 2009) . Then the matrix is fed into an attention operator by setting

Q = K = V = X (3)

. The output of the attention operator is converted back to a third-order tensor that is used as the final output. A challenging problem of non-local operators is the excessive usage of computational resources. If h = w, the time and space complexity of the non-local operator is O(h 4 × d) and O(h 4 ), respectively. The computational cost becomes even bigger on higher-order data like videos. The excessive usage of computational resources limits the application of attention operators in broader scenarios.

3. SIAMESE ATTENTION NETWORKS:

In this work, we propose a learnable similarity function known as the Siamese similarity function. This function uses a single-layer feed-forward network to compute similarity scores. Based on Siamese similarity function, we propose the Siamese attention operator, which dramatically reduces computational cost. We also describe how to build Siamese attention networks using this operator. 

3.1. SIAMESE SIMILARITY FUNCTION:

We analyze the problem of learning a similarity function given two vectors. To learn a similarity function, we employ a single-layer feed-forward neural network. Given two vectors a ∈ R d and b ∈ R d , the similarity score is computed using a trainable vector w ∈ R 2d as:

EQUATION

where

w = [w T a , w T b ] T with w a ∈ R d and w b ∈ R d .

Here, we ignore the bias term for notation simplicity. We consider the similarity function defined in Eq. ( 3) as two feed-forward networks that process two vectors separately. The similarity score is the sum of outputs from two different networks.

Unlike distance metrics, the non-negativity or triangle inequality do not need to hold from similarity functions. But we usually expect similarity measures to be symmetric, which means it outputs the same similarity score when two input arguments are swapped. Apparently, the similarity function defined by Eq. ( 3) does not have this property. To retain the symmetry property, we employ the same network while using both vectors to compute the similarity score. This leads to our proposed Siamese similarity function (Sia-sim), which follows the principle of Siamese networks (Bromley et al., 1994; Bertinetto et al., 2016) . The Siamese similarity function computes the similarity score between a and b as:

EQUATION

where w ∈ R d is a trainable parameter vector. Although the time complexity of computing Sia-sim is the same as that of dot product, we show that Sia-sim leads to a very efficient attention operator in Section 3.2. Figure 1 provides an illustration of the similarity functions defined in Eq. (3) and Eq. (4).

3.2. SIAMESE ATTENTION OPERATOR:

We describe the Siamese attention operator in the context of 1-D data, but it can be easily applied on high-order data by unfolding them into matrices. In this case, the inputs to an attention operator are Q ∈ R d×N , K ∈ R d×N , and V ∈ R d×N . We replace the similarity function in the attention operator by our Siamese similarity function, leading to the Siamese attention operator (SAO). Given a query vector q i in Q, SAO computes the response o i as:

EQUATION

where

v = 1 N N j=1 V :j ∈ R d .

SAO computes responses of all query vectors as: 

EQUATION

V softmax(K T Q) O(N 2 × d) O(N 2 ) Attn 1/N 1 N (V K T )Q O(N × d 2 ) O(d 2 ) SAO vw T Q + 1 N V K T w1 T N O(N × d) O(N × d)

where 1 N is a vector of ones of size N .

Note that we use 1 N here to make it mathematically precise. In practice, the term 1 N V K T w is the same to all query vectors. This means we only need to compute it once and share it for the computation of all responses. By computing K T w first, the time complexity of this term is O(N ×d). Similarly, the time complexity for computing the first term in Eq. ( 6) is O(N × d). Thus, the overall time complexity of SAO is O(N × d). Notably, when Q = K, we can save the computational cost by computing either w T Q or K T w. Table 1 provides the comparison of SAO and other attention operators. It can be seen from the comparison results that our SAO can significantly save computational resources compared to other attention operators.

In Eq. ( 5), the first response term vw T q i changes as the query vector q i , which we call a local response term. The second term 1 N V K T w is the same for all query vectors, which is a global response term. The local response term provides customized information to query vectors, while the global response term may include global information for SAO. In the experimental study part, we demonstrate the importance of the global response term to SAO.

3.3. SIAMESE ATTENTION MODULES:

Attention models have been shown to be effective in various fields. However, they are not widely applied in compact neural networks to date primarily due to their excessive computational cost. Based on our efficient SAO, it is possible to integrate attention operators in compact convolutional neural networks (CNNs) such as MobileNetV2 (Sandler et al., 2018) . In this section, we design a family of efficient Siamese attention modules (SAMs) that integrate our SAO with convolutional operators. These modules can be used for designing compact CNNs.

BaseModule: Bottleneck blocks with inverted residuals are main components in MobileNetV2. Each bottleneck consists of three layers; those are an 1×1 convolutional layer for expansion, a 3×3 depth-wise convolutional layer, and another 1×1 convolutional layer. The hyper-parameters of this module include the expansion factor r and the stride s. Given the input tensor X ( ) ∈ R h×w×c for the th block, the first 1×1 convolutional layer expands it into (r -1)c feature maps. We then concatenate the output with the input, resulting in X( ) ∈ R h×w×rc . The depth-wise convolutional layer with stride s outputs rc feature maps X( ) ∈ R In SAOs, we only apply linear transformation on the value matrix V to limit the computational cost and the number of trainable parameters. The original path extracts locality-based features, while the new path using SAO computes global features. In this way, AttnModule is capable of capturing both local and global information. We add a skip connection between inputs and outputs of SAO to enable better feature reuse and gradient back-propagation (He et al., 2016a) . 

In this section, we describe the model architecture of a compact convolutional neural network using our proposed SAOs and modules in Section 3.3, leading to Siamese attention networks (SANets).

We follow the basic architecture of MobileNetV2 (Sandler et al., 2018) and replace inverted residual blocks with our SAMs. The details of network architecture are described in Table 9 in the appendix. Following the practices in Wang et al. (2018) , we use AttnModules to replace the blocks with input spatial sizes of 28 × 28, 14 × 14, and 7 × 7. For the rest of the blocks, we use BaseModule with the same expansion factors. The SANet built on SAMs saves about 0.03 million parameters and 12 million MAdd compared to MobileNetV2. We show that SANets with SAOs significantly outperform MobileNetV2 while using less computational resources.

4. EXPERIMENTAL STUDIES:

In this section, we evaluate our methods using image classification and restoration tasks. We firstly evaluate our SAO in terms of computational resource usage by comparing with regular attention operators. We compare our designed SANets with compact CNNs on image classification tasks using the ImageNet ILSVRC 2012 dataset (Russakovsky et al., 2015) . We conduct ablation studies to investigate the benefits of our SAO and the trainable parameters in it. We also evaluate our methods in general application scenarios using image restoration tasks.

4.1. COMPARISON OF COMPUTATIONAL EFFICIENCY:

The theoretical analysis in Section 3.2 shows that our SAO can achieve significant efficiency advantages over regular attention operator especially on high-order and high-dimensional data. In this section, we use simulated experiments to evaluate the efficiency advantage of SAO over other attention operators. We build networks using different attention operators. The network is composed of a single attention operator to eliminate the influence of other factors. We apply the TensorFlow profile tool (Abadi et al., 2016) and report the required memory and CPU time on 2-D simulated data.

To fully evaluate the efficiency of attention operators, we vary the number of channels and spatial sizes of the input data.

The comparison results is summarized in of SAO over regular attention operators increases as the increase of spatial and dimension sizes. When comparing with other attention operators, our SAO is shown to be the most computationally efficient attention operator. The simulated results show that our SAO is an efficient attention operator that can operate high-dimensional and high-order data by consuming very few computational resources. Based on our efficient SAO, we build a family of compact CNNs in Section 3.4 for image classification tasks. To evaluate the effectiveness of our SAO and SANets, we compare our models with other compact CNNs (Howard et al., 2017; Sandler et al., 2018; Zhang et al., 2017; Gao et al., 2018) We provide the experimental setups on image classification tasks in the appendix.

4.2. RESULTS ON IMAGE CLASSIFICATION:

We compare our SANets with other compact CNNs and report the top-1 accuracy, the number of parameters, and MAdd in Table 3 . Mo-bileNetV2 (Sandler et al., 2018) is the previous state-of-the-art model in terms of the computational cost and model performance. Compared to MobileNetV2, our SANet significantly outperforms MobileNetV2 by a margin of 1% with a smaller number of parameters. By using our SAO, SANet can achieve new state-of-the-art performance with limited computational resources. Considering that we only make minor modifications from the architecture of MobileNetV2, the performance boost is significant. Compared to the module using regular convolutional layers, our proposed module uses SAO to obtain global features and avoids the excessive usage of computational resources.

Our SAO successfully overcomes the limitations of regular attention operator and applies attention mechanism on high-order and high-dimensional data with significant performance boost. Next, we will show our SAO is as effective as regular attention operator but can dramatically reduce the usage of computational resources. We have shown that our SAO has significant efficiency advantages over regular attention operators in Section 4.1. In this section, we conduct experiments to compare our SAO with Squeezeand-Excite block (Hu et al., 2018 ) and regular attention operators based on SANets using image classification tasks. Besides the regular attention operator, we also consider the one defined in Eq. ( 2) using scaling by 1/N for coefficients normalization, and the one with a pooling operation as in Wang et al. (2018) . In the attention operator with pooling operations, 2 × 2 pooling operations are applied to key matrix K and value matrix V , leading to reduced spatial sizes.

4.3. COMPARISON WITH REGULAR ATTENTION OPERATORS:

To ensure fair comparisons, we replace all SAOs in SANets with attention operators, attention operators with pooling operations, and attention operators defined in Eq. ( 2), denoted as AttnNet, AttnNet+Pool, and AttnNet 1/N , respectively. We also compare the actual computational usage among MobileNetV2, SANet and AttnNet. We summarize the comparison results in Table 4 and Table 5 . We can observe from the results that our SANet using SAOs achieves the same performance as AttnNet with regular attention operators but using significantly less computational resources. Compared to MobileNetV2, our SANet uses slightly extra resources but achieves significantly better performances. The results demonstrate that our SAO is as effective as the regular attention operator while dramatically reduces computational costs. The better performance of SANet over AttnNet+pool indicates that our SAO is more efficient and effective compared to the regular attention operator using pooling operations. Note that AttnNet 1/N and SANet achieve the same performance as AttnNet, which demonstrates the feasibility of replacing the expensive softmax(•) function with scaling by 1/N . Notably, all networks with attention operators significantly outperform SENet which squeezes the spatial information. This demonstrates the effectiveness of attention operators that effectively build long-range relationships and lead to better performances. We also explore the impact of the global item in SAO on the performance of SANets.

4.4. ABLATION STUDIES:

The comparison results are summarized in Table 6. By using SAOs, MobileNetV2 w SAO obtains a performance boost of 0.7% over that of the original MobileNetV2. The performance of 

4.5. RESULTS ON 3-D IMAGE RESTORATION:

In order to evaluate the effectiveness of our SAO in broader application scenarios, we conduct experiments on biological image restoration tasks, in particular, the 3D image denoising and the 3D image projection. The projection models map a noisy 3D image to a 2D plane, i.e., mapping 3D images from R h×w×c to 2D images in R h×w . We perform experiments on three different datasets collected by Weigert et al. (2018) ; those are Planaria, Tribolium and Flywing. Details of dataset are shown in Table 10 in the appendix. We adopt a general U-Net architecture (Ronneberger et al., 2015; C ¸ic ¸ek et al., 2016) as our baseline model. We add attention operators and our SAOs in bottom and decoder blocks. We apply linear transformations on Q, K and V in attention operators. The details of experimental setups are provided in the appendix. The comparison results are summarized in Table 7 . Clearly, U-Net models with attention operators outperform the baseline model, which demonstrates the effectiveness of attention mechanism. Compared to U-Net with regular attention operators, SAOs result in similar results. This indicates our SAO is as effective as regular attention operators. Overall, these experimental results demonstrate that our SAO retains its effectiveness in broader application scenarios. Since SAO involves extra trainable parameters in our Siamese similarity function, we study the impact of these trainable parameters in SANets.

4.6. PARAMETER STUDY OF SAO:

We use an all-ones vector to replace the trainable parameters w in Eq. ( 6). We denote the resulting model as SANet w/o params. Table 8 reports the comparison results. We can observe that SANet outperforms SANet w/o params by a margin of 0.2% with regard to the top-1 accuracy using only 960 more parameters. This demonstrates that the trainable parameters in SAOs are necessary since the importance of different features are weighted.

5. CONCLUSIONS:

In this work, we propose Siamese attention operators to overcome the excessive usage of computational resources of regular attention operators when being applied on high-order and/or highdimensional data. We observe that the similarity score in the attention operator is computed using the dot product, which leads to high computational cost. To address this, we propose Siamese similarity function, which employs a feed-forward network to compute the similarity score between two input vectors. By using the shared network on two vectors, Siamese similarity function is symmetric.

We use Siamese similarity function to replace dot product in regular attention operator, leading to Siamese attention operator. Theoretical analysis and experimental studies show that our SAOs significantly reduce the usage of computational resources. Based on SAOs, we design a family of efficient modules, leading to SANets. The evaluation on image classification tasks shows that our SANet significantly outperforms previous state-of-the-art models while using fewer trainable parameters and computational resources. In addition, we conduct experiments on 3-D image restoration to demonstrate the effectiveness of our SAOs in broader application scenarios. The parameter study shows that SAO brings great performance improvement with negligible extra trainable parameters. 

REFERENCES:

All citations refer to the references in the main paper.

:



Figures and Tables:

) is O(m × n × (d + p)), and the memory required to store the intermediate output E is O(m × n). If m = n and d = p, the time and space complexities of Eq. (1) are O(n 2 × d) and O(n 2 ), respectively.

Figure 1: Illustration of the similarity function using a single-layer feed-forward network (a) and the Siamese similarity function (b). In (a), different input elements use different weights. In (b), two input vectors share the same network. The lines with the same color indicate weights sharing.

×rc . Finally, the last 1×1 convolutional layer outputs d feature maps Y ( ) ∈ R h s × w s ×d . A skip connection is applied between X ( ) and Y ( ) when c = d and s = 1. The BaseModule is illustrated in Figure 2 (a).

We propose to integrate our SAO into the BaseModule, resulting in AttnModule as shown in Figure2 (b). Before the last 1×1 convolutional layer, we add a new parallel path by SAO, which outputs c feature maps. An average pooling layer with stride s is followed when s > 1, ×c . In the original path, both the first 1×1 convolutional layer and the depth-wise convolutional layer output (r -1)c feature mapsX( ) b ∈ R h s × ws ×(r-1)c . We concatenate and feed them into the last 1×1 convolutional layer, which outputs d feature maps Y ( ) ∈ R h s × w s ×d .

Figure 2: Architectures of the BaseModule (a) and AttnModule (b) as introduced in Section 3.3. The skip connection between inputs and outputs is used when s = 1 and c = d. The one between inputs and outputs of SAO is used when s = 1.

Comparisons among three attention operators in terms of time and space complexities. Attn and Attn 1/N denote the regular attention operators using softmax and scaling by 1/N for similarity scores normalization, respectively.

Table content: <table><tr><td>Operator</td><td>Computation</td><td>Time Complexity</td><td>Space Complexity</td></tr><tr><td>Attn</td><td/><td/><td/></tr></table>



Table content: <table/>

Comparisons between the regular attention operator, the regular attention operator with a pooling operation(Wang et al., 2018), and our proposed SAO in terms of the number of parameters, number of MAdd, memory usage, and CPU inference time on simulated data of different sizes. The input sizes are given in the format of "spatial sizes 2 × number of input channels". "Attn" denotes the regular attention operator. "Attn+Pool" denotes the regular attention operator which employs 2 × 2 pooling operations on K and V input matrices to reduce required computational resources.

Table content: <table><tr><td>Input</td><td>Operator</td><td>MAdd</td><td>Memory</td><td>Saving</td><td>Time</td><td>Speedup</td></tr><tr><td/><td>Attn</td><td>4.92m</td><td>20.97MB</td><td>0.00%</td><td>20.47ms</td><td>1.00×</td></tr><tr><td>14 2 × 64</td><td>Attn+Pool Attn 1/N</td><td>1.23m 1.61m</td><td>8.39MB 5.24MB</td><td>59.99% 75.01%</td><td>5.07ms 6.97ms</td><td>4.04× 2.94×</td></tr><tr><td/><td>SAO</td><td>0.05m</td><td>4.29MB</td><td>79.54%</td><td>3.09ms</td><td>6.62×</td></tr><tr><td/><td>Attn</td><td>157.35m</td><td>301.99MB</td><td>0.00%</td><td>450.22ms</td><td>1.00×</td></tr><tr><td>28 2 × 128</td><td>Attn+Pool Attn 1/N</td><td>39.34m 25.69m</td><td>100.66MB 37.75MB</td><td>66.67% 87.50%</td><td>119.55ms 70.52ms</td><td>3.77× 6.38×</td></tr><tr><td/><td>SAO</td><td>0.40m</td><td>33.88MB</td><td>88.78%</td><td>31.76ms</td><td>14.18×</td></tr><tr><td/><td>Attn</td><td>5,035.26m</td><td>5.04GB</td><td>0.00%</td><td>12.92s</td><td>1.00×</td></tr><tr><td>56 2 × 256</td><td>Attn+Pool Attn 1/N</td><td>1,258.82m 411.04m</td><td>1.34GB 0.28GB</td><td>73.38% 94.34%</td><td>4.82s 1.02s</td><td>2.68× 12.66×</td></tr><tr><td/><td>SAO</td><td>3.21m</td><td>0.26GB</td><td>94.65%</td><td>0.22s</td><td>58.21×</td></tr></table>



Table content: <table><tr><td colspan="4">: Comparisons between SANet and other</td></tr><tr><td colspan="4">CNNs in terms of the top-1 accuracy, the number of</td></tr><tr><td colspan="4">trainable parameters, and MAdd on the ImageNet</td></tr><tr><td>validation set.</td><td/><td/><td/></tr><tr><td>Model</td><td colspan="3">Top-1 Params MAdd</td></tr><tr><td>GoogleNet</td><td>0.698</td><td>6.8m</td><td>1550m</td></tr><tr><td>VGG16</td><td>0.715</td><td>128m</td><td>15300m</td></tr><tr><td>SqueezeNet</td><td>0.575</td><td>1.3m</td><td>833m</td></tr><tr><td>MobileNetV1</td><td>0.706</td><td>4.2m</td><td>569m</td></tr><tr><td colspan="2">ChannelNet-v1 0.705</td><td>3.7m</td><td>407m</td></tr><tr><td colspan="2">ShuffleNet 1.5x 0.715</td><td>3.4m</td><td>292m</td></tr><tr><td>MobileNetV2</td><td>0.720</td><td>3.47m</td><td>300m</td></tr><tr><td>SANet (ours)</td><td>0.730</td><td>3.44m</td><td>288m</td></tr></table>

Comparisons between SANet, the network using the same architecture as SANet with

Table content: <table><tr><td colspan="4">Squeeze-and-Excite block (denoted as SENet) or</td></tr><tr><td colspan="4">with regular attention operators (denoted as At-</td></tr><tr><td colspan="4">tnNet), the AttnNet using regular attention opera-</td></tr><tr><td colspan="4">tors with a pooling operation (AttnNet+Pool), and</td></tr><tr><td colspan="4">the AttnNet with regular attention operators using</td></tr><tr><td colspan="4">scaling by 1/N (denoted as AttnNet 1/N ) in terms of</td></tr><tr><td colspan="4">the top-1 accuracy, the number of total parameters,</td></tr><tr><td colspan="4">and MAdd on the ImageNet validation set.</td></tr><tr><td>Model</td><td>Top-1</td><td colspan="2">Params MAdd</td></tr><tr><td>SENet</td><td>0.724</td><td>3.44m</td><td>294m</td></tr><tr><td>AttnNet</td><td>0.730</td><td>3.44m</td><td>365m</td></tr><tr><td>AttnNet+Pool</td><td>0.729</td><td>3.44m</td><td>300m</td></tr><tr><td>AttnNet 1/N</td><td>0.730</td><td>3.44m</td><td>312m</td></tr><tr><td>SANet</td><td>0.730</td><td>3.44m</td><td>288m</td></tr></table>

Comparisons among MobileNetV2, SANet, and AttnNet in terms of top-1 accuracy, memory usage, and execution time. We report the actual memory usage using inputs of 128 × 224 2 × 3. The time reported here is the model execution time of 100 iterations.

Table content: <table><tr><td>Model</td><td>Top-1</td><td colspan="2">Memory Time</td></tr><tr><td>MobileNetV2</td><td>0.720</td><td>7.4GB</td><td>14.4s</td></tr><tr><td>SANet</td><td>0.730</td><td>7.5GB</td><td>14.9s</td></tr><tr><td>AttnNet</td><td>0.730</td><td>11.7GB</td><td>27.1s</td></tr></table>



Table content: <table><tr><td>To fully demonstrate the effectiveness of our SAO, we conduct ablation studies based on Mo-bileNetV2 and SANets. We add SAOs to Mo-bileNetV2 to observe if it benefits other net-works like MobileNetV2. We replace some in-verted residual blocks with our AttnModules by referencing the structure of SANet, which we denote as MobileNetV2 w SAO. We remove SAOs from SANet by replacing AttnModule with BaseModule, resulting in SANet w/o SAO.</td><td colspan="3">: Comparisons between MobileNetV2, MobileNetV2 with SAOs (MobileNetV2+SAO), SANet, SANet without SAO (SANet w/o SAO), and SANet without the global item in Eq. (5) in terms of the top-1 accuracy, and the number of total parameters on the ImageNet validation set. Model Top-1 Params SANet 0.730 3.44m SANet w/o SAO 0.721 3.44m SANet w/o global term 0.728 3.44m</td></tr><tr><td/><td>MobileNetV2+SAO</td><td>0.727</td><td>3.47m</td></tr><tr><td/><td>MobileNetV2</td><td>0.720</td><td>3.47m</td></tr></table>

Comparison results between U-Net, U-Net with regular attention operator (U-Net+Attn), and U-Net with SAO (U-Net+SAO) in terms of peak signal to noise ratio (PSNR), structural similarity index (SSIM), and normalized root mean square error (NRMSE). On the metrics followed by an ↑, the higher value indicates the better performance. Conversely, lower value indicates the better performance on the metrics followed by a ↓. SAO without using SAOs is 0.9% lower than that of SANet. Notably, SANet consumes the least computational costs while achieving the best performance. The results demonstrate that our SAO is consistently efficient and effective when being applied in different network architectures. Notably, it can be observed that the global item is important by comparing the performances between SANet and SANet w/o global term.

Table content: <table><tr><td>Dataset</td><td>Metric</td><td>U-Net</td><td>U-Net+Attn</td><td>U-Net+SAO</td></tr><tr><td/><td>PSNR↑</td><td>31.571</td><td>31.951</td><td>31.929</td></tr><tr><td>Planaria</td><td>SSIM↑</td><td>0.7707</td><td>0.7919</td><td>0.7898</td></tr><tr><td/><td>NRMSE↓</td><td>0.0268</td><td>0.0257</td><td>0.0257</td></tr><tr><td/><td>PSNR↑</td><td>32.433</td><td>32.571</td><td>32.549</td></tr><tr><td>Tribolium</td><td>SSIM↑</td><td>0.9171</td><td>0.9201</td><td>0.9196</td></tr><tr><td/><td>NRMSE↓</td><td>0.0241</td><td>0.0236</td><td>0.0239</td></tr><tr><td/><td>PSNR↑</td><td>21.958</td><td>22.361</td><td>22.329</td></tr><tr><td>Flywing</td><td>SSIM↑</td><td>0.5592</td><td>0.5903</td><td>0.5916</td></tr><tr><td/><td>NRMSE↓</td><td>0.0798</td><td>0.0763</td><td>0.0767</td></tr><tr><td>SANet w/o</td><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td colspan="3">: Comparisons between SANet and the</td></tr><tr><td colspan="3">SANet removing trainable parameters in SAOs</td></tr><tr><td colspan="3">(SANet w/o params) in terms of the top-1 accuracy</td></tr><tr><td colspan="2">and number of parameters.</td><td/></tr><tr><td>Model</td><td>Top-1</td><td>Params</td></tr><tr><td>SANet</td><td>0.730</td><td>3,449,408</td></tr><tr><td>SANet w/o params</td><td>0.728</td><td>3,448,448</td></tr></table>

Summary of datasets used in our image restoration experiments. Planaria and Tribolium are used for image denoising experiments and Flywing is used in image projection experiments.

Table content: <table><tr><td>Dataset</td><td>Task</td><td># Training</td><td>Patch Size</td><td># Testing</td><td>Testing Size</td></tr><tr><td>Planaria</td><td>Denoising</td><td>17,005</td><td>16×64×64×1</td><td>20</td><td>95×1024×1024</td></tr><tr><td>Tribolium</td><td>Denoising</td><td>14,725</td><td>16×64×64×1</td><td>6</td><td>50×800×800</td></tr><tr><td>Flywing</td><td>Projection</td><td>16,891</td><td>50×64×64×1</td><td>26</td><td>50×520×692</td></tr></table>

Peer review (iclr2020):
The authors introduce a novel self-attention operator for neural networks. Their self-attention operator computes similarity between elements a and b as (a+b)^Tw where w is a learned parameter and does not use the softmax operator. This leads to improvements in space and time complexity compared to regular self-attention which uses the dot product (a^Tb). They show that concatenating their operation with convolution brings improvements over the MobileNetv2 baseline on ImageNet classification and over U-Net on restoration tasks. Attention has been empirically shown to bring improvements in many visual tasks but certain methods (such as self-attention) can be quite expensive in computations and memory. Identifying cheaper attention mechanisms that obtain similar accuracy performance as expensive attention mechanisms is therefore an important direction for work. However, I take several crucial issues with this work and especially the evaluation/presentation of the methods: - Although this is the focus of the work, the authors do not report actual memory consumption and latency times for MobileNetv2, SANet and the other attention mechanisms. (There is no need for the simulated scenarios of Table 2 since we already know the theoretical complexities of the different methods). - The authors only compare their methods against regular self-attention (without or with pooling/softmax), and ignore a longstanding literature of other (potentially cheaper in terms of memory and computations) attention mechanisms in vision (see below). Without comparison to at least Squeeze-and-Excite, it is hard to evaluate the significance of the method presented in the draft. - The motivation for naming the method *siamese* is quite poor. Siamese networks typically are more complex than a single layer feed forward (which is just a dot-product). Furthermore, the siamese similarity (as introduced by the authors) does not respect the usual properties of similarity functions. For example siasim(a, 0) = a^tw = 1/2 siasim(a,a) can take arbitrary values including negative values (*a can be dissimilar with itself*) - X vs Q, K, V? Self-attention is incorrectly described as *a special case of the attention operator with Q = K = V*, instead of Q = XW^Q, K=XW^K, V = XW^V. - In Table 7, shouldn*t SANet w/o params have less params than SANet? In summary, the paper addresses an important challenge and proposes a technically sound method. However, the current draft has fundamental experimental flaws in its evaluation/presentation and lacks comparison against relevant cheap channelwise attention mechanisms (such as Squeeze-and-Excitation). I argue for rejection. Relevant literature: - channelwise attention: Squeeze-and-Excitation, Gather-Excite - Channelwise and spatial attention: Bottleneck Attention Module, Convolutional Block Attention Module - Relative Self-attention for vision: Attention Augmented Convolutional Networks, An Empirical Study of Spatial Attention Mechanisms in Deep Networks.