Siamese Attention Networks

1 INTRODUCTION . Deep learning networks with attention operators have demonstrated great capabilities of solving challenging problems in various tasks such as computer vision ( Xu et al. , 2015 ; Lu et al. , 2016 ) , natural language processing ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) , and network embedding ( Veličković et al. , 2017 ) . Attention operators are capable of capturing long-range relationships and brings significant performance boosts ( Li et al. , 2018 ; Malinowski et al. , 2018 ) . The application scenarios of attention operators range from 1-D data like texts to high-order and high-dimensional data such as images and videos . However , attention operators suffer from the excessive usage of computational resources when applied on high-order or high-dimensional data . The memory and computational cost increases dramatically with the increase of input orders and dimensions . This prevents attention operators from being applied in broader scenarios . To address this limitation , some studies focus on reducing spatial sizes of inputs such as down-sampling input data ( Wang et al. , 2018 ) or attending selected part of data ( Huang et al. , 2018 ) . However , such kind of methods inevitably results in information and performance loss . In this work , we propose a novel and efficient attention operator known as Siamese attention operator ( SAO ) to dramatically reduce the usage of computational resources . We observe that the excessive computational resource usage is mainly caused by the similarity function and coefficients normalization function used in attention operators . To address this limitation , we propose the Siamese similarity function that employs a feed-forward network to compute similarity scores . By applying the same network to both input vectors , Siamese similarity function processes the symmetry property . By using Siamese similarity function to compute similarity scores , we propose the Siamese attention operator , which results in a significant saving on computational resources . Based on the Siamese attention operator , we design a family of efficient modules , which leads to our compact deep models known as Siamese attention networks ( SANets ) . Our SANets significantly outperform other state-of-the-art compact models on image classification tasks . Experiments on image restoration tasks demonstrate that our methods are efficient and effective in general application scenarios . 2 BACKGROUND AND RELATED WORK . In this section , we describe the attention operator that has been widely applied in various tasks and on various types of data including texts , images , and videos . 2.1 ATTENTION OPERATOR . The inputs of an attention operator include three matrices ; those are a query matrix Q = [ q1 , q2 , · · · , qm ] ∈ Rd×m with each qi ∈ Rd , a key matrix K = [ k1 , k2 , · · · , kn ] ∈ Rd×n with each ki ∈ Rd , and a value matrix V = [ v1 , v2 , · · · , vn ] ∈ Rp×n with each vi ∈ Rp . To compute the response of each query vector qi , the attention operator calculates similarity scores between qi and each key vector kj using a similarity function . Frequently used similarity functions include dot product ( Luong et al. , 2015 ) , concatenation ( Bahdanau et al. , 2015 ) , Gaussian function , and embedded Gaussian function . It has been shown that dot product is the most effective one ( Wang et al. , 2018 ) , which computes sim ( qi , kj ) = kTj qi . After the normalization with a softmax operator , the response is computed by taking a weighted sum over value vectors ∑N j=1 vjsoftmax ( k T j qi ) . For all query vectors , the attention operator computes O = V softmax ( KTQ ) , ( 1 ) where softmax ( · ) is the column-wise softmax operator . The matrix multiplication between KT and Q computes a intermediate output matrix E that stores similarity scores between each query vector qi and each key vector kj . The column-wise softmax operator normalizes E and makes every column sum to 1 . Multiplication between V and the normalized E gives the output O ∈ Rp×m . Self-attention operator ( Vaswani et al. , 2017 ; Devlin et al. , 2018 ) is a special case of the attention operator with Q = K = V . In practice , we usually firstly perform linear transformations on input matrices . For notation simplicity , we use original input matrices in following discussions . The computational cost of the operations in Eq . ( 1 ) is O ( m× n× ( d+ p ) ) , and the memory required to store the intermediate output E is O ( m× n ) . If m = n and d = p , the time and space complexities of Eq . ( 1 ) are O ( n2 × d ) and O ( n2 ) , respectively . The matrix multiplication order in Eq . ( 1 ) is determined by the softmax operator , which acts as the normalization function . Wang et al . ( 2018 ) proposed to use scaling by 1/N as the normalization function on similarity scores . By this , the response of qi is calculated 1N ∑N j=1 vjk T j qi . The attention operator using scaling by 1/N computes all responses as : O = 1 N ( V KT ) Q . ( 2 ) By computing V KT first , the time and space complexities of Eq . ( 2 ) are O ( Nd2 ) and O ( d2 ) , respectively . When N > d , this saves computational resources compared to the attention operator in Eq . ( 1 ) . In practice , we usually have N > d in some parts of a neural network , especially on high-order data . 2.2 ATTENTION OPERATORS ON HIGH-ORDER DATA . Non-local operators ( Wang et al. , 2018 ) are essentially self-attention operators on high-order data like images and videos . Take 2-D data as an example , the input to a non-local operator is an image , which can be represented as a third-order tensor X ∈ Rh×w×c . Here , h , w , and c denote the height , width , and number of channels , respectively . The non-local operator converts the tensor into a matrix X ( 3 ) ∈ Rc×hw by unfolding along mode-3 ( Kolda & Bader , 2009 ) . Then the matrix is fed into an attention operator by setting Q = K = V = X ( 3 ) . The output of the attention operator is converted back to a third-order tensor that is used as the final output . A challenging problem of non-local operators is the excessive usage of computational resources . If h = w , the time and space complexity of the non-local operator is O ( h4 × d ) and O ( h4 ) , respectively . The computational cost becomes even bigger on higher-order data like videos . The excessive usage of computational resources limits the application of attention operators in broader scenarios . 3 SIAMESE ATTENTION NETWORKS . In this work , we propose a learnable similarity function known as the Siamese similarity function . This function uses a single-layer feed-forward network to compute similarity scores . Based on Siamese similarity function , we propose the Siamese attention operator , which dramatically reduces computational cost . We also describe how to build Siamese attention networks using this operator . 3.1 SIAMESE SIMILARITY FUNCTION . We analyze the problem of learning a similarity function given two vectors . To learn a similarity function , we employ a single-layer feed-forward neural network . Given two vectors a ∈ Rd and b ∈ Rd , the similarity score is computed using a trainable vector w ∈ R2d as : simw ( a , b ) = [ aT , bT ] w = d∑ i=1 ai × wi + bi × wd+i = aTwa + bTwb , ( 3 ) where w = [ wTa , w T b ] T with wa ∈ Rd and wb ∈ Rd . Here , we ignore the bias term for notation simplicity . We consider the similarity function defined in Eq . ( 3 ) as two feed-forward networks that process two vectors separately . The similarity score is the sum of outputs from two different networks . Unlike distance metrics , the non-negativity or triangle inequality do not need to hold from similarity functions . But we usually expect similarity measures to be symmetric , which means it outputs the same similarity score when two input arguments are swapped . Apparently , the similarity function defined by Eq . ( 3 ) does not have this property . To retain the symmetry property , we employ the same network while using both vectors to compute the similarity score . This leads to our proposed Siamese similarity function ( Sia-sim ) , which follows the principle of Siamese networks ( Bromley et al. , 1994 ; Bertinetto et al. , 2016 ) . The Siamese similarity function computes the similarity score between a and b as : Sia-simw ( a , b ) = d∑ i=1 ( ai + bi ) × wi = ( a+ b ) Tw , = Sia-simw ( b , a ) ( 4 ) where w ∈ Rd is a trainable parameter vector . Although the time complexity of computing Sia-sim is the same as that of dot product , we show that Sia-sim leads to a very efficient attention operator in Section 3.2 . Figure 1 provides an illustration of the similarity functions defined in Eq . ( 3 ) and Eq . ( 4 ) . 3.2 SIAMESE ATTENTION OPERATOR . We describe the Siamese attention operator in the context of 1-D data , but it can be easily applied on high-order data by unfolding them into matrices . In this case , the inputs to an attention operator are Q ∈ Rd×N , K ∈ Rd×N , and V ∈ Rd×N . We replace the similarity function in the attention operator by our Siamese similarity function , leading to the Siamese attention operator ( SAO ) . Given a query vector qi in Q , SAO computes the response oi as : oi = 1 N N∑ j=1 vj ( qi + kj ) Tw = 1 N N∑ j=1 ( vjq T i w + vjk T j w ) = 1 N N∑ j=1 vj qTi w + 1N N∑ j=1 vjk T j w =vwTqi + 1 N V KTw , ( 5 ) where v = 1N ∑N j=1 V : j ∈ Rd . SAO computes responses of all query vectors as : O = vwTQ+ 1 N V KTw1TN , ( 6 ) where 1N is a vector of ones of size N . Note that we use 1N here to make it mathematically precise . In practice , the term 1NV K Tw is the same to all query vectors . This means we only need to compute it once and share it for the computation of all responses . By computing KTw first , the time complexity of this term is O ( N×d ) . Similarly , the time complexity for computing the first term in Eq . ( 6 ) is O ( N × d ) . Thus , the overall time complexity of SAO is O ( N × d ) . Notably , when Q = K , we can save the computational cost by computing either wTQ or KTw . Table 1 provides the comparison of SAO and other attention operators . It can be seen from the comparison results that our SAO can significantly save computational resources compared to other attention operators . In Eq . ( 5 ) , the first response term vwTqi changes as the query vector qi , which we call a local response term . The second term 1NV K Tw is the same for all query vectors , which is a global response term . The local response term provides customized information to query vectors , while the global response term may include global information for SAO . In the experimental study part , we demonstrate the importance of the global response term to SAO .

Summary: The authors introduce a novel self-attention operator for neural networks. Their self-attention operator computes similarity between elements a and b as (a+b)^Tw where w is a learned parameter and does not use the softmax operator. This leads to improvements in space and time complexity compared to regular self-attention which uses the dot product (a^Tb).