Attention: Self-Expression Is All You Need

1 INTRODUCTION . Attention , i.e. , the ability to selectively focus on a subset of sensory observations , while ignoring other irrelevant information , is a central component of human perception . For example , only a few words in a sentence may be useful for predicting the next word , or only a small portion of an image may be relevant for recognizing an object . This property of biological systems has inspired the recent development of attention-based neural architectures ( Bahdanau et al. , 2014 ) , such as Transformers ( Vaswani et al. , 2017 ) , BERT ( Devlin et al. , 2018 ) , GPT ( Radford et al. , 2018 ; 2019 ) , RoBERTa ( Liu et al. , 2019 ) , and T5 ( Raffel et al. , 2019 ) , which have achieved impressive performance in multiple natural language processing tasks , including text classification ( Chaudhari et al. , 2019 ; Galassi et al. , 2020 ) , machine translation ( Ott et al. , 2018 ) , and question answering ( Garg et al. , 2020 ) . Attentionbased architectures have also led to state-of-the-art results in various computer vision tasks ( Khan et al. , 2021 ) , including image classification ( Dosovitskiy et al. , 2020 ) , object detection ( Carion et al. , 2020 ; Zhu et al. , 2020 ) , and visual question answering ( Tan & Bansal , 2019 ; Su et al. , 2019 ) . Much of the success behind attention-based architectures is attributed to their ability to capture longrange interactions among data tokens ( such as words and image patches ) via attention coefficients that are global , learnable and adapted to the input at test time . For example , while recurrent neural network architectures in natural language processing predict the next word in a sentence using information about a few previous words , self-attention mechanisms make a prediction based on interactions among all words . Similarly , while convolutional architectures in computer vision compute local interactions among image patches using weights that do not depend on the input image at test time , vision transformers compute global interactions that are adapted to the input at test time . In this paper , we show that many of the key ideas behind attention , which we briefly summarize in Section 2 , build upon a long history of prior work on manifold learning and image processing . In Section 3 we show that the scaled dot product attention mechanism is equivalent to kernelbased regression with the Gaussian kernel ( Nadaraya , 1964 ; Watson , 1964 ) , as recently pointed out in ( Chaudhari et al. , 2019 ; Zhang et al. , 2021a ) , and that more general attention mechanisms can be obtained by choosing other kernels . We also show in Section 3 that the non-local means image denoising algorithm ( Buades et al. , 2005 ) , which can also be understood as a form of kernel-based regression , is the main building block behind the vision transformer ( ViT ) ( Dosovitskiy et al. , 2020 ) . As a consequence , we argue that the key innovation of attention relative to kernel-based regression is not on its ability to capture global long-range interactions that are adapted to the input data ( something that non-local means already does ) , but rather on the use of many learnable parameters for defining attention . In contrast , classical kernel methods typically tune only the kernel bandwidth . In Section 4 we establish several connections between masked attention and Locally Linear Embedding ( LLE ) ( Roweis & Saul , 2000 ; 2003 ) . Specifically , we show that LLE learns a low-dimensional representation of a dataset using a masked self-attention mechanism where the masks are defined by the nearest neighbors of a data point . The resulting coefficients are not constrained to be nonnegative , thus allowing for both positive and negative attention . Moreover , they depend explicitly on multiple data tokens , unlike attention coefficients which depend only on a pair of tokens . We also show that LLE ’ s training objective can be interpreted as a fill in the blanks self-supervised learning objective . However , a key limitation of LLE is that its local neighborhood is pre-specified , so a data point can not attend to any other point . This issue is resolved by self-expressiveness , which connects every point to every other point and uses sparse regularization to reveal which points to attend to . In Section 5 we show that self-attention is closely related the notion of self-expressiveness of Elhamifar & Vidal ( 2009 ; 2013 ) ; Vidal et al . ( 2016 ) , wherein the data points to be clustered are expressed as linear combinations of other points with global coefficients that are adapted to the data and capture long-range interactions among data points . Such self-expressive coefficients are then used to define a data affinity matrix which is used to cluster the data . A first difference between self-attention and self-expressive coefficients is that the latter are not restricted to be non-negative , thus allowing for both positive and negative attention . A second difference is that self-expressive coefficients are not defined as a function of the tokens parametrized by learnable weights . Instead , the coefficients are learned directly using an unsupervised loss . A third difference is that self-expressive coefficients are typically regularized to be sparse or low-rank . As a consequence , we argue that the key innovation of self-attention relative to self-expressiveness is neither in its ability to capture global long-range interactions that are adapted to the data nor in the ability to learn such interactions ( something that self-expressiveness already does ) , but rather on the fact that attention mechanisms use multiple attention-heads in parallel and are stacked into deep architectures . We conclude with future directions on how to improve self-expressiveness using self-attention and vice versa . For example , we argue that the use of sparse regularization in ( Elhamifar & Vidal , 2009 ; 2013 ) to automatically select the most relevant coefficients is a more principled way of handling a large number of tokens than restricting attention to arbitrary local neighborhoods , as done e.g. , in criss-cross attention ( Huang et al. , 2019 ) . To achieve this , we suggest unrolling the sparse encoding mechanism in order to induce sparse attention maps through multiple layers of attention . We conjecture this may not only improve self-attention-based architectures through the use of sparse regularizers on the attention coefficients , but also improve subspace clustering methods by using self-attention , as recently proposed in ( Zhang et al. , 2021b ) . This could also allow one to extend subspace clustering methods to nonlinear manifolds by stacking multiple layers of self-expressiveness . 2 TRANSFORMERS , ATTENTION AND SELF-ATTENTION 2.1 TRANSFORMER The transformer architecture was originally designed for processing data sequences , e.g. , a sequence of words in a sentence . As shown in Figure 1 , each element of the sequence is first mapped to a vector space through a suitable embedding , e.g. , a Word2vec embedding of a word . Since the architecture does not depend on the position and order of the input sequence , a positional encoding is added to the each input embedding . The resulting input tokens are then processed by a multi-head attention layer . This layer computes output tokens as linear combinations of input tokens weighted by attention coefficients designed to capture long-range interactions among input tokens , such as word associations . The output tokens are then processed by a residual connection followed by layer normalization , a feed-forward network such as an MLP , and another residual connection and normalization layer . Therefore , the main component of the transformer architecture is the ( multi-head ) attention layer , which we describe next . 2.2 ATTENTION As illustrated in Figure 2 , the attention layer is designed to capture longrange interactions among three types of input tokens : queries , keys and values . It does so by comparing queries to keys to produce a set of attention coefficients , which are then used to generate linear combinations of the values . Specifically , let us denote the queries by matrix Q = [ q1 , . . . , qNq ] ∈ Rd×Nq , the keys by matrix K = [ k1 , . . . , kNk ] ∈ Rd×Nk , and the corresponding values by matrix V = [ v1 , . . . , vNk ] ∈ Rdv×Nk . The attention layer computes an attention coefficient cij = attn ( ki , qj ) ∈ [ 0 , 1 ] for each key-query pair and returns a linear combination of the values as follows : zj = Nk∑ i=1 vicij or Z = V C , where C = attn ( K , Q ) ∈ [ 0 , 1 ] Nk×Nq . ( 1 ) Intuitively , the attention coefficient cij measures the importance of key ki for representing query qj and the representation zj combines the values vi that are most important for qj . The are many possible choices for the attention mechanism , including additive attention , multiplicative attention and dot product attention . A common choice is scaled dot product attention , which applies a softmax operator to the dot product of keys and queries scaled by the square root of their dimension , i.e . : C = softmax ( K > Q√ d ) = exp ( k > i qj/ √ d ) ∑ i exp ( k > i qj/ √ d ) . ( 2 ) Since the coefficients are non-negative and add up to one , zj is a convex combination of the values . Let us illustrate the intuition behind attention using the following ( overly simplified ) examples : 1 . Suppose we would like to translate sentences from French to English . Let qj be a feature embedding for the jth word of a sentence in French , and let ki = vi be an embedding for the ith word of the corresponding sentence in English . Ideally , the attention mechanism should be designed such that the coefficient cij is large ( cij ≈ 1 ) only for key-query pairs ( i , j ) that correspond to the translation of French word i into English word j , in which case the output to French query qj will be its translation into English zj = vi . 2 . Suppose we are given an image-caption pair and we would like to find which regions in the image corresponds to each word in the caption . Assume we also have a collection of bounding boxes extracted from the image , e.g. , using an object detector . Let the queries be feature embeddings for the words in the caption and the keys and values be CNN features extracted from the bounding boxes . Ideally , the attention mechanism should be designed such that cij is large when the box i corresponds to word j . That is , the attention mechanism is designed to tell us which regions to pay attention to for each word . Of course , in order for multilingual word embeddings to align with each other , or for word embeddings to match image features , both features need to be mapped to a common latent space through a learnable transformation . We discuss such mappings in the next subsection in the context of selfattention , but such mapping also apply here . 2.3 SELF-ATTENTION . Let X = [ x1 , . . . , xN ] ∈ RD×N denote a set of data tokens , such as words or image patches . The goal of self-attention is to capture long-range interactions among such tokens . Such interactions are captured by first transforming these tokens into keys , queries and values via learnable coefficient matrices WK ∈ Rd×D , WQ ∈ Rd×D , and WV ∈ Rdv×d , respectively , as follows : K = WKX ∈ Rd×N , Q = WQX ∈ Rd×N , and V = WVX ∈ Rdv×n . ( 3 ) Then , we can define a set of transformed tokens using attention , e.g . : Z = V softmax ( K > Q/ √ d ) . ( 4 ) Let us illustrate the intuition behind self-attention using the vision transformer ( ViT ) proposed in ( Dosovitskiy et al. , 2020 ) . As shown in Figure 2.3 , the ViT divides an input image into a collection of patches and maps those patches to a set of vectors via a learnable linear projection . Each projected patch is augmented with a positional encoding for the location of the patch in the image . Since ViT is designed for image classification , an additional ( zero ) token is added to the input of the transformer . This token is expected to capture class information and is learned during training . The transformer encoder processes all these tokens using self-attention . Specifically , new tokens are formed as linear combinations of patches weighted by attention coefficients that capture relationships among image patches . Moreover , attention coeffi- cients relating the class token to patch tokens are expected to capture which patches to pay attention to in order to classify the image . The output class token is then passed through an MLP head to produce class probabilities . The network parameters ( input class token , patch projection , self-attention weights , encoder MLP , MLP head ) are learned using a cross-entropy loss for classification .

Summary: This paper studies the correlation between the attention mechanism and many prior arts. Heuristically, this paper links the currently hot topic, attention mechanism, with many milestones works in the past decades before the deep learning era, including subspace learning, sparse coding, kernel regression, non-local means, etc. Among these classic methods, many can be formulated as optimization problems. The minimizer to the optimization problem can be treated as the output of attention modules, and the optimization algorithm can be understood as the ``architecture'' of attention. Specifically, this work highlights the potential of sparse coding in designing the sparse attention mechanism when considering the self-expressiveness in subspace learning is essentially profoundly connected to the formulation of self-attention.