Generalizable Multi-linear Attention Network

1 Introduction . Multimodal learning draws increasing attention in recent years , which aims to process and understand the information from multiple modalities ( i.e . vision , language , audio ) with machine learning skills . Many endeavors have been devoted to the multimodal interaction and robust individual representation learning [ 29 , 6 , 39 , 40 ] . Existing multimodal interaction methods could be categorized into Transformer-based [ 29 , 6 ] and Non-Transformer-based methods [ 39 , 40 ] . With deep stacked attention blocks [ 30 ] and suitable number of training samples , Transformer-based methods could achieve relatively good performances with relatively high computational complexity . After obtaining the individual representations of all the modalities , joint representation learning [ 5 , 15 , 14 , 19 , 36 ] is performed by leveraging simple matrix and vector operations . Bilinear functions have been widely utilized in this field . Specifically , bilinear pooling [ 15 ] employs low-rank decomposition to approximate high-order tensor operations for input vectors . Bilinear attention network ( BAN ) [ 14 ] uses a bilinear attention distribution , on top of low-rank bilinear pooling . In general , BAN exploits bilinear interactions between two groups of input channels and bilinear pooling extracts the joint representations for each pair of channels . Besides , bilinear pooling could be extended to multi-linear pooling with acceptable computational complexity . However , BAN has the limitation to generalize to more modalities due to the complexity . For example , when we want to integrate the features with the same space Rd×T of m modalities , we would obtain a high-order tensor A ∈ RTm that captures multimodal interactions . It is observed that A scales exponentially with the number of modalities . Motivated by the observations , in this paper , we propose a novel method called generalizable multilinear attention network ( MAN ) , which could be leveraged to integrate as many modalities as possible ∗Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . with linear complexity . Concretely , we employ multi-linear pooling to extract the joint representations for multiple feature channels of different modalities . With the previous assumption , we could obtain Tm joint representations for m modalities in total . Following the main idea of BAN , an interaction tensor A ∈ RTm represents the attention weights of the joint representations ( in practise , we devise the combinational addition attention map ) . To reduce the computational complexity and space requirements , we devise a hierarchical approximation decomposition ( HAD ) mechanism for the high-order tensor A , leading to the efficient approximation decomposition . Specifically , considering the fact that softmax attention kernels can not be decomposed as linear operation directly , we devise the addition random features ( ARF ) mechanism to approximate the non-linear softmax function . Furthermore , we facilitate the sparse attention distributions by leveraging local sequential constraints ( LSC ) , which can be combined with ARF conveniently . We conduct extensive experiments on three tasks , the experimental results show that MAN could achieve competitive results compared with the state-of-the-art methods . To sum up , the contributions of our work are four-fold : • We propose the multi-linear attention network ( MAN ) , which is the extension of bilinear attention network ( BAN ) , to learn and utilize multi-linear attention distributions on top of the multi-linear pooling operations . To reduce the computational complexity and space requirements , we devise a hierarchical approximation decomposition mechanism for the combinational addition interaction tensor , leading to the efficient approximation decomposition . • Most importantly , considering that softmax attention kernels can not be decomposed as linear operation directly , we adopt the addition random features ( ARF ) approach to approximate the non-linear softmax functions with enough theoretical analysis . • To process multimodal sequential features , we introduce the local sequential constraints ( LSC ) , which can be combined with ARF conveniently . • To be compatible with Transformer , the multi-head and residual properties are introduced into MAN . We conduct extensive experiments on three different tasks , the experimental results show that MAN could achieve competitive results compared with the baseline methods . 2 Approach . In this section , we first introduce the low-rank multi-linear pooling , which is the extension of bilinear pooling [ 15 ] . Then we illustrate the details of multi-linear attention network . 2.1 Multi-Linear Pooling . We first review the low-rank bilinear pooling . [ 15 ] proposes to reduce the rank of bilinear weight matrix Wi ∈ Rd1×d2 with CP decomposition [ 7 ] . Wi could be rewrited as the multiplication of two smaller matrices UiVTi , where Ui ∈ Rd1×d and Vi ∈ Rd2×d , d denotes the value of rank with the constraint d ≤ min ( d1 , d2 ) . We calculate the scalar output fi as follows : fi = x TWiy ≈ xTUiVTi y = 1T ( UTi x VTi y ) ( 1 ) where 1 ∈ Rd consists of ones and denotes element-wise multiplication , note that we omit the bias term for convenience . When the output is a vector f ∈ Rc , the 1 is replaced by a pooling matrix : f = PT ( UTx VTy ) ( 2 ) where P ∈ Rd×c , U ∈ Rd1×d , and V ∈ Rd2×d . By leveraging the pooling matrix P , the number of parameters is reduced significantly . Based on the bilinear pooling , we introduce the calculation process of multi-linear pooling . Suppose the number of modalities is m and the feature ( channel ) dimensions of them are { dj |j ∈ [ 1 , m ] } , Eq . 2 is extended to : f = PT ( m∏ j=1 UTj vj ) = P T ( UT1 v1 UT2 v2 ... UTmvm ) ( 3 ) where vj ∈ Rdj and Uj ∈ Rdj×d denote the feature vector of j-th modality and corresponding projection matrix . Theoretically , P and { Uj |j ∈ [ 1 , m ] } are the components of d-rank approximation of the high-order tensor . 2.2 Multi-Linear Attention Network . Since the features of most modalities are multi-channel ( for example , video features have temporal dimension ) , we propose MAN to learn and utilize the multi-linear attention distributions on top of the multi-linear pooling operations . Suppose the feature matrices of m modalities are { Vj ∈ Rdj×Tj |j ∈ [ 1 , m ] } , we could obtain ∏m j=1 Tj groups of joint representations . To combine these information , we introduce the multi-linear attention map A ∈ R ∏m j=1 Tj as follows : f ′ k = ( ( A×1 ( VT1 U ′ 1 ) k ) ×2 ( VT2 U ′ 2 ) k ) ×3 ... ×m ( VTmU ′ m ) k ( 4 ) where U ′ j ∈ Rdj×K , ( VTj U ′ j ) k ∈ RTj , the operator ×j denotes the j-th mode product between a tensor and a vector , and f ′ k denotes the k-th element of intermediate representation . Note that Eq . 4 is a multi-linear model for m groups of input channels , where A is a multi-linear weight matrix . To make it easier to understand , Eq . 4 could be rewritten as follows : f ′ k = T1∑ t1=1 ... Tm∑ tm=1 At1 , t2 , ... , tm ( m∏ j=1 VTj , tjU ′ j , k ) ( 5 ) where Vj , tj ∈ Rdj denotes the tj-th channel of input Vj and U ′ j , k ∈ Rdj denotes the k-th column of Uj , At1 , t2 , ... , tm denotes an element in tensor A with indices [ t1 , t2 , ... , tm ] . The intermediate result f ′ ∈ RK consists of K elements [ f ′1 , ... , f ′ k , ... , f ′ K ] . Then the multi-linear joint representation is f = PT f ′ where f ∈ Rc and P ∈ RK×c . For convenience , we define the multi-linear attention network as a function of m multi-channel inputs parameterized by a multi-linear attention map as follows : f = MAN ( { Vj |j ∈ [ 1 , m ] } ; A ) ( 6 ) Combinational Additive Attention Map . We argue that multi-linear attention map should have several properties : ( 1 ) The attention logits are calculated by leveraging the interaction information between different modalities . Each modality is in a symmetric ( equal ) position . ( 2 ) Following the main idea of BAN , the attention map can be calculated with Hadamard product and tensor operations similarly . However , since the absolute values of most multiplicative factors are between 0 and 1 , directly employing softmax function may result in even attention distributions when the attention map is large . ( 3 ) As for the processing of multimodal sequential features , the time interval ( relative position ) of different modalities should be considered . In general , locally close moments are more relevant . Following the idea of Transformer , most positional encoding methods can be integrated with the inner product of two vectors . Therefore , we utilize simply combinational addition operation ( CAO ) to replace the Hadamard product . The attention map A is defined as follows : A : = softmax ( A ) , At1 , t2 , ... , tm = ( m∑ j=1 m∑ q=1,6=j ( VTq , tqU ′′ q ) ( U ′′T j Vj , tj ) ) ( 7 ) where U ′′ j ∈ Rdj×K ′ , A ∈ R ∏m j=1 Tj and the softmax function is applied element-wisely along all the dimensions . In Eq . 7 , the number of additive items ( VTq , tqU ′′ q ) ( U ′′T j Vj , tj ) is positively correlated with the number of modalities , resulting in the bigger difference between elements and sparser attention distributions . Besides , the combinational addition operation has satisfactory compatibility with local sequential constraints introduced in the subsequent section . Local Sequential Constraints ( LSC ) for Multimodal Temporal Features . When the number of modalities increases , the attention tensor A becomes larger and the probability distributions are much denser . As mentioned in the previous section , the features of temporal proximity are more relevant . So we devise an attenuation function related to the time steps of associated modalities . Suppose the number of time steps is T , we assign each time step a Ts -dimensional vector ( T s denotes the number of chunks and s adjusts the scale ) , which consists of e and −e . For example , the temporal vectors for the 1-st and 2-nd temporal chunks are [ e , −e , −e , ... , −e ] and [ e , e , −e , ... , −e ] , the inner product result is ( Ts −1 ) × e 2 . Similarly , the inner product result between any t1 and t2 time steps are ( Ts − |t2 − t1| ) × e 2 , which fits the property of attenuation function . In practice , we simply rewrite the Eq . 7 , concatenating the temporal vectors to the features of different modalities : At1 , t2 , ... , tm = ( m∑ j=1 m∑ q=1,6=j [ VTq , tqU ′′ q , Etq ] [ VTj , tjU ′′ j , Etj ] T ) ( 8 ) To balance the weights of modality association and local constraints , we adjust the value of e. Hierarchical Approximation Decomposition . Considering the fact that the size of combinational additive interaction tensor A scales exponentially to the number of modalities . We devise a hierarchical decomposition mechanism to cope with this problem . Without restrictions , A can be decomposed with CP algorithm : A = m⊗ j=1 Bj , At1 , t2 , ... , tm = 1 T ( m∏ j=1 Bj , tj ) ( 9 ) where 1 ∈ RH , Bj ∈ RH×Tj and H denotes the rank value , ⊗ is the defined outer product operation for multiple matrices , which is a bit different from traditional vectorial operation . The right equation is the detailed calculation for each element , where Bj , tj ∈ RH denotes the tj-th channel of Bj . We directly substitude the Eq . 8 into Eq . 5 as follows2 : f ′ = T1∑ t1=1 ... Tm∑ tm=1 ( 1T ( m∏ j=1 Bj , tj ) ) ( m∏ j=1 VTj , tjU ′ j ) = 1 T ( m∏ j=1 BjV T j U ′ j ) ( 10 ) where we obtain a simple conclusion . Bj ∈ RH×Tj and U ′ j ∈ Rdj×K can be treated as two linear projection weights for different dimensions , respectively . Addition Random Features . Although Eq . 9 is quite simple and pretty , the high-order tensor A can not be decomposed directly due to the non-linearity caused by softmax function . Thus , it is necessary to find a substitute approximate solution . Inspired by the Performer [ 3 ] , we propose the addition random features mechanism , which can be treated as an extension of vanilla random features mechanism for more than two modalities . The detailed process is shown as follows1 : 2The detailed proof and complete algorithm process are shown in the appendix . SM ( { vj |j ∈ [ 1 , m ] } ) = Ew∼N ( 0 , I K ′ ) m∏ j=1 exp ( w > vj − ‖vj‖2 2 ) ( 11 ) where vj denotes the feature vector of j-th modality , SM ( { vj |j ∈ [ 1 , m ] } ) is equal to exp ( ∑m j=1 ∑m k=1,6=j v T k vj ) , E and N ( 0 , IK′ ) denote expectation and sampling distribution . The detailed derivations can be found in the supplementary materials . Eq . 10 is employed to calculate At1 , t2 , ... , tm directly . Bj in Eq . 8 can be obtained by project the features of j-th modality directly with ARF mechanism . In this way , the number of random features is equal to the rank value H of A . Note that softmax function consists of exponential operation and sum normalization , we need the value of normalized denominator . In practice , we directly employ { Bj |j = [ 1 , m ] } to calculate it1 : ∑ A = ∑ ( m∏ j=1 Bj · 1 ) ( 12 ) where ∑ denotes the sum of all elements in the tensor and 1 ∈ RTj . We could keep ∑ A and utilize the integration result f ′ to divide it . Note that , during the calculation of ARF for softmax function , a reduction factor 1H is introduced into both numerator and denominator , thus it directly cancel out . We could treat the expectation operation in Eq . 10 as the summation operation . The core module of MAN is shown in Fig . 1 . Multi-head Multi-linear Attention . Following the main idea of Transformer [ 30 ] that projecting the features into multiple representation spaces could capture more information , we also adopt the multi-head strategy in the multi-linear attention . In practice , we add multiple linear mappings to the input features before fusing them ( i.e . multiple U ′ j and U ′′ j for { VTj U ′ j , V T j U ′′ j |j ∈ [ 1 , m ] } ) . The results of all the heads are concatenated . The calculation follows the process described above . Stacked Multi-linear Attention . The residual connection is proved to be effective in most structures [ 8 ] . Therefore , such strategy can also be introduced into multi-linear attention network . Specifically , Eq . 6 is modified as : f ln+1 = MAN ( f l n , { Vj |j ∈ [ 1 , m ] , j 6= l } ; A ) · 1T + f ln ( 13 ) where 1 ∈ RTl , f ln ∈ Rdl×Tl denotes the n-th output of l-th modality and f l0 = Vl ( if d1 g = d2 g = ... = dm g =K=c , g denotes the number of heads ) . Complexity . Without approximation decomposition , the computational complexity increases exponentially with respect to the number of modalities . Concretely , we need ∏m j=1 Tj low-rank multimodal pooling operations , leading to the unfriendly complexity O ( ∏m j=1 Tj ) . While the complexity scales linearly O ( ∑m j=1 Tj ) with the number of modalities by leveraging approximation decomposition .

Summary: For efficient multimodal learning having more than two modalities, they generalized the bilinear attention networks (Kim et al., 2018) to multi-linear attention networks using the same gist of low-rank pooling (Kim et al., 2017) and the sampling-based softmax approximation (Choromanski et al., 2021). To reduce the computational cost to build a multi-linear attention tensor, the proposed sampling-based approach was a novel idea using the well-established softmax approximation in the transformer-based architectures. Their extensive experiments on four datasets of three different tasks confirm that the proposed method consistently outperforms the comparative methods while giving the ablation study to assess the sub-modules one-by-one and the impact of the number of samples used in the softmax approximation.