Generalizable Multi-linear Attention Network

1 Introduction . Multimodal learning draws increasing attention in recent years , which aims to process and understand the information from multiple modalities ( i.e . vision , language , audio ) with machine learning skills . Many endeavors have been devoted to the multimodal interaction and robust individual representation learning [ 29 , 6 , 39 , 40 ] . Existing multimodal interaction methods could be categorized into Transformer-based [ 29 , 6 ] and Non-Transformer-based methods [ 39 , 40 ] . With deep stacked attention blocks [ 30 ] and suitable number of training samples , Transformer-based methods could achieve relatively good performances with relatively high computational complexity . After obtaining the individual representations of all the modalities , joint representation learning [ 5 , 15 , 14 , 19 , 36 ] is performed by leveraging simple matrix and vector operations . Bilinear functions have been widely utilized in this field . Specifically , bilinear pooling [ 15 ] employs low-rank decomposition to approximate high-order tensor operations for input vectors . Bilinear attention network ( BAN ) [ 14 ] uses a bilinear attention distribution , on top of low-rank bilinear pooling . In general , BAN exploits bilinear interactions between two groups of input channels and bilinear pooling extracts the joint representations for each pair of channels . Besides , bilinear pooling could be extended to multi-linear pooling with acceptable computational complexity . However , BAN has the limitation to generalize to more modalities due to the complexity . For example , when we want to integrate the features with the same space Rd×T of m modalities , we would obtain a high-order tensor A ∈ RTm that captures multimodal interactions . It is observed that A scales exponentially with the number of modalities . Motivated by the observations , in this paper , we propose a novel method called generalizable multilinear attention network ( MAN ) , which could be leveraged to integrate as many modalities as possible ∗Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . with linear complexity . Concretely , we employ multi-linear pooling to extract the joint representations for multiple feature channels of different modalities . With the previous assumption , we could obtain Tm joint representations for m modalities in total . Following the main idea of BAN , an interaction tensor A ∈ RTm represents the attention weights of the joint representations ( in practise , we devise the combinational addition attention map ) . To reduce the computational complexity and space requirements , we devise a hierarchical approximation decomposition ( HAD ) mechanism for the high-order tensor A , leading to the efficient approximation decomposition . Specifically , considering the fact that softmax attention kernels can not be decomposed as linear operation directly , we devise the addition random features ( ARF ) mechanism to approximate the non-linear softmax function . Furthermore , we facilitate the sparse attention distributions by leveraging local sequential constraints ( LSC ) , which can be combined with ARF conveniently . We conduct extensive experiments on three tasks , the experimental results show that MAN could achieve competitive results compared with the state-of-the-art methods . To sum up , the contributions of our work are four-fold : • We propose the multi-linear attention network ( MAN ) , which is the extension of bilinear attention network ( BAN ) , to learn and utilize multi-linear attention distributions on top of the multi-linear pooling operations . To reduce the computational complexity and space requirements , we devise a hierarchical approximation decomposition mechanism for the combinational addition interaction tensor , leading to the efficient approximation decomposition . • Most importantly , considering that softmax attention kernels can not be decomposed as linear operation directly , we adopt the addition random features ( ARF ) approach to approximate the non-linear softmax functions with enough theoretical analysis . • To process multimodal sequential features , we introduce the local sequential constraints ( LSC ) , which can be combined with ARF conveniently . • To be compatible with Transformer , the multi-head and residual properties are introduced into MAN . We conduct extensive experiments on three different tasks , the experimental results show that MAN could achieve competitive results compared with the baseline methods . 2 Approach . In this section , we first introduce the low-rank multi-linear pooling , which is the extension of bilinear pooling [ 15 ] . Then we illustrate the details of multi-linear attention network . 2.1 Multi-Linear Pooling . We first review the low-rank bilinear pooling . [ 15 ] proposes to reduce the rank of bilinear weight matrix Wi ∈ Rd1×d2 with CP decomposition [ 7 ] . Wi could be rewrited as the multiplication of two smaller matrices UiVTi , where Ui ∈ Rd1×d and Vi ∈ Rd2×d , d denotes the value of rank with the constraint d ≤ min ( d1 , d2 ) . We calculate the scalar output fi as follows : fi = x TWiy ≈ xTUiVTi y = 1T ( UTi x VTi y ) ( 1 ) where 1 ∈ Rd consists of ones and denotes element-wise multiplication , note that we omit the bias term for convenience . When the output is a vector f ∈ Rc , the 1 is replaced by a pooling matrix : f = PT ( UTx VTy ) ( 2 ) where P ∈ Rd×c , U ∈ Rd1×d , and V ∈ Rd2×d . By leveraging the pooling matrix P , the number of parameters is reduced significantly . Based on the bilinear pooling , we introduce the calculation process of multi-linear pooling . Suppose the number of modalities is m and the feature ( channel ) dimensions of them are { dj |j ∈ [ 1 , m ] } , Eq . 2 is extended to : f = PT ( m∏ j=1 UTj vj ) = P T ( UT1 v1 UT2 v2 ... UTmvm ) ( 3 ) where vj ∈ Rdj and Uj ∈ Rdj×d denote the feature vector of j-th modality and corresponding projection matrix . Theoretically , P and { Uj |j ∈ [ 1 , m ] } are the components of d-rank approximation of the high-order tensor . 2.2 Multi-Linear Attention Network . Since the features of most modalities are multi-channel ( for example , video features have temporal dimension ) , we propose MAN to learn and utilize the multi-linear attention distributions on top of the multi-linear pooling operations . Suppose the feature matrices of m modalities are { Vj ∈ Rdj×Tj |j ∈ [ 1 , m ] } , we could obtain ∏m j=1 Tj groups of joint representations . To combine these information , we introduce the multi-linear attention map A ∈ R ∏m j=1 Tj as follows : f ′ k = ( ( A×1 ( VT1 U ′ 1 ) k ) ×2 ( VT2 U ′ 2 ) k ) ×3 ... ×m ( VTmU ′ m ) k ( 4 ) where U ′ j ∈ Rdj×K , ( VTj U ′ j ) k ∈ RTj , the operator ×j denotes the j-th mode product between a tensor and a vector , and f ′ k denotes the k-th element of intermediate representation . Note that Eq . 4 is a multi-linear model for m groups of input channels , where A is a multi-linear weight matrix . To make it easier to understand , Eq . 4 could be rewritten as follows : f ′ k = T1∑ t1=1 ... Tm∑ tm=1 At1 , t2 , ... , tm ( m∏ j=1 VTj , tjU ′ j , k ) ( 5 ) where Vj , tj ∈ Rdj denotes the tj-th channel of input Vj and U ′ j , k ∈ Rdj denotes the k-th column of Uj , At1 , t2 , ... , tm denotes an element in tensor A with indices [ t1 , t2 , ... , tm ] . The intermediate result f ′ ∈ RK consists of K elements [ f ′1 , ... , f ′ k , ... , f ′ K ] . Then the multi-linear joint representation is f = PT f ′ where f ∈ Rc and P ∈ RK×c . For convenience , we define the multi-linear attention network as a function of m multi-channel inputs parameterized by a multi-linear attention map as follows : f = MAN ( { Vj |j ∈ [ 1 , m ] } ; A ) ( 6 ) Combinational Additive Attention Map . We argue that multi-linear attention map should have several properties : ( 1 ) The attention logits are calculated by leveraging the interaction information between different modalities . Each modality is in a symmetric ( equal ) position . ( 2 ) Following the main idea of BAN , the attention map can be calculated with Hadamard product and tensor operations similarly . However , since the absolute values of most multiplicative factors are between 0 and 1 , directly employing softmax function may result in even attention distributions when the attention map is large . ( 3 ) As for the processing of multimodal sequential features , the time interval ( relative position ) of different modalities should be considered . In general , locally close moments are more relevant . Following the idea of Transformer , most positional encoding methods can be integrated with the inner product of two vectors . Therefore , we utilize simply combinational addition operation ( CAO ) to replace the Hadamard product . The attention map A is defined as follows : A : = softmax ( A ) , At1 , t2 , ... , tm = ( m∑ j=1 m∑ q=1,6=j ( VTq , tqU ′′ q ) ( U ′′T j Vj , tj ) ) ( 7 ) where U ′′ j ∈ Rdj×K ′ , A ∈ R ∏m j=1 Tj and the softmax function is applied element-wisely along all the dimensions . In Eq . 7 , the number of additive items ( VTq , tqU ′′ q ) ( U ′′T j Vj , tj ) is positively correlated with the number of modalities , resulting in the bigger difference between elements and sparser attention distributions . Besides , the combinational addition operation has satisfactory compatibility with local sequential constraints introduced in the subsequent section . Local Sequential Constraints ( LSC ) for Multimodal Temporal Features . When the number of modalities increases , the attention tensor A becomes larger and the probability distributions are much denser . As mentioned in the previous section , the features of temporal proximity are more relevant . So we devise an attenuation function related to the time steps of associated modalities . Suppose the number of time steps is T , we assign each time step a Ts -dimensional vector ( T s denotes the number of chunks and s adjusts the scale ) , which consists of e and −e . For example , the temporal vectors for the 1-st and 2-nd temporal chunks are [ e , −e , −e , ... , −e ] and [ e , e , −e , ... , −e ] , the inner product result is ( Ts −1 ) × e 2 . Similarly , the inner product result between any t1 and t2 time steps are ( Ts − |t2 − t1| ) × e 2 , which fits the property of attenuation function . In practice , we simply rewrite the Eq . 7 , concatenating the temporal vectors to the features of different modalities : At1 , t2 , ... , tm = ( m∑ j=1 m∑ q=1,6=j [ VTq , tqU ′′ q , Etq ] [ VTj , tjU ′′ j , Etj ] T ) ( 8 ) To balance the weights of modality association and local constraints , we adjust the value of e. Hierarchical Approximation Decomposition . Considering the fact that the size of combinational additive interaction tensor A scales exponentially to the number of modalities . We devise a hierarchical decomposition mechanism to cope with this problem . Without restrictions , A can be decomposed with CP algorithm : A = m⊗ j=1 Bj , At1 , t2 , ... , tm = 1 T ( m∏ j=1 Bj , tj ) ( 9 ) where 1 ∈ RH , Bj ∈ RH×Tj and H denotes the rank value , ⊗ is the defined outer product operation for multiple matrices , which is a bit different from traditional vectorial operation . The right equation is the detailed calculation for each element , where Bj , tj ∈ RH denotes the tj-th channel of Bj . We directly substitude the Eq . 8 into Eq . 5 as follows2 : f ′ = T1∑ t1=1 ... Tm∑ tm=1 ( 1T ( m∏ j=1 Bj , tj ) ) ( m∏ j=1 VTj , tjU ′ j ) = 1 T ( m∏ j=1 BjV T j U ′ j ) ( 10 ) where we obtain a simple conclusion . Bj ∈ RH×Tj and U ′ j ∈ Rdj×K can be treated as two linear projection weights for different dimensions , respectively . Addition Random Features . Although Eq . 9 is quite simple and pretty , the high-order tensor A can not be decomposed directly due to the non-linearity caused by softmax function . Thus , it is necessary to find a substitute approximate solution . Inspired by the Performer [ 3 ] , we propose the addition random features mechanism , which can be treated as an extension of vanilla random features mechanism for more than two modalities . The detailed process is shown as follows1 : 2The detailed proof and complete algorithm process are shown in the appendix . SM ( { vj |j ∈ [ 1 , m ] } ) = Ew∼N ( 0 , I K ′ ) m∏ j=1 exp ( w > vj − ‖vj‖2 2 ) ( 11 ) where vj denotes the feature vector of j-th modality , SM ( { vj |j ∈ [ 1 , m ] } ) is equal to exp ( ∑m j=1 ∑m k=1,6=j v T k vj ) , E and N ( 0 , IK′ ) denote expectation and sampling distribution . The detailed derivations can be found in the supplementary materials . Eq . 10 is employed to calculate At1 , t2 , ... , tm directly . Bj in Eq . 8 can be obtained by project the features of j-th modality directly with ARF mechanism . In this way , the number of random features is equal to the rank value H of A . Note that softmax function consists of exponential operation and sum normalization , we need the value of normalized denominator . In practice , we directly employ { Bj |j = [ 1 , m ] } to calculate it1 : ∑ A = ∑ ( m∏ j=1 Bj · 1 ) ( 12 ) where ∑ denotes the sum of all elements in the tensor and 1 ∈ RTj . We could keep ∑ A and utilize the integration result f ′ to divide it . Note that , during the calculation of ARF for softmax function , a reduction factor 1H is introduced into both numerator and denominator , thus it directly cancel out . We could treat the expectation operation in Eq . 10 as the summation operation . The core module of MAN is shown in Fig . 1 . Multi-head Multi-linear Attention . Following the main idea of Transformer [ 30 ] that projecting the features into multiple representation spaces could capture more information , we also adopt the multi-head strategy in the multi-linear attention . In practice , we add multiple linear mappings to the input features before fusing them ( i.e . multiple U ′ j and U ′′ j for { VTj U ′ j , V T j U ′′ j |j ∈ [ 1 , m ] } ) . The results of all the heads are concatenated . The calculation follows the process described above . Stacked Multi-linear Attention . The residual connection is proved to be effective in most structures [ 8 ] . Therefore , such strategy can also be introduced into multi-linear attention network . Specifically , Eq . 6 is modified as : f ln+1 = MAN ( f l n , { Vj |j ∈ [ 1 , m ] , j 6= l } ; A ) · 1T + f ln ( 13 ) where 1 ∈ RTl , f ln ∈ Rdl×Tl denotes the n-th output of l-th modality and f l0 = Vl ( if d1 g = d2 g = ... = dm g =K=c , g denotes the number of heads ) . Complexity . Without approximation decomposition , the computational complexity increases exponentially with respect to the number of modalities . Concretely , we need ∏m j=1 Tj low-rank multimodal pooling operations , leading to the unfriendly complexity O ( ∏m j=1 Tj ) . While the complexity scales linearly O ( ∑m j=1 Tj ) with the number of modalities by leveraging approximation decomposition .

Summary: Motivated by the high computational cost of multi-modal multi-linear pooling, this paper proposed an efficient attention based multi-modal fusion mechanism. Low-rank bilinear pooling [1] is expended to multi-modal multi-linear case, following with the Multi-Linear Attention Network (MAN). The MAN combines cross modal attention and temporal feature, which is also decomposed based on Performers. Experiments show that the FLOPs decrease with a stable performance on multimodal down-stream task.