Generalizable Multi-linear Attention Network
Authors: Tao Jin
Zhou Zhao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multimodal
Attention Mechanism
Tensor Decomposition

Abstract: The majority of existing multimodal sequential learning methods focus on how to obtain effective representations and ignore the importance of multimodal fusion. Bilinear attention network (BAN) is a commonly used fusion method, which leverages tensor operations to associate the features of different modalities. However, BAN has a poor compatibility for more modalities, since the computational complexity of the attention map increases exponentially with the number of modalities. Based on this concern, we propose a new method called generalizable multi-linear attention network (MAN), which can associate as many modalities as possible in linear complexity with hierarchical approximation decomposition (HAD). Besides, considering the fact that softmax attention kernels cannot be decomposed as linear operation directly, we adopt the addition random features (ARF) mechanism to approximate the non-linear softmax functions with enough theoretical analysis. We conduct extensive experiments on four datasets of three tasks (multimodal sentiment analysis, multimodal speaker traits recognition, and video retrieval), the experimental results show that MAN could achieve competitive results compared with the state-of-the-art methods, showcasing the effectiveness of the approximation decomposition and addition random features mechanism.

Generalizable Multi-Linear Attention Network
Tao Jin
Zhou Zhao" Zhejjang University
Zhejiang University j jint__] =zjuuzuu. edu. cn
zhaozhoudzju. edu..du.co cn
Abstract
The majority existing multimodal sequential learning methods focus on how obtain powerful individual representations and neglect to effectively capture the multimodal joint representation. Bilinear attention network (BAN) is a com- monly used integration method, which leverages tensor operations associate the features different modalities. However, BAN has a poor compatibility for modalities, since the computational complexity of the attention map increases exponentially with the number of modalities. Based on this concern, we propose a new method called generalizable multi-linear attention network (MAN), which can associate more modalities in acceptable complexity with hierarchical approxima- tion decomposition. Specifically, considering the fact that softmax attention kernels cannot be decomposed as linear operation directly, we adopt the addition random features mechanism approximate the non- non-linear softmax functions with enough theoretical analysis. Furthermore, we also introduce the local sequential constraints, which can be combined with ARF conveniently, as positional information. We conduct extensive experiments on several datasets of corresponding tasks, the experimental results show that MAN could achieve competitive results compared with baseline methods, showcasing the effectiveness of our contributions.
Introduction
Multimodal learning draws increasing attention in recent years, which aims to process and understand the information from multiple modalities (i.e. vision. language, audio) with machine learning skills. Many endeavors have been devoted to the multimodal interaction and robust individual
representation learning (29 6 39 40
Existing multimodal interaction methods could be categorized
into Transformer-baat based 29 and Non-Transformer-b
methods [39 40 With deep stacked
attention blocks (30) and suitable number of training samples, Transformer-basee methods could achieve relatively good performances with relatively high computational complexity. After obtaining individual representations of all modalities, joint representation learning [5]
15
14
19 36)
is performed by leveraging simple matrix and vector operations. Bilinear functions have been widely utilized in this field. Specifically, bilinear pooling [15] employs low-rank decomposition to approximate high-order tensor operations for input vectors. Bilinear attention network (BAN) [14] uses bilinear attention distribution, on top of low-rank bilinear pooling. In general. BAN exploits bilinear interactions between two groups of input channels and bilinear pooling extracts the joint representations for each pair of channels. Besides, bilinear pooling could be extended to multi-linear pooling with acceptable computational complexity. However, BAN has the limitation to generalize to more modalities due to the complexity. For example, when we want to integrate the features with the same space RdXT of m modalities, we would obtain a high-order tensor E R 7 that captures multimodal interactions. It is observed that A scales exponentially with the number of modalities.
Motivated by the observations, in this paper, we propose novel method called generalizable multi- linear attention network (MAN), which could be leveraged to integrate as many modalities as possible
Corresponding Author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

with linear complexity. Concretely, we employ multi- minti-linear pooling to extract the joint representations for multiple feature channels of different modalities. With the previous assumption, we could obtain T joint representationss for m modalities in total. Following the main idea of BAN, an interaction lensor A E RI' represents the attention weights of the joint representations (in practise, we devise the combinational addition attention map). To reduce the computational complexity and space requirements, we devise hierarchical approximation decomposition (HAD) mechanism for the high-order tensor A, leading to the efficient approximation decomposition. Specifically, considering the fact that softmax attention kemnlss cannot be decomposed as linear operation directly, we devise the addition random features (ARF) mechanism approximate the non-linear softmax function. Furthermore, we facilitate the sparse attention distributions by leveraging local sequential constraints (L.SC), which can combined with ARF conveniently. We conduct extensive experiments on three tasks, the experimental results show that MAN could achieve competitive results compared with the state-of-the-ar methods. To sum up, the contributions of our work are four-fold:
We propose the multi-lines attention network (MAN), which is the extension of bilinear attention network (BAN), to learn and utilize multi-linear attention distributions on top of the multi-linear pooling operations. To reduce the computational complexity and space requirements, we devise a hierarchical approximation decomposition mechanism for the combinational addition interaction tensor, leading to the efficient approximation decomposition.
Most importantly, considering that softmax attention kernels cannot be decomposed as linear operation directly, we adopt the addition random features (ARF) approach to approximate the non-linear softmax functions with enough theoretical analysis.
To process multimodal sequential features, introduce the local sequential constraints (LSC), which can be combined with ARF conveniently.
To be compatible with Transformer, the multi-head and residual properties are introduced into MAN. We conduct extensive experiments on three different tasks, the experimental results show that MAN could achieve competitive results compared with the baseline methods.
2 Approach
In this section, first introduce the low-rank multi-linear pooling, which is the extension of bilinear pooling 15).
Then we illustrate the details of multi-linear attention network.
2.1 Multi-Linear Pooling
We first review the low-rank bilinear pooling. 15 proposes to reduce the rank of bilinear weight matrix W, Rdixd, with CP decomposition 77. W. could be rewrited as the multiplication of two smaller matrices U,V?, where U, E TRd, xd and V, E Rd2x xd d denotes the value of rank with the constraint d < min(di, d2). We calculate the scalar output fa follows:
f = w X 1st (UfxoV?y)
(I)
where 1 Rd consists of ones and denotes element- multiplication, note that we omit the bias term for convenience. When the output is vector f E R', the 1 is replaced by a pooling matrix:
TT (U"xoV'y)
where P E gdxe U E Rdidd and V E gdaxd, By leveraging the pooling matrix P. the number of parameters is reduced significantly. Based on the bilinear pooling, we introduce the calculation process of multi-linear pooling. Suppose the number of modalities is and the feature (channel) dimensions of them are {d,lū E m]). Eq. is extended to:
= 7
o U v
(3)

---

where s,,,, E Rd: denotes the ty-th channel of input V, and U j. E Rd denotes the k-th column of Uj. At,.a..... denotes an element in tensor A with indices [t, fm.. The intermediate result f' E R consists of K elements [ Se. [k] Then the multi-linear joint representation ff' where f E R. and P IR Kxe, For convenience, we define the linear attention B, B
V
V
B,
V
2/
V
A
B2
V U2
B2V
Fusion
V/3
V
B3
Vg'Ug
B, V
Multi--lnnar Attention Map
Multi linear Attention Networks
Figure The core module of multi-linear attention network (only one head), where provide detailed input and output of each step, making it easy to understand. Note that, for convinience, we give an example of three modalities.
where Vj E R& and U, E Rd xd denote the feature vector of j-th modality and corresponding projection matrix. Theoretically, P and {U,li m]] the components of d-rank approximation of the high-order tensor.
2.2 Multi-Linear Attention Network
Since the features of most modalities are multi-channel (for example, video features have temporal dimension), we propose MAN to learn and utilize the multi-linear attention distributions on top of the multi-linear pooling operations. Suppose the feature matrices m modalities are {V, E Rdd [1, m|}, we could obtain ITy= T, groups of joint representations. To combine these information, we
introduce the multi lineer attention map A E RI T as follows:
S/ ((A X1 (VTU))) X2 (VTU,)) X3 Xm (V
m U m.)
(4)
where U, RRIK (VTU,)& E RR, the operator x, denotes the j-th mode product between tensor and vector, and f denotes the k-th element of intermediate representation. Note that Eq. a multi-linear model for groups input channels, where A is multi-linear weight matrix. To
make it easier to understand, Eq. could be rewritten as follows:
m
1
j.ly U,A))
(5)
11=1
j=1
network as function of m multi-channel inputs parameterized by a multi-linear attention map as follows:
MAN((VV e m1] m]]]])
(6)
Combinational Additive Attention Map. We argue that multi-linear attention map should have several properties: (1) The attention logits are calculated by leveraging the interaction information between different modalities. Each modality is in a symmetric (equal) position. (2) Following the main idea of BAN, the attention map can be calculated with Hadamard product and tensor operations similarly. However, since the absolute values of most multiplicative factors are between 0 and 1, directly employing softmax function may result in even attention distributions when the attention map is large. (3) for the processing of multimodal sequential features, the time interval (relative position) of different modalities should be considered. In general, locally close moments are more

---

relevant. Following the idea of Transformer, most positional encoding methods can be integrated with the inner product of two vectors. Therefore, we utilize simply combinational addition operation (CAO) to replace the Hadamard product. The attention map A is defined as follows:
A := softmaxx()))
A Po
Σ Σ
ai, (U) s1,))
j=l
where Uj E RRdx A RIT;= Tú and the softmax function is applied element-wisely along all the dimensions. Eq. 7 the number additive items (V q,c U, Vvtt, is positively correlated with the number of modalities, resulting in the bigger difference between elements and sparser attention distributions. Besides, the combinational addition operation has satisfactory compatibility with local sequential constraints introduced in the subsequent section.
s.t,))
Local Sequential Constraints (LSC) for Multimodal Temporal Features. When the number of modalities increases, the attention tensor A becomes larger and the probability distributions are much denser. As mentioned in the previous section, the features of temporal proximity are more relevant. So we devise an attenuation function related the time steps of associated modalities. Suppose the number of time steps is T, we assign each time step a -dimensional vector ( denotes the number of chunks and adjusts the scale), which consists of e and e. For example, the temporal vectors for the 1-st and 2-nd temporal chunks are [e, e, .... -e] and [e, e, -G, .... -e]. the inner product result is ( - 1) x e². Similarly, the inner product result between any 1 and &2 time steps are (2 It. 1- tıl) x 2 which fits the property of attenuation function. In practice, we simply rewrite the Eq. concatenating the temporal vectors to the features of different modalities:
jig Uj. E,,) )
V U" 9:4
Et,
(8)
2=
1=1,11
To balance the weights of modality association and local constraints, adjust the value of e.
Hierarchical Approximation Decomposition. Considering the fact that the size of combinational additive interaction tensor A scales exponentially to the number of modalities. We devise a hierarchi- cal decomposition mechanism cope with this problem. Without restrictions. can be decomposed with CP algorithm:
A
X j=1
Bj,
A At..g.....
1 1T [ B3,,)
where 1 E RR B, E HXT; and H denotes the rank value, @ is the defined outer product operation for multiple matrices, which is a bit different from traditional vectorial operation. The right equation is the detailed calculation for each element, where B Bj,,, E Iid denotes the t,-th channel of By. We directly substitude the Eq. into Eq.
5 as follow?
T
Tm
D
Jdy U,) = 1f I B
(10)
where we obtain simple conclusion. B, E RHXY and Uj E BddxK x can be treated as two linear projection weights for different dimensions, respectively.
Addition Random Features. Although Eq. iis quite simple and pretty, the high-order tensor A cannot be decomposed directly due to the non-lineed non-linearity caused by softmax function. Thus, it is necessary to find a substitute approximate solution. Inspired by the Performer [3), we propose the addition random features mechanism, which can be treated extension of vanilla random features mechanism for more than two modalities. The detailed process is shown as follows!
Thee detailed proof and complete algorithm process shown in the appendix.

---

v 2
SM([v,li E (1, = R (
II exp w T v
(11)
where oj denotes the feature vector of j-th modality, SM([vylī E [1,ml]] is equal to j=1 E k= v v2), E and N x0,Tx*) x*) denote expectation and sampling distribution. The detailed derivations can be found in the supplementary materials. Eq. To employed to calculate At, dta....... directly. B, in Eq. 8 can
exp(_== m
obtained by project the features of j-th modality directly with ARF mechanism. In this way. the number of random features is equal to the rank value H of A. Note that softmax function consists of exponential operation and sum normalization, we need the value of normalized denominator. In practice, we directly employ {Bylj = [1. mm]] to calculate it!:
m 1)
ΣA
Σ
IIIB, B,
(12)
where Σ denotes the sum of all elements in the tensor and 1 E RR. We could keep Σ A and utilize the integration result f' to divide it. Note that, during the calculation of ARF for softmax function, reduction factor π is introduced into both numerator and denominator, thus it directly cancel out. We could treat the expectation operation in Eq. ao as the summation operation. The core module of MAN is shown in Fig. I
Multi-head Multi-linear Attention. Following the main idea of Transformer 30 that projecting the features into multiple representation spaces could capture more information, we also adopt the multi-head strategy in the multi-linear attention. In practice, we add multiple linear mappings to the input features before fusing them (i.e. multiple U and U for (V U VIJ [1, ml} ). The results of all the heads are concatenated. The calculation follows the process described above.
Stacked Multi-linear Attention. The residual connection is proved to be effective in most structures [. Therefore, such strategy can also be introduced into multi- linear attention network. Specifically, Eq. is modified as:
F1. = MAN((F (Vylú E (1, (1:A) 1
(13)
where E R" fl E RdXTT denotes the n-th output of 1-th modality and f V (if
-a.K=k g denotes the number of heads).
Complexity. Without approximation decomposition, the computational complexity increases expo- nentially with respect to the number of modalities. Concretely, we need [] Tú low-rank multimodal pooling operations, leading to the unfriendly complexity OQI, Tý). While the complexity scales linearly 01E;=110 T3)
with the number of modalities by leveraging approximation decomposition.
Experiments
3.1
Datasets
We evaluate MAN on three challenging tasks, multimodal sentiment analysis, multimodal speaker traits recognition, and multimodal video retrieval. this section, we provide brief introduction of the datasets (CMU-MOSI [41) for multimodal sentiment analysis, POM (25) for multimodal speaker 33 and LSMDC (28 for multimodal video retrieval).
traits recognition, MSR-VTV -TR-VITTA
The CMU-MOSI dataset is a collection of 93 opinion videos from YouTube movie reviews. Each video consists of multiple opinion segments (2199 segments in total) and each segment annotated with the score in the range [-3, 3). where -3 and indicate highly negative and positive. There 1284 segments in the training set, 229 in the validation set, and 686 in the test
CMU-MOSI:
POM: POM multimodal speaker traits recognition dataset made up of 903 movie review videos. Each video is annotated for various personality and speaker traits, specifically: Confident (con),
5

---

Model Metric
BA
F1
MAE
Corr
MA
MV-LSTM [27
73,9
74.0
1.019
0.601
33.2
TFN 38
73.9
73.4
1.040
0.633
32.1
MARN [40]
77.1
77.0
0.968
0.625
34.7
MFN [39
77.4
77.3
0.965
0.632
34.1
RAVEN 31
78.0
0.915
0.691
LMF 19
76.4
75.7
0.912
0.668
32.8
MAN Mon-Transformer)
79.4
79.3
0.894
0.697
37.8
MulT 29
83.0
82.8
0.870
0.698
40.0
MAN (Transformer)
82.7
83.0
0.866
0.707
40.5
Table I: MAN achieves superior performances over baseline models for CMU-MOSI dataset. We report BA (binary aceuracy), F1, Corr (Pearson Correlation Coefficient), and MA (Multi-class accuracy, all higher is better), MAE (Mean-absolute Error, lower better).
Con
Pas
Voi
Dom
Cre
Viv
Exp
Ent
Model Trait
MA7
MA7
MA7
MA7
MA7
MA7
MA7
MA7
MV-LSTM [27]
25.6
28.6
28.1
34.5
25.6
32.5
32.5
29.6
TFN [8
24.1
31.0
31.5
34.5
24.6
25.6
27.6
29.1
MARN 40
29.1
33.0
31.5
MFN 39
34.5
35.5
37.4
41.9
34.5
36.9
36.0
37.9
LMF
29
35.9
35.9
34.8
39.6
34.5
35.9
37.8
36.5
MAN (Non- Transfornffr)
37.5
38.3
37.9
43.6
37.0
38.7
40.5
38.9
MulT
34.5
34.5
36.5
38.9
37.4
36.9
37.9
39.4
MAN (Tranformer)
37.9
38.9
37.9
44.8
37.9
39.9
40.9
40.4
Res
Tru
Rel
Out
Tho
Ner
Per
Hum
Model Trait
MA5
MA5
MA5
MA5
MA5
MA5
MA7
MA5
MV-LSTM 27
33.0
52.2
50.7
38.4
37.9
42.4
26.1
38.9
TFN 38
30.5
38.9
35.5
37.4
33.0
42.4
27.6
33.0
MARN 40
36.9
52.2
47.3
31.0
44.8
MFN 39
38.4
57.1
53.2
46.8
47.3
47.8
34.0
47.3
LMF
29
35.5
54.2
53.2
44.8
42.7
43.5
34.9
45.8
MAN (Non- TranTronnffrrr
39.6
59.5
55.2
47.4
46.5
47.0
37.2
49,8
MulT 29
41.4
60.6
54.2
43.3
49.3
46.3
33.5
43.3
MAN (Transformer)
41.9
61.4
55.2
49.1
44.8
48.3
38.3
51.2
Table 2: MAN achieves superior performances over baseline models in POM dataset (multimodal personality traits recognition). MA(5,7) denotes multi-class accuracy for (5,7) classes.
Passionate (Pas), Voice Pleasant (Voi), Dominant (Dom), Credible (Cre), Vivid (Viv), Expertise (Exp), Entertaining (Ent), Reserved (Res), Trusting (Tru), Relaxed (Rel), Outgoing (Out), Thorough (Tho), Nervous (Ner), Persuasive (Per) and Humorous (Hum). The training, validation, and test set distributions are approximately 600, 100, and 203, respectively.
MSR-VTT: MSR-VTT is composed of 10K YouTube videos, collected using 257 queries from commercial video search engine. Each video 10 to 30s long. and is paired with 20 natural sentences describing it, obtained from Amazon Mechanical Turk Workers. We report results the train/test splits introduced in 34 that uses 9000 videos for training and 1000 for testing.
LSMDC: It contains 11081 short video clips (about 45s) extracted from 202 movies. Each clip is annotated with a caption, extracted from either the movie script or the audio description. The test set composed of 1000 videos, from movies not presented in the training set.

---

Model Metric
BA
F1
MAE
Corr
MA
w/o CAO
78,4
78.3
0.924
0.668
35.2
w/o LSC
78.4
78.3
0.928
0.673
36.5
w/o HAD
79.4
79.3
0.901
0.693
38.1
MAN Non--Transfone
79.4
79.3
0.894
0.697
37.8
w/o CAO
81.5
81.6
0.880
0.692
38,9
w/o LSC
81.4
81.3
0.883
0.698
39.8
w/o HAD
82.6
82.8
0.872
0.707
40.5
MAN (Transformer)
82.7
83.0
0.866
0.707
40.5
Table 3: Ablation study on CMU-MOSI dataset.
Metric Model
w/o CAO
w/o HAD
w/o LSC
MAN
FLOPs
8.31 X 10
10.23 x 10th
3.07 X 10
3.87 X 10S
Table Computational complexity different variants on CMU-MOSI, note that we only consider the FLOPs of one-layer integration module.
3.2 Experiments for Multimodal Sentiment Analysis and Speaker Traits Recognition
Data Preprocessing: Each dataset (CMU-MOSI, POM) consists of three modalities. including textual, visual, and audio modalities. For textual features, employ the pre-trained 300-dimensional Glove embeddings [24]. For visual features, we utilize Facet [11] to indicate 35 facial action units, which records facial muscle movement for representing the basic and advanced emotions. For audio features, we use COVAREP [ acoustic analysis framework. To align the different modalities along the temporal dimension, we perform word alignment with P2FA 37 which aligns the modalities the word granularity.
Experimental Details: Transformer-based and Non-Transformer-baeee methods both have their practical meanings of existence. For example, without enough labeled data, non- non-transformer based methods may be optimal solution due to less computational complexity. Henceforce, combine MAN with both Transformer-basee and Non- Non-Transformer- based backbones and make corresponding comparisons with existing methods. We refer respectively. Note that we do not remove the original structures of [29] and 19), but add MAN as an extra module. We fuse their results for the tasks. The hyperparameterss of MAN include Adam learning rate 0.001, the structure of integration network (1 layer of integration block, with hidden sizes of 40, number of heads 10, number of random features 24). "Hidden size" denotes the common size of di, dm.....dm.. We divide all the time steps into chunks and apply local sequential constraints. Furthermore, for the specific hyperparameters of backbones, we use similar values as original papers.
(29 and [19] for our implementation of backbones,
Results: Table D presents the overall comparison of MAN and existing methods on CMU-MOSI dataset. As for the -on-Transformer- based methods, we could observe that MV-LSTM, MFN, MARN, RAVEN perform worse than MAN as they pay more attention to multimodal interaction and representation learning, and ignore the importance of subsequent multimodal integration. Besides, MAN achieves the best performances on the metrics among the existing multimodal integration methods TFN, LMF and gains large margin. Since TFN, LMF neglect the fine-grained temporal interaction which includes rich structured information for multimodal modeling. Particularly, the performance of MAN decreases the MAE from 0.912 0.894 compared to the best counterparts. As for the Transformer-basel methods, MAN also performs better than MulT on all the metrics, which demonstrates that, to some extent, subsequent multimodal joint representation learning could be integrated with the carly multimodal interaction, even though only multimodal interaction has been able achieve competitive performances. In general, the best performances of MAN attribute the advanced hierarchical approximation decomposition which imposes fine-grained temporal correlation with acceptable computational complexity, as well as the addition random features mechanism that provides support for the approximation decomposition. Table shows the experimental results of different methods on speaker traits recognition dataset POM, where we report the multi-class accuracy of all the traits. The similar observation could be found from the table, MAN achieves competitive performances compared with both Transformer-be of the traits. The only difference is that the metrics of Transformer- based and Non-Tr Transformer-baat
and Non- Transformer-base methods on most

---

methods are close. Therefore, even the Transformer has great potential in multimodal learning, its ability may be limited by the number of training samples. We argue that Non- Ten---raffomr-bbiiii methods still have their meanings in the real-world applications.
0.70
31.
31.0
37.6
0.80
37.6
0.40
38.0
MAN
355
MAN
Wibh H1LL
w/ HAD
0.70
35.6
20
28 25
Num. Features
Num. Fwahires
Figure 2: The evaluation of different numbers of random features, where we report the values of Corr and MA CMU-MOSI.
We set some control experiments on CMU-MOSI to verify the effectiveness of MAN and the results are shown in Table and Table A where "w/o CAO" denotes the model without combinational addition operation in Eq. 7 we simply employ the continued Hadamard product and do not decompose the attention map, this way, the local constraints also cannot be utilized. "w/o HAD" denotes the model without hierarchical approximation decomposition, but with other contributions. "w/o LSC" denotes the model with all the contributions except for local sequential constraints. We could observe that "w/o HAD" performs similarly to MAN with much more computational complexity, since "w/o HAD" needs to generate a high-order attention map with exponential complexity. Besides, MAN and "w/o HAD" perform much better than "w/o CAO", since "w/o CAO" generates denser attention distributions and cannot capture really important information. Additionally, by comparing the metrics of "w/o LSC" and MAN, we argue that the local constraints have impacts on better capturing the correlation between multimodal sequential features. Since we utilize addition random features mechanism to approximate the softmax function, the number of random features should be considered. We examine the performances of MAN Non-Transformer) on CMU-MOSI with different values of H. As shown in Fig. 2 when the value of H is big enough, MAN achieves competitive performances of Corr and MA close to "w/o HAD".
Ablation Study:
3.3 Experiments for Multimodal Video Retrieval
Data Preprocessing:
The videos contain abundant multimodal information. Thus, we use multiple pre-trained models for extracting features. Concretely, we utilize following seven experts: Motion
embeddings are extracted from S3D 32) trained on the kinetics dataset. Scene embeddings are
extracted with DenseNet-166 [TO trained on the Places365 dataset [42]. OCR embeddings are extracted in three stages. First, the pixel link text detection model is used to detect the overlaid text. Then, the detected boxes are passed through the text recognition model. Finally, each character sequence is encoded using a word2vec embedding. Audio embeddings are obtained with a VGGish model, trained on the YouTube-SSl 8m dataset. Speech features are extracted using the Google Cloud speech API, to extract word tokens from the audio stream, which are then encoded via pre-trained word2vec embeddings [22]. Face features are extracted by ResNet-50 g trained for face classifi- cation on the VGGFace2 dataset. Appearance features are extracted from the final global average pooling layer of SENet- 154 9 trained on ImageNet.
Experimental Details: We also combine MAN with Transformer-based and Non-Transformer--bsse backbones. For MAN -Non-Tranfformn
each word of the text query is encoded with pre-trained word2vec embeddings and then passed through the GPT model. For MAN (Transformer), we adopt the basic structure of [6] and train the word vectors from scratch. We refer to [18] and (6) for the implementation of backbones and keep the original structures, only adding MAN an extra module. We further fuse their results. The hyperparameters of MAN include Adam learning rate 5 x 10-5, which we decay by multiplicative factor 0.95 every 1000 optimization steps, the structure of integration network (1 layer of integration block, with hidden size 512, number of heads 8, number of random features 512). "Hidden size" denotes the common size of d1, d2.....m.. We divide the time steps into 10 chunks and employ local sequential constraints. We also use the same hyperparameters of specific backbones as the original papers.
8

---

Text
Video
Video
Text
Model Metric
R@ 1t
R@51 51
@@ 101
MdR
R@ It
R@ 11
R 101
MdRl
JSFusion (34
10.2
31.2
43.2
13
HT [11
14.9
40.2
52.8
9
CE [8
20.9
48.8
62.4
6
20.6
50.3
64.0
5.3
MAN Non-Transformett
21.4
51.0
63.5
5
21.6
51.3
65.4
5
MMT 66
24.6
54.0
67.1
4
24.4
56.0
67.8
4
MAN (Transformer)
24.1
55.7
68.1
4
24.4
57.1
68.5
4
Table 5: Retrieval performances on the MSR-VTT dataset.
Text
Video
Video - Text
Method Metric
R@ 11
R@ 5t
R@ 10
MdR4
R@ 11
R@51
R@ 101 MdR4
CT-SAN 35
5.1
16.3
25.2
46
JSFusion
9.1
21.2
34.1
36
CCA 16
rrep. by (00)
7.5
21.7
31.0
33
MEE 20
10.1
25.6
34.6
27
CE 18
11.2
26.9
34.8
25.3
MAN (on-Transformer)
11.8
27.9
36.0
24
MMT
13.2
29.2
38.8
21
12.1
29.3
37.9
22.5
MAN (Transformer)
13.6
30.0
39.4
20
12.5
29.8
38.7
22
Table 6: Retrieval performances on the LSMDC dataset.
Results: We report the evaluation results of MAN and the competing text-video retrieval methods on MSR-VTT (Table and LSMDC O In order to be fair, we still divide all the methods into two categories, Non-Transformer-baasee and Transformer-basee methods. for the former, MAN outperforms existing caate-of-the-ar methods in all the metrics on both MSR-VTT VTT and LSMDC Benefiting from the fine-grained temporal correlation different modalities, the final representation contains not only the global multimodal information, but also local temporal interaction information. When we replace the backbone with Transformer, the metrics of MMT and MAN (Transformer) very close, in other words, the effect of MAN is diminished. It may be because that the sample number MSR-VTT and LSMDC is sufficient to stimulate the potential of Transformer.
4 Related Work
Bilinearr Model. proposes multimodal compact bilinear pooling network (MCB), which calculates the outer product between the visual and textual embeddings. The count-sketch projection is utilized project the product on lower-dimensional space. [15] proposes multimodal low-rank bilinear pooling network (MLB), where the full bilinear interaction between image and question spaces are parametrized by a tensor. To limit the number of free parameters, this tensor is constrained to be of low rank [1] proposes multimodal tucker network (MUTAN), which reduces the size of mono-modal embeddings while modeling their interaction as accurately as possible with full bilinear integration scheme. [14] proposes bilinear attention network (BAN) to utilize the given vision-language information seamlessly by finding the bilinear attention distributions. Concretely, BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair channels. Based on BAN, [2] and [17] propose relational reasoning by constructing iinengrained graph between different input channels. 12 also tries to decompose the temporal projection tensor, but without theoretical support.
Non-Linear Approximation. Many endeavors have been devoted to the approximation of non-linear function. [3 employs a generalizable statistic expectation to approximate the non-linear kernels, specifically, [3] first defines a basic transformation (approximation of simply dot-producty and then utilizes Maclaurin expansion to expand out the non-linear functions to employ conclusion of basic transformation. Such approximation accumulates two approximate errors. [26] proposes Tensor Sketching to approximate polynomial kernel functions based on Fourier transform. develops pair- level integration of two vectors by leveraging the conclusions of [3] and (26), such method achieves

---

superior results on visual classification. Recently, 3 and [23] introduce the kernel decomposition into Transformer, they mainly approximate the exponential kernel with statistic expectation. The existing kernel decomposition methods are all proposed to process two vectors, considering the multimodal tasks, we first try to generalize the decomposition methods to more objects.
Conclusion
We propose generalizable method called multi-linear attention network (MAN) to integrate many modalities as possible. Compared with the existing multimodal joint representation learning methods, MAN conducts the hierarchical approximation decomposition (HAD) to reduce the exponential computational complexity. Considering that softmax function cannot decomposed directly, we propose the addition random features mechanism (ARF) to reconstruct the non-linear operation with orthogonal projections. Furthermore, we employ local sequential constraints (LSC) to suppress the association scores of features that are virtually irrelevant. In the future, we would focus on how to combine powerful interaction methods (i.e. Transformer) with joint representation learning methods (i.e. MAN) more seamlessly.
Acknowledgments
This work was supported in part by the National Key R&D Program of China under Grant No. No.2020YFC0832505. National Natural Science Foundation of China under Grant No.62072397 and Zhejiang Natural Science Foundation under Grant LR19F020006.
References
[1] Hedi tucker fusion for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2612-2620, 2017.
Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal
[2] Remi Cadene, Hedi Ben- Younes, Matthieu Cord, and Nicolas Thome. Murel: Multimodal relational reasoning for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1989-1998, 2019.
[3] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint 24794 2020.
[4] Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. Covarep collaborative voice analysis repository for speech technologies. ICASSP. 2014.
[5] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Mareus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv: 1606.006000 01847, 2016.
[6] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In European Conference on Computer Vision (ECCV), volume 5. Springer, 2020.
[7] Richard A Harshman et al. Foundations of the parafac procedure: Models and conditions for an" explanatory" multimodal factor analysis. 1970.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[9] Jie Hu, Li Shen, and Gang Sun. anxxxetttannnexiiiiiiioi networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7132-7141, 2018
[10] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708 2017.
10

---

[11] iMotions. Facial expression analysis. 2017.
[12] Tao Jin, Siyu Huang, Yingming Li, and Zhongfei Zhang. Dual low-rank multimodal fusion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 377-387, 2020.
[13] Purushottam Kar and Harish Karnick. Random feature maps for dot product kernels. In Artificial intelligence and statistics, pages 583-591 PMLR, 2012.
[14] Jin-Hwa Kim, Jaehyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. arXiv preprint arXiv:1805.07932. 2018.
[15] Jin-Hwa Kim, Kyoung- Woon On, Woosang Lim, Jeonghee Kim, Jung- Woo Ha, and Byoung- Tak Zhang. Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.0422,, 2016.
[16] Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Associating neural word embeddings with deep image representations using fisher vectors. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4437-444 2015.
[17] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10313-10322, 2019.
[18] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv: /907. 13487, 2019.
[19] Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. In ACL, 2018.
[20] Antoine Miech, Ivan Laptev, and Josef Sivic. Learning text- video embedding from incomplete and heterogeneous data. arXiv preprint /=Xvv0/00.002/// 2018.
[21] Antoine Miech, Dimitri Zhukov, Dean-BBatiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Hlwtol00mg Learning text-video embedding by watching hundred million narrated video clips. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2630-26400, 2019.
[22] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 300.311 2013.
[23] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. arXiv preprint arXiv:2103.0214: 2021.
[24] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[25) Verónica Porez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. Utterance-level multimodal sentiment analysis. ACI., 2013.
[26) Ninh Pham and Rasmus Pagh. Fast and scalable polynomial kernels via explicit feature maps. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 239-247, 2013.
[27] Shyam Sundar Rajagopalan, Louis-Philippe Philippe Morency, Tadas Baltrusaitis, and Roland Goecke. Extending long short-term memory for multi-view structured learning. In ECCV, 2016.
[28] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3202-3212, 2015.
11

---

[29] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Public Access, 2019.
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint N.XY::1706.037666 2017.
[31] Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7216-7223, 2019.
[32] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spa- tiotemporal feature learning: ercerracc trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 305-321, 2018.
[33] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1288-52966 2016.
[34] Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In Proceedings of the European Conference Computer Vision (ECCV), pages 471-487, 2018.
[35] Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. End-to-end concept word detection for video captioning. retrieval, and question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3165-3177 2017.
[36] Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In ICCV, 2017.
[37] Jiahong Yuan and Mark Liberman. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America, 2008.
[38] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In EMNLP, 2017.
[39] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis- Philippe Moreney. Memory fusion network for multi-view sequential learning. In AAAI, 2018.
[40] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij. Erik Cambria, and Louis Phiil-Philippe Morency. Multi-attention recurrent network for human communication comprchension. In AAAI, 2018.
[41] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint 1661v:1606.062559 2016.
[42] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 4066):1552-144, 2017.
12


Reviews:
[{"review_id": "rD1_y_KM5iR", "reviewer": "Reviewer_G3yM", "summary": "This paper extends the bilinear attention network (BAN) to process more modalities. Through production operation, the proposed model can accept multiple modalities as input. Then the authors utilize CAO, LSC, HAD and ARF to optimize the multi-linear model. This model achieves some great results on several tasks. However, there still remain some problems.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. In this paper, the approximation of softmax seems a little interesting.\n2. In experiments, the proposed model can derive some good results.\n\nCons:\n1. The fusing method is not really novel for the reason that the fundamental idea is BAN and utilizing a high-order tensor to mix the modalities has been proposed, like Mutan[1].\n2. On the softmax approximation:\n    (a) Although I can find some novelty in the softmax approximation, I do not understand why the authors have to decompose softmax(A) rather than A. The model compression and flop reduction come from decomposition on A. So what is the advantage of decomposing softmax(A)?\n    (b) It can be better to add more details on the softmax approximation. Specifically, this approximation can not replace the exp operation, so what is the concrete effect of the approximation? And how this approximation can help with decomposition?\n    (c) In the ablation experiment, this paper discussed the influence of the random feature numbers on performance. However, it is weird that the performance of MAN decreases at about number 33 in the left figure and about 29 in the right figure. This phenomenon can be a fatal instability of MAN. Can the authors explain the reason? In addition, as an approximation, the analysis of how the random feature numbers influence the similarity between ARF and w/o HAD is necessary.\n    (d) Will the feature number influence the time consumption?\n3. The proposed model does not improve appear on the Transformer comparison.\n\n[1]Ben-Younes, Hedi, et al. \"Mutan: Multimodal tucker fusion for visual question answering.\" Proceedings of the IEEE international conference on computer vision. 2017.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "idH2yoT7qa_", "reviewer": "Reviewer_62Vh", "summary": "Motivated by the high computational cost of multi-modal multi-linear pooling, this paper proposed an efficient attention based multi-modal fusion mechanism. Low-rank bilinear pooling [1] is expended to multi-modal multi-linear case, following with the Multi-Linear Attention Network (MAN). The MAN combines cross modal attention and temporal feature, which is also decomposed based on Performers. Experiments show that the FLOPs decrease with a stable performance on multimodal down-stream task.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed Multi-Linear Attention Network (MAN) gets noticeable experimental results on various of multimodal tasks, including Multimodal Sentiment Analysis, Speaker Traits Recognition and Video Retrieval. Both transformer-free backbone and transformer-based backbone can benefit from this generalizable method. However, it would be better if some detailed issues can be settled as follow:\n\nIn the Approach section, the decomposition of bilinear pooling and efficient multi-modal attention are widely explored in the past few years. This paper combined the well-known techniques [2, 3, 4, 5]. However, the task is novel to some extent, the author expands these methods to multi-modal case and combine them effectively. So, it would be better if the proposed MAN can be described more explicit. Specifically, how the multi-linear attention map works cross modalities and time chunks, or how the Hierarchical Approximation Decomposition decrease the FLOPs. What’s more, the differences between MAN and Performers should be explained more clearly for novelty. It’s not clear how this work differs from previous contributions, but the related works are adequately cited.\n\nIn the Related Work section, only some mathematical works are cited. However, the proposed MAN is proposed to applied in multi-modal tasks, so, it would be better if some multimodal works are cited. In particular, the audio-visual-text task--Video-Audio Scene-Aware Dialog, is mostly related to this work, so some related works can be included [7,8], similar with Vide Retrieval [9,10].\n\nIn the Experiments section, the proposed MAN gets great performance, and it may be better if some other experimental results are included. Firstly, the paper is focus on multi-modal multi-linear model, and the computational cost is related to not only the number of modality but also the length of the modality sequence. So, in my opinion, the experiments would be better if the discussion of sequence length is included as [2]. Similarly, the performance may not be promised when the sequence length is small. Secondly, in the Ablation Study subsection, only two tables are given to show the performance and the FLOPs, however, the discussion about the time cost is not describe clearly, so it would be better if this issue is added to the Ablation Study. Thirdly, the sequence attention gets great improvement in the submission, to get temporal context in a multimodal sequence, what are the advantages of your proposed method LSC compared with early temporal extraction, such as using temporal backbone [6, 7]. It may be more abundant if this point is discussed.\n\nIn general, the submission is clearly written with great experimental performance, the claims are well supported with enough theoretical analysis, but the novelty of it is limited. It would be more convincing if the Ablation Study is expended. The methods used is appropriate and efficient and the work is a complete work, which can be applied as an additional module to most multimodal fusion structure. And others are likely to use the ideas or build on them. The submission addresses a difficult task in a better way than previous work and advances the state of the art in a demonstrate way.\n\nPost Rebuttal:\n\nThe authors' response has well addressed my previous concerns, especially the clarification of multi-linear attention map and hierarchical approximation decomposition and the differences between MAN and Performers. Moreover, the authors also accordingly provide some new experimental results w.r.t. my previous comments. Such clarification and results has further strengthened the core contribution of this paper. Hence, I decide to raise my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author mostly evaluated the strengths but rarely analyze the weaknesses of their work such as what if the length of the modality sequence is too short, in my opinion, shorter sequence length may influence the performance of the HAD part.\n\nAccording to the list of potential negative societal impacts provided in Ethics Guidelines, there is no potential negative societal impact in this submission.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "EX6HEv7RaFR", "reviewer": "Reviewer_ERDD", "summary": "For efficient multimodal learning having more than two modalities, they generalized the bilinear attention networks (Kim et al., 2018) to multi-linear attention networks using the same gist of low-rank pooling (Kim et al., 2017) and the sampling-based softmax approximation (Choromanski et al., 2021). To reduce the computational cost to build a multi-linear attention tensor, the proposed sampling-based approach was a novel idea using the well-established softmax approximation in the transformer-based architectures. Their extensive experiments on four datasets of three different tasks confirm that the proposed method consistently outperforms the comparative methods while giving the ablation study to assess the sub-modules one-by-one and the impact of the number of samples used in the softmax approximation.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this work is well-written to deal with an interesting topic, more-than-two modality learning, in an efficient manner. The idea used in this approach is straightforward but having clever handling of the computational cost, which is one of the major concerns in practical multimodal learning. I believe the manuscript can be improved further by considering some points below:\n\n### Clarity\n\nc1) Please clarify the proposed method, \"combinational addition operation (CAO) to replace the Hadamard product.\" (L109) I think it deserves to be more specifically elaborated. \n\nc2) The $v_j$ in Eqn. 11 should be defined to prevent possible misleading.\n\nc3) Could you compare with the other methods in terms of the number of parameters, FLOPS (as in Table 4), and inference time? Especially, the proposed method uses sampling to approximate the softmax operation. How much does it impact the computational time compared with the other methods or the controlled method which does not use the sampling? \n\nc4) In the Performers, they said their random samples should be periodically redrawn for stable training (Choromanski et al., 2021). Doesn't it affect multimodal training?\n\n\n### Suggestion\n\ns1) The temporal order of Multimodal Temporal Features. The proposed one is not considering the temporal order. Why don't you use the two features, $(t_1, -1)$ and $(1, t_2)$, where the dot-product results in $t_1 - t_2$, to consider the temporal order instead of the absolute difference of $t_1$ and $t_2$ in L123?\n\ns2) Could you add a paragraph dedicated to the methods using softmax approximation in Related Work?\n\ns3) Please explicitly summarize the limitation of this work, which may include the time and space complexities as described in the comment of c3.\n\n\n### Minors\n\nm1) why do you call it \"addition random features (ARF)\"\n\nm2) L71, rewrited -> rewritten\n\nm3) L72,81, the rank is \"at most\" d, not the value of rank if U or V is not full rank.\n\nm4) L104, \"...employing softmax function may result in even attention distributions when the attention map is large.\" -> could you elaborate this sentence on what you want to mean by this?\n\n----------\nI willingly defend the paper and hold my score of 8. I do not see the idea as incremental from BAN or MUTAN. For the more-than-two modality learning, the computational cost of high-order tensor is prohibitive, aside from the theoretical application of previous bimodal methods. Because of that, I believe the novelty of this work from the combination of different but appropriate strategies to achieve the goal of *multimodal* learning, providing extensive empirical validations. However, it can be seen as having a limitation in a theoretical viewpoint, possibly due to relying on the previous core works.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the suggestion in the main review. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are very positive about the paper and is clearly an accept (6,7,8). Perhaps the nature of the multimodal tasks that are chosen are slightly contrived making it not quite worth of a Spotlight.
Decision: Accept (Poster)