Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Motivated by the high computational cost of multi-modal multi-linear pooling, this paper proposed an efficient attention based multi-modal fusion mechanism. Low-rank bilinear pooling [1] is expended to multi-modal multi-linear case, following with the Multi-Linear Attention Network (MAN). The MAN combines cross modal attention and temporal feature, which is also decomposed based on Performers. Experiments show that the FLOPs decrease with a stable performance on multimodal down-stream task.
main_review: The proposed Multi-Linear Attention Network (MAN) gets noticeable experimental results on various of multimodal tasks, including Multimodal Sentiment Analysis, Speaker Traits Recognition and Video Retrieval. Both transformer-free backbone and transformer-based backbone can benefit from this generalizable method. However, it would be better if some detailed issues can be settled as follow:

In the Approach section, the decomposition of bilinear pooling and efficient multi-modal attention are widely explored in the past few years. This paper combined the well-known techniques [2, 3, 4, 5]. However, the task is novel to some extent, the author expands these methods to multi-modal case and combine them effectively. So, it would be better if the proposed MAN can be described more explicit. Specifically, how the multi-linear attention map works cross modalities and time chunks, or how the Hierarchical Approximation Decomposition decrease the FLOPs. What’s more, the differences between MAN and Performers should be explained more clearly for novelty. It’s not clear how this work differs from previous contributions, but the related works are adequately cited.

In the Related Work section, only some mathematical works are cited. However, the proposed MAN is proposed to applied in multi-modal tasks, so, it would be better if some multimodal works are cited. In particular, the audio-visual-text task--Video-Audio Scene-Aware Dialog, is mostly related to this work, so some related works can be included [7,8], similar with Vide Retrieval [9,10].

In the Experiments section, the proposed MAN gets great performance, and it may be better if some other experimental results are included. Firstly, the paper is focus on multi-modal multi-linear model, and the computational cost is related to not only the number of modality but also the length of the modality sequence. So, in my opinion, the experiments would be better if the discussion of sequence length is included as [2]. Similarly, the performance may not be promised when the sequence length is small. Secondly, in the Ablation Study subsection, only two tables are given to show the performance and the FLOPs, however, the discussion about the time cost is not describe clearly, so it would be better if this issue is added to the Ablation Study. Thirdly, the sequence attention gets great improvement in the submission, to get temporal context in a multimodal sequence, what are the advantages of your proposed method LSC compared with early temporal extraction, such as using temporal backbone [6, 7]. It may be more abundant if this point is discussed.

In general, the submission is clearly written with great experimental performance, the claims are well supported with enough theoretical analysis, but the novelty of it is limited. It would be more convincing if the Ablation Study is expended. The methods used is appropriate and efficient and the work is a complete work, which can be applied as an additional module to most multimodal fusion structure. And others are likely to use the ideas or build on them. The submission addresses a difficult task in a better way than previous work and advances the state of the art in a demonstrate way.

Post Rebuttal:

The authors' response has well addressed my previous concerns, especially the clarification of multi-linear attention map and hierarchical approximation decomposition and the differences between MAN and Performers. Moreover, the authors also accordingly provide some new experimental results w.r.t. my previous comments. Such clarification and results has further strengthened the core contribution of this paper. Hence, I decide to raise my score to 7.
limitations_and_societal_impact: The author mostly evaluated the strengths but rarely analyze the weaknesses of their work such as what if the length of the modality sequence is too short, in my opinion, shorter sequence length may influence the performance of the HAD part.

According to the list of potential negative societal impacts provided in Ethics Guidelines, there is no potential negative societal impact in this submission.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper extends the bilinear attention network (BAN) to process more modalities. Through production operation, the proposed model can accept multiple modalities as input. Then the authors utilize CAO, LSC, HAD and ARF to optimize the multi-linear model. This model achieves some great results on several tasks. However, there still remain some problems.
main_review: Pros:
1. In this paper, the approximation of softmax seems a little interesting.
2. In experiments, the proposed model can derive some good results.

Cons:
1. The fusing method is not really novel for the reason that the fundamental idea is BAN and utilizing a high-order tensor to mix the modalities has been proposed, like Mutan[1].
2. On the softmax approximation:
    (a) Although I can find some novelty in the softmax approximation, I do not understand why the authors have to decompose softmax(A) rather than A. The model compression and flop reduction come from decomposition on A. So what is the advantage of decomposing softmax(A)?
    (b) It can be better to add more details on the softmax approximation. Specifically, this approximation can not replace the exp operation, so what is the concrete effect of the approximation? And how this approximation can help with decomposition?
    (c) In the ablation experiment, this paper discussed the influence of the random feature numbers on performance. However, it is weird that the performance of MAN decreases at about number 33 in the left figure and about 29 in the right figure. This phenomenon can be a fatal instability of MAN. Can the authors explain the reason? In addition, as an approximation, the analysis of how the random feature numbers influence the similarity between ARF and w/o HAD is necessary.
    (d) Will the feature number influence the time consumption?
3. The proposed model does not improve appear on the Transformer comparison.

[1]Ben-Younes, Hedi, et al. "Mutan: Multimodal tucker fusion for visual question answering." Proceedings of the IEEE international conference on computer vision. 2017.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: For efficient multimodal learning having more than two modalities, they generalized the bilinear attention networks (Kim et al., 2018) to multi-linear attention networks using the same gist of low-rank pooling (Kim et al., 2017) and the sampling-based softmax approximation (Choromanski et al., 2021). To reduce the computational cost to build a multi-linear attention tensor, the proposed sampling-based approach was a novel idea using the well-established softmax approximation in the transformer-based architectures. Their extensive experiments on four datasets of three different tasks confirm that the proposed method consistently outperforms the comparative methods while giving the ablation study to assess the sub-modules one-by-one and the impact of the number of samples used in the softmax approximation.
main_review: Overall, this work is well-written to deal with an interesting topic, more-than-two modality learning, in an efficient manner. The idea used in this approach is straightforward but having clever handling of the computational cost, which is one of the major concerns in practical multimodal learning. I believe the manuscript can be improved further by considering some points below:

### Clarity

c1) Please clarify the proposed method, "combinational addition operation (CAO) to replace the Hadamard product." (L109) I think it deserves to be more specifically elaborated. 

c2) The $v_j$ in Eqn. 11 should be defined to prevent possible misleading.

c3) Could you compare with the other methods in terms of the number of parameters, FLOPS (as in Table 4), and inference time? Especially, the proposed method uses sampling to approximate the softmax operation. How much does it impact the computational time compared with the other methods or the controlled method which does not use the sampling? 

c4) In the Performers, they said their random samples should be periodically redrawn for stable training (Choromanski et al., 2021). Doesn't it affect multimodal training?


### Suggestion

s1) The temporal order of Multimodal Temporal Features. The proposed one is not considering the temporal order. Why don't you use the two features, $(t_1, -1)$ and $(1, t_2)$, where the dot-product results in $t_1 - t_2$, to consider the temporal order instead of the absolute difference of $t_1$ and $t_2$ in L123?

s2) Could you add a paragraph dedicated to the methods using softmax approximation in Related Work?

s3) Please explicitly summarize the limitation of this work, which may include the time and space complexities as described in the comment of c3.


### Minors

m1) why do you call it "addition random features (ARF)"

m2) L71, rewrited -> rewritten

m3) L72,81, the rank is "at most" d, not the value of rank if U or V is not full rank.

m4) L104, "...employing softmax function may result in even attention distributions when the attention map is large." -> could you elaborate this sentence on what you want to mean by this?

----------
I willingly defend the paper and hold my score of 8. I do not see the idea as incremental from BAN or MUTAN. For the more-than-two modality learning, the computational cost of high-order tensor is prohibitive, aside from the theoretical application of previous bimodal methods. Because of that, I believe the novelty of this work from the combination of different but appropriate strategies to achieve the goal of *multimodal* learning, providing extensive empirical validations. However, it can be seen as having a limitation in a theoretical viewpoint, possibly due to relying on the previous core works.
limitations_and_societal_impact: Please refer to the suggestion in the main review.
review_title: null

Meta-review: The reviewers are very positive about the paper and is clearly an accept (6,7,8). Perhaps the nature of the multimodal tasks that are chosen are slightly contrived making it not quite worth of a Spotlight.
Decision: Accept (Poster)