Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on multi-modal multi-task learning. They customize Mixture-of-Experts into the transformer layer to do efficient MTL. They achieve good performance on the HighMMT dataset. 
strength_and_weaknesses: My major concern with the paper is the technical novelty. I will illustrate the issue from three aspects: 1. Using MoE to do multi-modality is not new. 2. The technique it uses (MoE attn and MoE Mlp) is exactly the same as previous work. Multi-router MoE is more like a clever trick rather than a solid technical contribution. 3. No much interesting finding in the paper.


1. Using MoE to do multi-modality is not new.
[1, 3] scale up on big model and data and also use MoE mlp to do efficient learning. Both text modality and vision modality have been explored. [2] also, focus on multi-modality and also use experts for each modality. [2] is not exactly the MoE, but the idea that using different experts for different modalities is not new.

2. The technique it uses (MoE attn and MoE Mlp) is exactly the same as previous work. 
MoE MLP is exactly the same as in [1,3]  and many other MoE papers. MoE Attn directly applies moe on qkv in the attention layer, which has been explored by the switch transformer[4]. (in their Tabel 10) and also mention by [1,3] before.  Multi-router MoE is more like a clever trick rather than a solid technical contribution. 
So I don't think there is enough technical contribution in this paper.

3. Since the technical novelty is restricted, I would expect more interesting findings in experiments. However, it occurs to me that the experiment part is mainly about the performance gain and I don't find some interesting conclusion/insight. Perhaps the writers can illustrate more about that in the rebuttal.

Overall, the paper is more like a direct implementation of some known technique. So I would keep a negative rating for now.

[1] Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
[2] VL-BEIT: Generative Vision-Language Pretraining
[3] Scaling Vision with Sparse Mixture of Experts
[4] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity




-------------------AFTER REBUTTAL-----------------------

I appreciate the detailed response from the authors. I also carefully read the comment from other reviewers.
However, I still hold my original opinion that the paper needs to improve its technical novelty to be accepted as a conference paper.

1. Multi-modality multi-task setting.
Both authors and I agree that there are some similar settings in previous papers. I appreciate the author giving a detailed explanation of the differences. However, these are minor differences to me and can not convince me of a major contribution.

2. MoE Mlp.
Both authors and I agree it is the same as previous work.

3. MoE Attention and multi-router.
The authors explain the difference in rebuttal. I believe some simple techniques could make huge differences but in this model, these contributions are more likely to be incremental rather than a game changers. 

4. Insight and some interesting finding on the new task setting with moe.
These parts are missing from the rebuttal.

Therefore, i would keep my rating.

clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: See above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents M^3SAT, a sparse transformer model based on mixture-of-experts (MOE) for multimodal multi-task learning. Sequences of tokens from different modalities are concatenated together and processed by transformer-based encoder layers that incorporate MOE in the attention and FFN modules. A different decoding head produces the outputs for each task. The method is shown to outperform HighMMT on the MultiBench tasks with less computational cost.
strength_and_weaknesses: Strengths: 
1) The paper is clearly written and shows strong quantitative results on the MultiBench tasks for M^3SAT compared to HighMMT for less computational cost.
2) The authors also perform thorough ablation studies where they demonstrate the contribution of their proposed encoder.
3) The authors analyze metrics such as gradient positive sign purity, intertask affinity and optimal gradient blending to support their claims that the method alleviates some of the challenges of multi-task and multimodal learning.

Weaknesses: 
1) For inputs such as videos and audio data that have a temporal component, the feature fusion can be performed along the temporal axis (and is usually done so for many tasks such as speech recognition). Could the authors comment on the decision to instead fuse modalities by concatenating their tokens along the sequence axis, as shown in the Figure 1? Doesn't this remove the temporal correspondence between modalities? Maybe this has a larger impact on tasks other than those in MultiBench.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and high quality. Transformer-based MOEs are not novel from a technical standpoint, but the application is effective and the results are convincing. Authors state they will release code upon acceptance.
summary_of_the_review: Overall, I think this is a good paper that advances the field of multimodal multitask learning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The proposed work aims to tackle the multimodal multitask optimization problem with sparsely activated mixture-of-experts models where both self-attention and FFN networks are implemented as routed expert networks. This implementation achieves better disentanglement of parameters which reduces overfitting for modalities that are "simple" compared to other modalities in a task. In addition, these models outperform dense counterparts in performance while have lower computation(FLOPS) on several benchmarks.
strength_and_weaknesses: ### Strengths
- The improvements across the various tasks on MultiBench clearly shows the effectiveness of $M^3SAT$. 
- Authors perform extensive ablation study comparing various routing mechanisms and mixture of expert networks types.
- From Table 3 ablations, the mixture-of-experts implementation on self-attention clearly shows significant improvements on the evaluated tasks. This is very interesting as standard MoE architectures apply experts to FFN layer and I am curious how this would apply to other MTL problems like MT which have one modality only.
- Authors ablate thoroughly the different routing mechanisms and it is conclusive from these results that single routing is effective for this setup and tasks.


### Weaknesses
- It is not clear to me why those specific 7 tasks were chosen among 20 tasks from MultiBench.
- When comparing MoE model with a dense model with similar computation(FLOPs), it is unclear whether the gains are coming from capacity increase or due to efficient parameter space separation due to different experts specializing in different tasks/modalities. One important comparison to disentangle these factors will be comparing MoE with dense models with similar capacity. Of course, the counterargument that MoE can provide larger capacity with lower FLOPs is valid, but for better understanding of where the improvements are coming from it is important to compare similar capacity models. 
- From Table 2, in the `Large setting` of the tasks, the improvements are not as significant as `Medium setting`. It is also unclear how the model capacities are chosen for these different settings. Were the models optimized for a particular setting through sweeping over different values of $N$ and top-$K$? Which other hyper-parameters are significant for these choices?
- Given that most of the ablations for MTL are done on the `Medium Setting` tasks, where the improvements are significant for the ‘ENRICO’ task, it is unclear whether $M^3SAT$ improves across all types of tasks and settings, and how well it generalizes. For example in Table 4, other tasks like PUSH and AV-MNIST do not show any improvements to the overfitting problem.
clarity,_quality,_novelty_and_reproducibility: The approach is very clearly described, but some details about training are not very clear to me. In terms of novelty, application of MOE to self-attention to MTL is novel. Reproducibility might be a bit difficult as there is very less information about how different model hyper-parameters are chosen and I encourage the authors to share more details about the models used for the three types of evaluation settings.
summary_of_the_review: Overall, the proposed method is interesting as applying converting self-attention MOE layers can bring improvements to different types of MTL problems. The method works for some tasks in a MTL setting quite well. However, some parts of the claims are not very clear and how well they will generalize to different tasks/modalities setting is unclear. 

Looking forward to the discussion with the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes $M^3SAT$ for the multi-modal multi-task learning task, which tailors the mixture-of-experts (MoEs) into both the self-attention and the feed-forward networks (FFN) of a transformer backbone. $M^3SAT$ can prevent training conflicts among diverse modalities and tasks, and restrains the problem of simple modal prone to overfitting.
strength_and_weaknesses: 1. In the multi-modal multi-task learning, authors propose a multi-modal multi-task mixture of expert model, which solves the training conflicts among tasks and restrains the easy modality from overfitting.
2. The proposed $M^3SAT$ has the remarkable improvement in performance and computational cost.

clarity,_quality,_novelty_and_reproducibility: 1. Recently, there are some works in multi-modal multi-task learning that also uses mixture-of-experts (MoEs) in the self-attention layers and FFN layers, such as Uni-Perceiver-MoE. The novelty of this paper needs to be clearly stated.
2. Please further explain the difference between modality-specific router and task-specific router as mentioned in the paper. Compared to task-specific router, is modal-specific router more effective for the overfitting problem in multi-modal learning? Compared to modal-specific router, is task-specific router more effective for the gradient conflict problem in multi-task learning? In addition, is it possible to use both modal-specific router and task-specific router in the FFN layer or the self-attention layer?
3. In the $M^3SAT$, whether there is a linear correlation between the number of layers using MOE and model performance？
summary_of_the_review: It is useful to reduce computational costs. The authors should further state the novelty of the proposed method and provide more details.

-------------------AFTER REBUTTAL-----------------------

Thanks for the response, my concern has been partially addressed. However, I think the novelty of the paper needs further improvement. 
1. I appreciate the authors for explaining in detail the differences between this work and Uni-Perceptionr-MoE. However, these differences are not particularly innovative and do not qualify as a major contribution.
2. We note that compared to modality-specific routers, task-specific routers are not more effective for the gradient conflict problem in multi-task learning. I think the interpretability of modality-specific routers and task-specific router still needs further clarification.

Therefore, I would keep my rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a transformber based model using mixture-of-experts for multimodal multi-task learning. The reviewers generally agree with there are performance gains and that the ablation studies are convincing. However, the reviewers also concern the improvement compared to the baseline is not so significant. Even Reviewer yAsT gives positive feedback, this remains to be an issue. The other reviewers also find there is a limited technical contribution for the paper. The AC agrees with the majority of the reviewers on above issues and recommends rejecting the paper.
Decision: Reject