Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents an architectural improvement for attention based large scale task models. They start from the uni-perceiver, an attention based model to learn multimodal representations that is amenable to few-shot generalization via a prompting mechanism. The uni-perceiver has a token based modality conditioning and they 1) they introduce more flexible conditioning and 2) use that conditioning as inputs to a gating mechanism applied to the weights attention and MLPs in the uni-perceiver model. This is motivated by insightful but unsurprising experiments showing that in multi-task learning gradients coming from different tasks can have opposite directions thus specializing weights using a gating mechanism is meant to mitigate this effect. They show a broad set of experiments to demonstrate benefits of the new conditioning and the pretraining and generalization benefits of the approach. 
strengths_and_weaknesses: strenghts:
- the CMoE idea and validation seems well executed
- the attribute routing seems reasonable and novel

weaknesses:
- some details are missing (see below).
questions: - the details on the gating could be improved. how do you train the gating ? is this more expensive in training than in inference ? Do you find collapse there or spurious gating solutions ? do you need to do something prevent these solutions ?

- how many experts are you using ? There must be a tradeoff between the number of experts and optimization ie with many experts less gradients are propagated through the other weights. do you have any insights that might help the readers ?
limitations: yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes several MoE routing strategies, i.e., token-level, context-level, task-level, modality-level and attribute-level on top of uni-perceiver, to mitigate task interference, showing noticeable improvements. 
 
strengths_and_weaknesses: \+ Different routing strategies, i.e., token-level, context-level, task-level, modality-level and attribute-level is evaluated. These strategies, though are quite common in their forms, are the first time being evaluated and proved effective in vision-language pretrained models, as far as I know.

\+ The paper has conducted extensive experiments, including without finetuning (WT), prompt tuning (PT), finetuning (FT), and zeroshot. All showing satisfactory results given its number of params and training data. 

\+ The task interference problem is tested and shown as a clear motivation of using MoE.

\- The technical novelty is still somehow limited though it demonstrates effectiveness.

\- Some details of experiments are missing. e.g., It's not clear which conditional strategy is adopted at default, in all the experiments, it's not clear how the prompt tuning is implemented.

\- The term zero-shot in this paper seems to indicate data of unseen domain, since task id alone does not make it possible to achieve zero-shot inference for an unseen task in a totally different form.  


questions: 1. Which conditional strategy is exactly adopted in the rest of the experiments? I see statement of `#param is the parameters required during model deployment' in table 4. I wondering if the adopted strategy is data-independent so that we can use only part of the whole parameters to do the inference. 

2. What's the training FLOPS of both MoE and non-MoE version of the model? 

3. How can such models do zero-shot inference on unseen tasks (in different form, instead of different domain) when task identifier is only id rather than general task representation like instruction? If using attribute condition, the 8-bit attribute seems also less expressive to be general.

4. How is the prompt tuning implemented? If the authors use exactly the same method with prior works, it's recommended to cite them or describe the detail in a simple way.

5. Just curious about how PT_{1%} compares to FT_{1%} and PT_{100%} to FT_{100%}.

6. OFA also reports the result of MS COCO caption using CE optimization only. 



limitations: The authors have addressed their limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a generalist model (ie a model that handles multiple modalities and numerous tasks) based on a Mixture of Experts (MoE). The paper argues that, since current generalist models often share modules across different tasks and modalities, interference among different tasks is responsible for performance degradation when compared to specialist models. To mitigate interference, the proposed method dynamically routes the execution of specific tasks to specialist modules within the MoE. 
strengths_and_weaknesses: - The proposed approach (a generalist Conditional MoEs) is well motivated. Different tasks can cause interference in gradient directions and lead to sub-optimal performance in individual tasks. Sparse mixture of experts, dynamically allocated based on a task prompt, has the potential to solve this problem while still retaining the ability to perform zero-shot task generalization.
- The existence of task interference was well-validated through an interesting analysis of image classification, masked language modeling, and captioning. It showed that while image classification and captioning can cooperate when learning lower layers in the model, they interfere with each other at higher layers.
- The paper proposes several approaches to conditioning the routing mechanism with task-related tokens (ie information that describes the current task and modality). The proposed routing strategies make sense, and they were shown to have an impact on the overall performance. It is was interesting to see that one of the best strategies was to rely on a hard-coded task description (summarized in Table 2). Would new tasks require new descriptions? Would the model benefit from a more generalized task description? Perhaps using learned task embedding, eg using something similar to task2vec ([https://arxiv.org/pdf/1902.03545.pdf](https://arxiv.org/pdf/1902.03545.pdf))?
- Experimental results (Table 3) show that interference has a detrimental impact on performance (as specialist models always have higher performance than a model which shares all parameters across tasks). The paper also shows that the proposed MoE is able to mitigate the impact of interference, achieving almost the same performance as specialist models on multiple tasks. Having said that, it is still disappointing that there is little evidence of task transfer, as the generalist model does not improve performance in most tasks (except for COCO Captioning B@4). I believe this is still a limitation that should be clearly stated in the paper.
- Finally, it is impressive that the proposed model was able to achieve competitive performance on several tasks (like object and action recognition, cross-modal retrieval and language understanding with little to no extra data. This ability to do zero-shot or few-shot transfer highlights the benefits of generalist models in general (which the current approach still retains).

In sum, I believe this is a strong paper. While the ideas of mixture-of-experts and the existence of interference in multi-task learning problems are not new, this paper successfully applies these concepts to generalist models, and achieves good results on several datasets.
questions: see Strengths And Weaknesses
limitations: see Strengths And Weaknesses
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes to augment the previously proposed Uni-Perceiver model with mixture-of-experts (MoE) routing so that parameters of a model trained on several tasks can specialize by task and data modality. This choice is motivated by appealing to the phenomenon of task-interference, where the contributions to the gradient updates from different tasks for shared parameters do not perfectly align. The paper explores several ways to parameterize the MoE routing, and settles on a strategy that learns to route based on a featurization that encompasses task, token, and modality type. The resulting architecture is evaluated against ablations and strong baselines on several image, video, and text domains.
strengths_and_weaknesses: **Strengths**

- The proposed MoE input featurization appears to typically improve performance when plugged into the Uni-Perceiver model. 

- The MoE strategy used is more flexible than strict per-task weight sharing, while relying on similar amounts of knowledge about the input domains and tasks. 

- Appendix Table 10 provides a very useful overview of the training protocols of a variety of specialist and generalist models.

**Weaknesses**

- It’s unclear that task interference (as characterized in section 3.1) either limits performance by generalist models or (if it does) that it’s addressed by the proposed MoE mechanism. Task interference is central to the claims and justification of the model, so evidence that this effect is a limiting factor is essential. See **Questions** below for more details. 

- The results in Tables 4 and 5 are very hard to interpret, as they contain dozens of numbers with no clear evidence of overall improvement and no comparisons at matched performance in terms of wall-clock training, FLOPs, or other computation or memory measures.

- The proposed model doesn’t appear to strongly outperform any of the baseline models or open up new applications domains (it is applied only on standard tasks that are well-addressed by specialist methods or by other generalist methods that do not appear to require solutions to the proposed task-interference problem). What is the use-case for this model?

questions: - Does interference (as discussed in Table 1) correlate with task performance? It’s unclear that gradients of different orientations between tasks is evidence of difficulty optimizing their average, as claimed in L34-36.

- Does the proposed MoE block reduce the average interference metric discussed in Table 1 and used to motivate the proposed system? This is used to motivate and justify the claims at several places in the text (e.g. abstract, intro, section 2, sections 4.5), but no evidence is presented for changes to this effect.

- The MoE routing relies on a hand-designed featurization that appears to be unlikely to scale beyond a small number of tasks and datasets (as opposed to the token-level routing, which is very general). How does token-level routing perform outside of the ablation study shown in Table 3?

- The paper does not mention the Perceiver architecture by name, but only to the subsequent Uni-Perceiver architecture. This is historically misleading.

- L298: “we propose Conditional” -> “we propose Conditional MoEs”

- Footnote under L258: "konw" -> "know"
limitations: n/a
review_title: null

Meta-review: metareview: This paper proposes a generalist model Uni-Perceiver using conditional mixture of experts for processing multi-tasks. Most reviewers acknowledged the technical novelty of this work in CMOE, attribute routing, as well as extensive experiments. Reviewer KJqa argues the unclear affect of task interference for MTL. The authors reply well from a gradient perspective and experiments show task interference can be reduced. The other concerns about clarity in paper presentation and results are also addressed. The meta-reviewers thus recommend accepting it.
confidence: Certain
Decision: Accept