Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs
Authors: Jinguo Zhu
Xizhou Zhu
Wenhai Wang
Xiaohua Wang
Hongsheng Li
Xiaogang Wang
Jifeng Dai
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generalist Models
Task Interference
Mixture of Experts

Abstract: To build an artificial neural network like the biological intelligence system, recent works have unified numerous tasks into a generalist model, which can process various tasks with shared parameters and do not have any task-specific modules. While generalist models achieve promising results on various benchmarks, they have performance degradation on some tasks compared with task-specialized models. In this work, we find that interference among different tasks and modalities is the main factor to this phenomenon. To mitigate such interference, we introduce the Conditional Mixture-of-Experts (Conditional MoEs) to generalist models. Routing strategies under different levels of conditions are proposed to take both the training/inference cost and generalization ability into account. By incorporating the proposed Conditional MoEs, the recently proposed generalist model Uni-Perceiver can effectively mitigate the interference across tasks and modalities, and achieves state-of-the-art results on a series of downstream tasks via prompt tuning on 1% of downstream data. Moreover, the introduction of Conditional MoEs still holds the generalization ability of generalist models to conduct zero-shot inference on new tasks, e.g., videotext retrieval and video caption. Code and pre-trained generalist models are publicly released at https://github.com/fundamentalvision/Uni-Perceiver.

Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs
Jinguo Zhu Zhu
Xizhou Zhu Wenhai Wang?, Xiaohua Wang',
Hongsheng Li Xiaogang Wang", Jifeng D
Xi'an Jiaotong University Tense'Time Research Shanghai Al Laboratory "The Chinese University of Hong Kong TTingghu University
Abstract
To build an artificial neural network like the biological intelligence system, recent works have unified numerous tasks into generalist model, which can process vari- ous tasks with shared parameters and do not have any task-specific modules. While generalist models achieve promising results on various benchmarks, they have per- formance degradation some tasks compared with task-specialized models. In this work, we find that interference among different tasks and modalities is the main fac- tor this phenomenon. To mitigate such interference, Mixture-of of- of-Experts (Conditional MoEs) to generalist models. Routing strategies under different levels of conditions are proposed to take both the training/inference cost and generalization ability into account. By incorporating the proposed Condi- tional MoEs, the recently proposed generalist model Uni-Perceiver mitigate the interference across tasks and modalities, and achieves state-of-the-ae results on series of downstream tasks via prompt tuning on I % of downstream data. Moreover, the introduction of Conditional MoEs still holds the generalization ability of generalist models to conduct zero-shot inference on new tasks, e.g. video- text retrieval and video caption. Code and pre trained generalist models are publicly released /tttts://ggtt http/ggtttttuet.e/egg wwwm//unddametttlviiioooooo
introduce the Conditional
effectively
pre-trained
Perceiver
Introduction
Generalist models that handle multiple modalities and numerous tasks have been long pursued by focus developing specialized models with task-specific modules. When these models are applied new tasks, the specifically-desigee components need to be redesigned on demand and fine-tuned on sufficient downstream data. As result, their model size increases with the number of diverse downstream tasks, conflicting with the goal of generalist models.
the machine learning community. However, previous researches [5 89
86 have made preliminary attempts to build generalist models by modeling various tasks into unified formulation. With the unified modeling, large-scale pre-training on various datasets enables the generalist models to process different downstream tasks using shared parameters. These generalist models not only achieve competitive performance on
Recently, some pioneers 993
pre-training tasks (79. 3 84 [6], but also can perform zero-shot inference on novel tasks without
introducing additional parameters
[93 (2)
However, compared specialized models with specific parameters for each task, generalist models with shared parameters would suffer from the task-interference issue - different tasks with shared parameters may conflict with each other (88) The same issue is also observed multilingual NLP
"Equal contribution. 'This work done when Jinguo Zhu is an intern at Shanghai AI Laboratory. ""Corresponding to Jifeng Dai <daijifeng eed..n.....duu.c...
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

models 4 81 83 We argue that the iusk-interference issue is mainly caused by the inconsistent optimization multi-task learning. As shown in Tab. I during the training phase of generalist models, the gradient directions of different tasks would be inconsistent or even opposite. Thus, if multiple tasks share parameters, the optimal update direction of the shared parameters will be uncertain, resulting in sub-optimal performance.
Allowing conflicting modalities and tasks to use separate parameters should effectively mitigate the interference issue in generalist models. Mixture of Experts (MoEs) 43 23
provides a potential solution, which learns to activate sub-networks dynamically without introducing any task-specific modules. Nevertheless, vanilla MoEs [67] select the experts according to token representations, which suffers from high training/inferenee cost and neglects the information of different tasks and modalities. In this work, we argue that routing strategies of MoEs require special design when applied to generalist models for mitigating the -eteeeetereeeece issue.
To address the ieskiinterferece issue in generalist models, we propose Conditional Mixture-of- Experts (Conditional MoEs), which improve vanilla MoEs by introducing information under different levels of conditions, including token-level, context-level, modality-level, task-level, and predefined token attributes. In this case, vanilla Moïs a token-level variant of our Conditional MoEs, which can be replaced by other-level variants to implement stronger generalist models. We carefully discussed the training/inferecee cost and generalization ability of different variants, and ablated their performances in mitigating the interference issue of generalist models. Notably, Conditional MoEs with predefined token attributes introduces 8-bit attribute embedding to describe the information of currently processed task and modalities, which demonstrate excellent computational and memory efficiency and good generalization ability.
To verify the effectivvness of Conditional MoEs, we incorporated it with the recently proposed generic perception model Uni-Perceiver 93 by replacing the linear projection in self-attention and FFN blocks with conditional MoE layers. Experiments demonstrate that, by mitigating task interference with our proposed Conditional MoËs, Uni-Perceiver can be pre-trained on various tasks jointly without performance degradation, while its generalization to other tasks can be maintained simultaneously. Our main contributions are as follows:
We carefully analyze the inskiinterference issue in generalist models, and provide an explanation from the gradient direction perspective as well metric to quantify the issue.
We propose Conditional MoFs address the fask-interference issue in generalist models. By intro- ducing the information of currently processed task and modalities, Conditional MoEs effectively mitigate the interference issue, while keeping low computational and memory cost.
Compared with previous SOTAs, our generalist model with 1% downstream data prompt tuning achieves competitive performance, while only <5% training data and <10% training cost used. We hope this work can serve as a solid baseline for generalist models and motivate further research.
2 Related Works
Specialized Models. CNNs
Previous research focuses on building specialized models for specific tasks.
47
70 and ViTs 20
53
76
80 are developed for image classification. Subsequent
works de-ign them to adapt to diverse downstream visual tasks, e.g., object detection (63] and
segmentation 15 48 In NLP, different architectures are specifically designed for neural machine translation 777, natural language understanding [19], and natural language generation [51). As for vision-language tasks, previous works usually combined modality-specific encoders and representa- tion fusion modules together 13 [44] Recently, [89] 65 71 integrate several specialized models into a single one to handle diverse tasks. Such integrated specialized models are cquipped with multiple task-specific modules to adapt to many downstream tasks possible. However, these methods still follow the task-specific paradigm, which conflicts with the objective of generalist models.
Vanilla Generalist Models. Vanilla generalist models handle different tasks and modalities with shared parameters. Uni-Perceiver [33 formulates various perception tasks as finding the maxi- mum likelihood target for each input through the similarity of their representations. OFA (79). Flamingo 3 and VIM [84 attempt to unify different tasks into sequence-to-sequeee
genera- tion. UniCORN (66) and Gato (22 further incorporate bounding box and reinforcement learning tasks into the unified formulation, respectively. These generalist models not only achieve competitive performance on pre-training tasks with shared parameters, but also can perform zero-shot inference
2

---

Standard Encoder Block
'Eccdder Block with depandent parametrization
Encoder Block oixpers
takers
lask2
bask takens
bask2 takens
wsighted
FFN
aaski FFN
Task]
MaE FFN
JCOUTT
ROUUITY
Layer Norm
Layer Norm
Layer Norm
2
Attention
Altention
Tastl
Tesk2 Attenton
M Attention
Gating
Layer Norm
Layer Norm
Layer Norm
task2 tokens
task2 takens
taskd takers
lask2 tokens
task1 takens
task2 tokens
task3 tokens
task2 tokenss
Figure Comparisons of fulle-shared standard encoder block, task-specific encoder block with task-dedicated parameters, and encoder block with efficient MoE parameterization.
on new tasks 662 93 However, these methods rarely investigate the potential interference among different modalities and tasks, which could result in the performance degradation of generalist models.
Multi-Task Learning. Multi-task learning
8 has been widely studied in the community
of vision [27] [74] 721. language [25] 16, [50] and vision-lant lision-language learning [10] 55
29]. While
multi-task training enables collaboration between tasks, it may also introduce the task interference
problem [8
83 28 36 72). To mitigate the task-interference issue, some works [14] 24 38 propose
dynamically adjust the loss weight for each task, while others 990 40 36) instead task-dedicated
parameters. However, methods with task-specific parameters are difficult to generalize to new tasks and do not meet the requirements of generalist models.
Mixture Experts (MoEs). MoEs has shown its remarkable ability to scale neural networks [67 43
23
64
21.
first proves the effectiveness of MoEs by stacking MoE layers in the LSTM models. further introduce this approach to Transformer architectures. [2 40
[67
train language models with trillion parameters successfully by utilizing simplified MoE routing strategy and efficient training techniques. There are also some works applying MoEs to CNNs for computer vision tasks [ 85 82).
66
43
Recently, V-MoE (64) successfully employs MoEs ViTs, showing promising performance many visual tasks. Task-MoE [42] focuses on applying MoEs for multilingual translation to mitigate the interference among different languages. In this work, aim explore MoEs under different conditions for general models.
3 Methodology
In this section, we first analyze the task-interference problem from the gradient direction perspective. Based on the analysis, we propose Conditional Mixture-of models, which introduces parameters conditioned by information of different levels to mitigate the lask-interference issue with negligible overhead.
Mixture-of-Experts (Conditional MoEs) for generalist
3.1 Task Interference
To quantify the interference of the j-th task on the i-th task, we estimate the change in loss L, of the i-th task, when optimizing the shared parameters 0 according to the j-th task Lj as:
A) = TZz, [T((z=;0= Ta(a;; 0
Volu(a)) AER, VoL,(1,)))
VoLi(a)) [v,,,
Vola(z:)) (1)
where x; and xy are the sampled training batches of the i-th and j-th tasks, respectively, and À the learning rate. Without loss of generality. we only consider the update direction igmoring the update norm. Then, the interference of the j-th task on the i-th task can be quantified as:
I, = Ex; (AgLe(1.))
(2)
L. (1x2)
3

---

Table The average interference metric I,, of the task j on the task i the 4-th/12-nd FFN blocks. To calculate the interference metric, we sample 100 batches for each tasks, and record the gradients based on the pre-trained Uni-Pen Prii-Pererrive-TTi The red value indicates that the task j has a negative impact on the task i, and the green value indicates positive impact.
(a) The 4-th FFN Block
(b) The 12-nd FFN Block
Task / ImgCI.S S (Img)
MLM (Text)
Caption (exg-Text)
Task
ImgCT S (1mg)
ML.M (Texi)
Caption (Tmg-Texx)
Tisk
Task
ImgCLS (1mg)
1.00
-0.57
1.29
ImgCLS (1mg)
1,00
-2.91
-2.45
MLM (Text)
0.07
1,00
0.68
MLM (Tex))
-1.65
1.00
-1.05
Cuption (Img- Text)
0.01
0.01
1,00
Caption (Img- Texi)
-0.11
0.19
1,00
where the denominator is used to normalize the loss change scale. As reported in Tab. [ we sample 100 batches for each tasks, and record the gradients calculate the average interference metric I of the j-th task on the i-th task at the 4-th/ 1idd FFN blocks. We see that, at shallow layers, the image caption task has positive impacts on image classification and masked language modeling, suggesting that cooperation between different tasks exists. While at deep layers, tasks with different optimization objectives hardly enhance each other, and the gradient directions may even opposite.
Fig. T summarizes three mainstream architectures for multi- models. The first is the standard with parameters fully shared by different tasks, which suffers from task interference problem as analyzed above. The second is task-speeific parameterized architecture [8
architecture 993 31 32
65 29 7 equipped with dedicated parameters for each task. Although this architecture address the interference problem by task-specific parameters, it is difficult to generalize to new tasks that did
not emerge in the training phase. Unlike the above two architectures, the fixture-of-Experre (MoE) architecture 167 43 23
40 64] activates models sparsely according to different given inputs by
selectively utilizing different subset of the model parameters. The sparse routing mechanism makes possible to train very large generalist models, which maximizes the collaboration and meanwhile mitigates the interference problem. In this work, we focus on exploring Conditional MoEs for general models, whose experts are gated by conditions from different levels.
3.2 Conditional Mixture-of-Experts (Conditional MoEs)
We first describe the prototype of Conditional MoEs, and then provide its specific instantiations under different conditions, as as the application to generalist models.
Prototype. Given any token 2; in the input sequence X {ai conditional MoEs with E experts firstly introduces a gate decision vector G E IR that dispatches different input tokens to different experts, which is calculated as:
G Lop, (softmax (W, 9 R(as) +c)
where R(-) defines general routing strategy for gate decision, which is alternative under different conditions. W, y the trainable weights gate decision and e is the noise term. The topx(-) operator sets all values to be zero except the largest & values. Since G only has k < E non-zero values, the token 2; routed to only small number of experts. After getting the gate decision vector G. the corresponding output yJ is the weighted combination of each expert's computation on 2; as:
y1 Σ
Ge
W. WI.
(4)
where W. is the linear projection weights of the e-th expert and gate decision G. determines how much the e-th expert contributes to the output y;- Note that, experts with Ge 0 does not need to be computed for saving computation.
In Conditional MoEs, the routing strategy R(-) plays an important role in the multi-modality and multi-task training of generalist models. By sparsely activating experts according to different conditions, Conditional MoEs can mitigate the interference issue while maintaining the generality of the pretrained model. Next. we introduce variants with specific routing strategies under different conditions, as shown in Fig.
Token-Level Routing, Similar to vanilla MoEs 67
43
23
64 the Leken-Level MoEs directly
use the token representation for the routing strategy, which can be written as:
Heoken (x:) di.
(5)

---

Taskl Input
Task2 input
Different modalities
Taskl target tokens
A
A
Task2 target tokens
Token feature
Context feature routing
feature
Modality
Task
Attriutte
routing
reuting
routing
routing
Expert1
A
D4
A
O 0 A
AAA
A
Expert2
A A
Â A A
A
OO
OO
Expert3
AÄA
A
AAAAA A
Expert4
AAAA A
Token level routing
Context level routing
Moda ity level routing
Task level routing
Attribute reuling
Figure 2: Comparisons of routing strategies with the top-1 gate decisions under 2-task training.
The routing strategy of token-level MoEs is an identical function, where the gate decision only depends on each token's 's own representation.
Lent Routing. Tokens with similar representations may appear in conflicting tasks, whose optimal expert decisions should be different to mitigate the task interference. Therefore, to help gate function making more reliable decisions, we explore the combination of global context and local token representation. The routing strategy utilizing global context can be expressed as:
Remiex (2.) concat (2,,, attnpool(X)),
(6)
where concat(-) indicates the concatenation operation, X = (rr)/in is sequence of all tokens in the current sample, and attnpool(-) indicates the attention pooling operator (6]).
Modality-Level Routing. Most current practice uses modality-specific encoders with independent parameters for different modality inputs. Inspired by this, we also explore to leverage the modality of the current token as a routing strategy:
Raodal embed(idmmda((zz))
Here, embed(.) represent the embedding layer, and (-)aal()) indicates the modality index of current token z,. The routing function will assign this token to experts according to its modality embedding.
Tesk- Routing, In addition to modality information, task information can also be used to guide gate functions make reliable decisions for mitigating the task interference. Similar to Eqn. 77. the routing strategy can be formulated as:
(8) where i-)assk()) is the task index of current token 24. The task embedding for this task will be used to compute gate decision. Since all tokens from one task have the same task embedding, all tokens corresponding to this task will be routed the same of experts.
Rask (u:) = embee((dhnaa
Attribute Routing. Among the aforementioned variants, token-level, context-level, and modality- level routing strategies only focus input tokens but omit information about the currently processed task. While task-level routing strategy relies on task-specific ids, it limits the generalization ability to downstream tasks. To introduce the information of currently processed task and modalities without losing the generalization ability of generalist models, we propose to introduce token attributes to assist the gate decision.
As described in Tab. 22, the attributes of the current token are represented as an 8-dimensional binary embedding, whose attributes include the modalities of current task and token (index 0~55), the causation type of the model (index 6), and the token source (index 7). As result, the designed token attributes provide comprehensive information of currently processed task meanwhile keeping the task generalization ability. Based on the attribute embedding, the routing strategy is expressed as:
(x,( layernorm (Watr aatr attr(æ:)).
(9)
5

---

Table 2: The 8-dimensional binary embedding used for attribute-level routing strategy. The attribute embedding is assigned to a token by checking whether the statements of the eight descriptions match the current token. For example, the attribute embedding for any token from the input sequences of image classification task should be [1. 0, 1, 0, O, 1/. Please refer to the Appendix for detailed look-up table of attribute embeddings for all tasks in our work.
Index
Descriptions
Yes No
0
Visual modality exists in the inputs of the current task.
1
Text modality exists in the inputs of the current task.
0
2
Visual modality exists in the largets of the current task.
0
3
Text modality exists in the targets of the current task.
4
The modality current token visual.
The modality current token text.
The attention mask current token is causal.
The current token cornes from inpuls, not the targets,
Here, attr(x;) is the 8-dimensional binary attribute embedding of the current token zz; as described in Tab. 2 W is the learnable weights to transform the attribute embedding to latent representation, and layernorm(-) denotes the layer normalization [6] for training stabilization.
Application to Generalist Models. Without loss of generality, we explore the application of Conditional MoEs to the generalist model Uni-Perceiver [93]. which uses Transformers to handle various modalities and tasks with shared parameters. We replace linear projection layers in both self-attention and FFN blocks with Conditional-Mon layers (see Fig. I.
3.3 Comparison of Conditional-MoE Variants
As illustrated in Fig. 2 among the variants of Conditional MoEs, token-level and context-level MoEs are data-dependent, while modality-level, task-level, and attribute MoEs are inta-inneeendent.
Training and Inference Cost. Compared to dense models with the same number parameters, all Conditional MoE variants significantly reduce the computational cost benefiting from the sparse routing mechanism. Due to the dependence of input data, the memory consumption of token-level and context-leve MoEs is relatively high during model training, and model parallelism is required to relieve memory cost by partitioning experts across multiple devices, leading to heavy inter-device communication overhead. This problem persists when using pre-trained models for task-specific inference, where all experts need to be loaded into memory and might be activated by any token.
Different from data-dependent Conditional MoEs, data iate-inndeenndett variants such modality level, task-level, and attribute MoEs have excellent memory efficiency, since only top-k experts need to be activated for all tokens with the same modality/task//ttrbbuuc inference phase, the experts in a data- inttaaindeeeeentt MoE layer can be merged into a single lincar projection using reparameterization techniques. In this case, the computation cost of the network with data-independent Conditional MoEs will be equivalent to a dense model without MoEs.
Moreover, in both training and
Generalization Ability. We hope mitigate the fast-interference issue in generalist models, while keeping their generalization ability to new downstream tasks. While token-level, context-level, and modality-level MoEs without task-specific designs do not harm the generalization ability, they ignore the task-level information which essential to resolve the task interference. Conversely, taskl routing strategy is tied to specific task id, which is difficult to generalize to new downstream tasks. Attribute MoEs introduce predefined token attributes to comprehensively describe the information of currently processed task and modalities, which can be transferred to new downstream tasks without any task-specific modifications. This gives attribute MoEs the potential to mitigate task interference without losing generalization ability.
Experiments
In this section, we first describe our experimental setup. Then, we confirm the task-interference issue in the generalist model Uni-Perceiver [93] and ablate the the ability of different Conditional MoEs to mitigate task interference. Finally, large-scale training is conducted to verify the effectiveness of our proposed Conditional and its generalization ability to novel tasks.

---

Table 3: The performance of different routing strategies for Conditional MoEs. The base model is Uni- Uni-Perceiver with BERT inn. We also illustrate the task-specific variant where each task has its own specialized parameters. The training and validation performance reported on three tasks: image classification on ImageNet- IK [8. image caption on COCO Caption [ and Masked I Language MMd)l on Books& Wiki. The best results within a tolerance of % are in bold.
task-specific parameterization
training time
inferenec time
ImageNet-Ik TacQuic
COCO Caption
MLM
model
facc
tacquis
1B
TacCuin
-pplaa
1.0x
1.0x
47.3
68.3
49.2
18.2
54.5
5.86
Uni- Rii-Pereeiver-Ii
D3
1.1x
1.0x
53.3
73.5
52.6
20.4
60.5
4.48
Conditional MoEs
188x
2.2x
53.1
72.7
52.9
20,9
58.3
4.96
Conditional MoEs coneest
2.2x
2.6x
52.5
73.1
52.8
21.5
58.6
4.86
Conditional MoEs modalty
1.4x
1.0x
51.7
72.6
52.1
21.8
57.5
5.06
Conditional MoEs tnk
1.4x
1.0x
52.9
73.2
52.7
21.2
59,9
4.56
Conditional MoEs attribute
1.4x
52.8
73.3
53.1
23.0
60.0
4.56
4.1
Datasets
We use the same datasets in Uni-Perceives [93 to pre-train our model Specifically, ImageNet-
21k [18] is used for image classification pre-training. Kinetics-700 [37 and Moments in Time 157
are used for video classification pre-training. Language modeling task is trained on BookCorpus 94
& English Wikipedia (Books&Wike For language modeling with image clues and image-text
retrieval, we use a combination of -mage-text-pair datasets: SBU Captions (SBU) [58] Visual
Genome [4]] COCO Caption [12]. CC3M [66], CC12M
[ and YFCC [35]. Following Uni-
Perceiver, ImagenetIK [8], Kinetics-400 [37], COCO Caption [[2], and Flickr30k (59 are utilized
to evaluate the performance of generalist models downstream tasks. We also use two datasets that evaluate the generalization ability to novel tasks: MSVD 11 and GLUE 78 Additionally, all
dataset licenses are included in Appendix.
4.2 Implementation Details
We incorporate the vanilla generalist model Uni-Perceiver with Conditional MoEs for experiments with three different variants: Uni-Perceiver-TT Ti (Tiny). Uni-Perceiver-B (Base), and Uni- L (Large). Please refer to Appendix for architecture hyperparameters. If not specified, the input image resolution is set to 224. In each training iteration, each GPU independently samples single task and dataset. The gradients of different GPUs are synchronized after the gradient back- bookkpropagatio. We use the AdamW optimizer with base learning rate of 0.0005 and weight decay of 0.05. Similar to 52 61]. we find setting B2 = 0.98 and = 10- helps improve stability when large-scale training. Besides, gradient clipping with 0.5 is used to stabilize training.
Uni-Perceiver-B B and Uni- Lei--eerceiee--L are equipped with Conditional-Mort layer for every other layers while UniiPeercepiinnTii Ti use Conditional MoEs in all layers. A normal noise is also used on the gate logits following (64] for a better exploration for new potential experts. not specialized,
top-2 gate function is used. For other hyper-parameter of MoE layers, please refer to Appendix.
4.3 Ablation Studies
This part explores whether Conditional MoEs can effectively mitigate task interference in generalist models and compares different routing strategies. Tab. Jubmmarzmssse the performance of Uni-Perceiver and its variants on three typical tasks. Compared with task-specific parameterization, the performance degradation of Uni-Perceiver confirms the existence of task interference. Incorporating Conditional MoEs with any routing strategy can mitigate the musk-interference issue and significantly improve the performance. Among these five routing strategies, token-level, context-level, and modality-level MoEs deliver slightly worse performance. We argue the missed task information is critical for resolving the task interference. Besides, the data-dependent MoEs, i.e., token-level and context-level, have relatively higher training and inference cost, while the other three MoEs have excellent efficiency by using reparameterization techniques. Although both task-level and attribute MoBs achieve good performance, the specialized task-id design in task-level MoEs makes difficult to generalize to new tasks. Therefore, the Conditional MoEs with attribute routing strategy will be used.
As far know, all datasets do not contain any personally identifiable information or offensive content.
7

---

Table The performance of incorporating Conditional MoEs with Uni-Perceiver on image classi- fication, video classification and image-text retrieval. "#param" is the parameters required during model deployment. "#data" is the amount of visual training samples involved. "WT", "PT", and "FT" indicate w/o tuning, prompt tuning, and fine-tuning, respectively. "1%" and "100%" indicate the proportion of downstream data used for tuning. "FT 100s means fine-tuning with larger image size. The subscript number next score indicates that different image resolution than 224 is used.
i These methods use > x training data size and > 10x training cost than ours.
Image Classification accuracy ImageNet-Ik.
(b) Video classification accuracy on Kinetics-400.
Method
4param
Adata
WT
PTys
FT
FTuest
26
83.
Method
4param
PTys
FTRos
MTTBB 183
B6M
15.5M
$4.0
85 85.5
TimeSronmer-B
121.4M
14.2M
80.7
VIFL 18
VATT-B 2
87.9M
238M
79.6sa0
OHA 79
472M
60.6
84.9
VATT-L
306.IM
238M
82.1
CLIP
307M
400M
76.2
VIVIT-I 5
>307M
14.2M
81.7
ALIGN
63
480M
18B
76.4299
88.6gs
VIVIT-L 3
>307M
300M
84.9
CoCa-B
Plorence
37
637M
86M
900M
4.8
K3.7
82,6
90,0
88. 30ss
FFlorence h9
647M
900M
86.5
CoCa-L
303M
1.8B
81.8
90.2
CCoCs
87
2.1B
4.8B
88, 9776
Flamingo-31
3.2B
2.3B
71.0g3
Bn-PPereeiver-BB
86M
44.1M
74.5
74.8
77.7
Uni- Fen-Peeccive--8
86M
44
44
79.2
80.9
84.0
K5.2
Conditional MoEs
86M
44.1M
76.8
77.2
79.3
Conditional MoEs
86M
44
14
80.3
82.0
84.5
85.8
Uni Perceiver-I.
303M
44.1M
79.5
80.0
81.9
Uni Perceiven-L
354M
303M
82.7
84.2
86.2
87.0
Conditional MoEs
303M
44.1M
82.1
83.0
84.2
Conditional MoEs
303M
41.M
83.1
84,9
86.4
87.0ga
(c) Image-text retrieval R@! performance.
Flickr30K
MSCOCO Caption
Method
Image
Text
Image PTis
Image
Text Image
fpaaam
Adata
WT
PTys
ITTtte
WT
WT
PTss
WT
PTis 17
ImageBERT UNITER-B
30
170M
10M
70.7
87.0
54.3
73.1
44.0
66.4
32.3
50.5
146M
9.6M
80.7
66.2
72.5
50.3
UNITER-L L 13
363M
9.6M
83.6
87.3
68.7
75.6
65,7
52.9
VILT 39
87M
9.7M
73.2
74.8
56.5
61.5
55.0
64.4
40.4
42.7
FLAVA 71
215M
70M
67.7
65.2
42.7
38.4
C1.IP 61
417M
400M
88.051
58 58.4
37.8
TALIGN
820M
1.8
88
88.6218
75.722
84.9
58. 58.62s
77. Ozao
45.6
59.9
Florence
893M
900M
90.9
90.9
97.2aa
76.7
87.9
64.7
47.2344
TCoCa-B S7
383M
4.8B
89.8515
76.8
63.8s1e
47.533%
1CoCa.L
37
787M
4,80
92.5
80.451
66.3
51.25%
"Flaming-38
3.2B
2.3B
89.3:10
79.5
65 65.9110
48.0xza
Peiceverei
71.1
76.0
77.5
50.7
51.9
53.9
Conditional MoFs
167M
MM
82,1
91.3
93.6
72.4
78.5
79.8
64.6
68.9
70.5
51.6
52.6
54.1
Uni 1.
354M
44
83.7
92.1
94.7
74.2
80.0
82.1
67.8
73.3
74.4
54.1
56.2
57.9
Conditional MoEs
505M
4M.1M
83.6
92.4
94.1
75.9
80.6
83.7
67.9
73.3
74.7
55.3
57.1
58.3
4.4 Evaluation on Pre-training tasks
Large-scale training conducted to verify the effectiveness of our method, we first evaluate it on tasks
involved in pre-training. Specifically, we use widely-used Imagenet- 1k [18 and Kinetics-400 (37 to evaluate image and video classification respectively, and use popular Flickr30k 59 and COCO
Caption 12 to evaluate image caption and image-text retrieval.
Tab. 4 and Tab. 5a show the results on the four pre-training tasks. We see that Uni- Uni-perceiver with Conditional MoEs consistently outperforms vanilla Uni-perceiver by a large margin. Without any tuning, models achieve comparable performance with task-specific SOTAs trained with similar model size and training data size. Note that, our approach a generalist model pretrained on a unified task formulation, while task task-specific approaches are trained specifically for the target task.
When prompt tuned on only 1% downstream data, the performance of our models are boosted to a level close to counterparts that use >50x training data sizes and > 10x training cost. For the prompt tuning of our models, only a small amount of parameters are tuned, and the encoder is still fixed and shared among different tasks, indicating that generalist models with Conditional MoEs can handle different tasks with significant low cost than counterparts.
We further fine-tune our models with 100% of the downstream data. In this case, our model achieves performance on-par with or better than the SOTAs trained with similar data size on all these tasks, which proves generalist models with Conditional MoEs has learned high-quality representations.
8

---

Table The performance of incorporating Conditional MoEs with Uni-Perceiver on image caption, natural language understanding, video-text retrieval and video caption, where the last three tasks are not involved pre-training.
(a) Image caption BLEU@@ performance, * means meth- ods use region features as network inputs. indicates that Cider optimization is used.
(b) Natural language understanding (novel task) fine-
tuned on GLUE. BERTBASE records from [34]. Visu-
alBERT and LXMERT record from (30). "RoBERTa
uses 10x training text tokens than
MSCOCO Cuption
UUnifedd VLP
+pumm
36M
3.0M
data
WT
PT.s
36.5
PTss PTTe
30.1
Mathod
MNLI
(Acc)
QNLI
(Acc)
QQP (F1)
RTE (Ace)
SST-2
(Aec)
MRPC (F1)
CoLA
(Mcc)
OSCAR--
4m
154M
6.5M
36.5
L
384M
6.5M
37.4
LXMERT
75.3
80.4
UNICORN
VisuaBBEKT
$1.6
87.0
86.0
56.6
99,4
82.1
38.6
BLIP-B
252M
120M
39.
SimVLM-B
B
81
83.4
88.5
87.2
63.9
90.9
B4.4
46.7
BI-P--1
48
473M
129M
40.
ВEKТ-B
73
88.4
53.3
89.0
54.7
CLI--VIL
3.99M
400M
40.2
92.3
91.3
93.2
88.0
SimVLM
632M
11.2
40..0m
OFA-R
84.3
91.1
S3.4
70,8
92.7
90.6
52.3
472M
OFA-I. h9
86.6
92.8
88.9
73.6
94.7
91.4
53.1
HOSCAR
380M
0.5M
41.7
91.9
90.2
"RoEERT--L
90.3
94.7
92.2
86.6
96,4
90.9
68.0
CCCC SY
2:10
4.80
40.9p%
Uni Perceher--B
79.7
87.3
86.7
71.1
99.3
86.0
43.1
Uni
124M
35.5
26.4
Conditional Mol's
+ Conditional MoEs
Uni-PPneerier-I.
Conditional Molls
505M
154M
167M
46.1M
4M
25,5
33.2
15.3
29.3
36.8
58.6
37.3
10.2
40.5
15.8
15.1
15.9
30.7
33.7
$2.0
36.2
35.5
Uni Pereeiverr
+ -Condtiionl MoE's
82.5
85.7
89.2
91.9
89.5
87.7
73.7
78.4
91.2
93.4
91.2
90.2
52.0
57.4
(c)
Video-text retrieval (novel task) Recall @ 1 and video caption (novel task) BLEU @4 performance on MSVD.
Video WT PTys
Text FTpes
Video MTwm
Caption
Method
#puram
*data
WT PTys
WT PT:s IThe
CLIP2vidco
2
132M
400M
58.7
47.0
HunYoan_ow Dvr 56
364M
400M
68.0
52.7
ORG-TRL 21
86M
2.0M
54.3
Peroeiva--BB
62.7
62.8
38.7
15.8
59,5
Conditional MoEs
167M
44.1M
52.8
63.6
65.0
40.0
45.3
47.8
23.4
60.0
65.4
Uni Perceeveeie..
354M
1M
45.4
65.5
68.2
34.2
48.6
50.8
24.7
67.2
68.3
Conditional MoEs
505M
44. IM
45.7
66.4
67.6
41.9
50.3
52,3
24.6
67.6
68.9
4.5 Generalization to Novel Tasks
The generalization ability is the most attractive aspect of generalist models, while the dynamic sub-networks activation of Conditional MoEs should maintain this ability while mitigating task interference. To verify this, we conduct experiments on video caption, video-text retrieval, and natural language understanding tasks, which did not appear in pre-training. As shown in Tab. 5C our Uni-Perceiver equipped with Conditional MoEs could generalize to vidco-related tasks very well. They can obtain reasonable zero-shot performance Uni-Perceiver with a great margin. Moreover, Uni-Perceiver-MooS can achieve comparable results to SOTA methods with similar training cost by further conducting prompt tuning with only 1% data. Beyond that, Conditional MoEs can significantly boost the performance of Uni-Perceiver on GLUE benchmarks (Tab. 5b). owing to its excellent ability to resolve task interference in generalist models.
those tasks and also perform better than vanilla
5 Conclusion
In this paper, we propose Conditional MoËs to address the task-interference issue in generalist models. By sparsely activate sub-networks without introducing any task-specific designs, generalist models can be pre-trained on multiple tasks jointly without performance degradation, while keeping the generalization ablity to novel tasks. We incorporate Conditional MoEs with the recently proposed generalist model Uni-Perceiver With prompt tuning on 1% downstream data, the proposed sparse generalist model achieves competitive performance with previous SOTAs using only <5% training data and <10% training cost. We hope this work can motivate further research in generalist models.
Limitations. Our method is currently verified generalist models with millions of parameters. For generalist models with billions of parameters, whether the iask-interference issue exists and whether our method is still effective questionable, which we leave them to future work.
Potential Negative Societal Impact. This work shares the common negative impacts of large-scale training, which may consume lots of electricity and result in increased carbon emissions. This method also learns from a large number of datasets that may contain data biases.

---

References
[1] A. Abbas and Y. Andreopoulos. Biased mixtures of experts: Enabling computer vision inference under data transfer limitations. TIP. 2020.
[2] Akbari, Yuan, R. Qian, W.-H. Chuang, S.-F Chang, Y. Cui, and B. Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NIPS, 2021.
[3] J. Alayrae, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Milli- can, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
[4] N. Arivazhagan, A. Bapna, O. Firat, D. Lepikhin, M. Johnson, M. Krikun, M. X. Chen, Y. Cao, G. Foster, C. Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.0500 2019.
[5] A. Amnb, M. Dehghani. G. Heigold, C. Sun, M. Lučić, and C. Schmid. Vivit: A video vision transformer. arXiv preprint 1IXXi:2003..501 2021.
[6] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 160.006600 2016.
[7] G. Bertasius, H. Wang, and I.. Torresani. Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05095. 2021.
[8] R. Caruana. Multitask learning. Machine learning, 1997.
S. Changpinyo, P. Sharma, N. Ding, and R. Sorieut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. CVPR, 2021.
[10] D. S. Chaplot, L. Lee, R. Salakhutdinov, D. Parikh, and D. Batra. Embodied multimodal multitask lcarning. arXiv preprint 1rXiv:1902.01383 2019
2019.
[11] D. Chen and W. B. Dolan, Collecting highly parallel data for paraphrase evaluation. In ACL, 2011.
[12] X. Chen, H. Fang, T-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 504.00325, 2015.
[13] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. 2020.
[14] Z. Chen, Badrinarayanan, C.-Y. Lee, and A. Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. ICMI,, 2018
[15] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, Masked-attention mask transformer for universal image segmentation. 2022.
[16] K. Clark, M.-T. Luong, U. Khandelwal, C. D. Manning, and Q. V. Le. Bam! born-again multi-task networks for natural language understanding. arXiv preprint arXiv: 1907: 007.04829, 2019.
[17] M. Crawshaw. Multi-task lcarning with deep neural networks: A survey, arXiv preprint 20090000,, 2020.
[18] Deng, W. Dong. Socher, L.-J. Li, Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
[19] J. Devlin, M.W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004805, 2018.
[20] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint axXiv:2010.11929, 2020.
[21] N. Du, Y. Huang. A. M. Dai. Tong. D. Lepikhin. Y. Xu, M. Krikun, Y. Zhou, A. Yu, O. Firat, et Glam: Efficient scaling of language models with mixture-of-experts arXiv preprint arXiv:1112.06900 2021.
[22] H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint a00iv:2166.1 11097 2021.
10

---

[23] W. Fedus. B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03911 2021.
[24] M. Guo, A. Haque, D.-A. Huang. S. Yeung, and L. Fei-Fei. Dynamic task prioritization for multitask learning. In ECCV, 2018.
[25] K. Hashimoto, C. Xiong, Y. Tsuruoka, and R. Socher. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv: 1611v::611.0088// 2016.
[26] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[27] K. He, G. Gkioxari, Dollár, and R. Girshick. Mask r-cmn. In ICCV, 2017.
[28] C. Hokamp, J. Glover, and D. Gholipour. Evaluating the supervised and zero-shot performance of multi-lingual translation models. arXiv preprint arXiv:1906.09673. 2019.
[29] R. Hu and A. Singh. Unit: Multimodal multitask learning with unified transformer. arXiv preprint arXiv:2 10772, 2021.
[30] T. Iki and A. Aizawa. Effect of visual extensions on natural language understanding in vision-and-language models, arXiv preprint arXiv: 104.000666 2021.
[31] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. lonescu, D. Ding. S. Koppula, A. Brock, E. Shelhamer, O. Hénaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver io: A general architecture for structured inputs & outputs, 2021.
[32] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
[33] C. Jia, Y. Yang. Y. Xia. Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung. Z. Li, and T. Duerig, Scaling up visual and vision vision-language representation learning with noisy text supervision. In International Conference Machine Learning, pages 4904-4916. PMLR, 2021.
[34] H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and T. Zhao. Smart: Robust and efficient fine-tun tuning for pre-trainn pre-trained natural language models through principled regularized optimization. arXiv preprint arXiv: 1911 1911.03377 2019.
[35] S. Kalkowski, C. Schulze, A. Dengel, and D. Borth. Real-time analysis and visualization of the 100m dataset. In Proceedings of the 2015 workshop commmniity-rrganizen multimodal mining: opportunities for novel solutions, pages 25 30, 2015.
[36] M. Kanakis, D. Bruggemann, S. Saha, S. Georgoulis, A. convolutions for incremental multi-task learning without task interference. 2020.
Obukhov, and L. V. Gool. Reparameterizing
[37] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijyanarasimhan. F. Viola. T. Green, T. Back, P. Natsev, et al. The kinetics human action video dataset. arKiv preprint urXiv:1705.06940 2017.
[38] A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In CVPR, 2018
[39] W. Kim, B. Son, and L. Kim. Vilt: Vision-and-languaee transformer without convolution or region 2021.
supervision. arXiv preprini arXiv:2102.083:
[40] Y. J. Kim, A. A. Awan, A. Muzio, A. F. C. Salinas, L. Lu, A. Hendy, S. Rajbhandari, Y. He, and H. H. Awadalla. Scalable and efficient moe training for multitask multilingual models. arXiv preprint arXiv:2 109. 10465, 2021.
[41] R. Krishna, Y. Zhu, O. Groth, J. Johnsson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 123(1):32 73, 2017.
[42] S. Kudugunta, Y. Huang, A. Bapna, M. Krikun, D. Lepikhin, M.-T. Luong, and O. Firat. Beyond distillation: levk mixture-of-expee experts for efficient inferenee. arXiv preprint arxiv:21 2021.
[43] D. Lepikhin, H. Lec, Y. Xu, D. Chen, Firat, Y. Huang, M. Krikun, N. Shazeer, and Chen, Gshard: Scal- ing giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[44] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language image pre- prairiin for unified vision- language understanding and generation. arXiv preprint arXiv:2201.12086. 2022.
11

---

[45] Li, M. Yatskar, D. Yin. C.-. Hsieh, and K.-W. Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:198..00 03557, 2019.
[46] X. Li, X. Yin, C. Li. P. Zhang. X. Hu, L. Zhang. L. Wang. H. Hu, L. Dong, F. Wei, et al. Oscar: Object-semme serenttemm aligned pre-training for vision-language tasks. ECCV, 2020.
[47] Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou. A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Transactions on Neural Newworks and Learning Systems, pages 1-21, 2021. doi: 10.1 200./NNNS..021.30555
[48] Z. Li, W. Wang, E. Xie, Z. Yu, A, Anandkumar, J. Alvarez, T. Lu, and P. Luo. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. 2022.
[49] Z. Lin, L. Wu, M. Wang, and L. Li. Learning language specific sub-network for multilingual machine translation. arXiv preprint urXiv:2005.09259, 2021.
[50] P. Liu, X. Qiu, and X. Huang. Adversarial multi-task learning for text classification. arXiv preprint S.KV:1704.057444 2017.
[51] X. Liu, Zheng, Z. Du. M. Ding, Qian, Z. Yang, and Tang. Gpt understands, too. arXiv preprint arXiv:2103. 10385, 2021.
[52] Y. Liu, M. Ott, N. Goyal. J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 2019.
[53] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. ICCV, 2021.
[54] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint (XXi::908..0266. 2019.
[55] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee. 12-in-1: Multi-task vision and language representation learning. CVPR, 2020.
[56] S. Min, Kong, R.-. Tu, D. Gong, C. Cai, Zhao, C. Liu, Zheng, H. Wang, Z. Li, et Hunyuan_tv tvr for text-video retrivial. arXiv preprint urXiv:2204.03338 2022.
[57] M. Monfort, A. Andonian, Zhou, K. Ramakrishhan, S. A. Bargal, T. Yan, L. Brown, Q. Fan, D. Gutfreund, C. Vondrick, et al. Moments in time dataset: one million vidcos for event understanding. TPAMI, 2019.
[58] V. Ordonez, G. Kulkarni, and T. Berg. Im2text: Describing images using 1 million captioned photographs. NeurIPS, 2011.
[59] B. A. Plummer, L. Wang. C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebrik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015.
[60] D. Qi, L. Su, J. Song, E. Cui, T. Bharti, and A. Sacheti. Imagebert: Cross-modal pre-training with large-scale supe-supe image-text data, arXiv preprint arXiv:2001.0783 07966. 2020.
[61] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. J. Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint orXiv:2103.000 103,000.0000 000000 2021.
[62] S. Reed, K. Zolna, E. Parisotto. S. G. Colmenarejo, A. Novikov, G. Barth-Maron M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Fecles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordhar, and N. de Freitas. A generalist agent, 2022.
[63] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 2015.
[64] C. Riquelme, Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. NIPS, 2021.
[65] J. Shao, S. Chen, Y. Li, K. Wang, Z. Yin, Y. He, J. Teng, Q. Sun, M. Gao, J. Liu, et al. Intern: A new lcarning paradigm towards general vision. arXiv preprint arXiv: 111-21111.8887 2021.
[66] P. Sharma, N. Ding. S. Goodman, and R. Soricut, Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. ACL, 2018
12

---

[67] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-offexperts layer. arXiv preprint arXiv:1701.06336 2017.
[68] N. Shazeer, Y. Cheng. N. Parmar, D. Tran, A. Vaswani, P. Koanantakool, P. Hawkins. H. Lee, M. Hong, C. Young, et al. besh--eensorfoo: Deep learning for supercomputers. NIPS, 31, 2018.
[69] S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.-W. Chang. Z. Yao, and K. Keutzer. How much can clip benefit iision-and-lanuuae tasks? arXiv preprint arXiv: 067.06 2021.
[70] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 1556, 2014.
[71] A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kicla. Flava: A foundational language and vision alignment model. arXiv preprint arXiv:21/2.04480 2021.
[72] T. Standley, A. Zamir, D. Chen, L. Guibas, J. Malik, and S. Savarese. Which tasks should be learned together in multi-task learnng?? In ICML, 2020.
[73] A. Steiner, A. Kolesnikov, X. Zhai, Wightman, J. Uszkoreit, and L. Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXxi:22066..0 100.7000, 2021.
[74] G. Strezoski, N. v. Noord, and M. Worring. Many task learning with task routing. ICCV, 2019.
[75] H. Tan and M. Bansal. Lxmert: I Learning cross-modality encoder representations from transformers, arXiv preprint Xiv: /908: 07490, 2019.
[76] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[77] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I., Jones, A. N. Gomez, F... Kaiser, and I. Polosukhin. Attention is all you need. In NeurlPS, 2017.
[78] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 180.00680, 2018.
[79] P. Wang, A. Yang, R. Men, Lin, Bai, Z. Li, J. Ma. C. Zhou, Zhou, and Yang. Unifying architectures, tasks, and modalities through a simple sequence- sqquence-t--equenee -o-uuuencce learning framework. arXiv preprint or Xiv:2202.0305:
2022.
[80] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[81] X. Wang, Y. Tsvetkow, and G. Neubig. Balancing training for multilingual neural machine translation. arXiv preprint arXiv:2004.0 06748, 2020.
[82] X. Wang, F. Yu, Dunlap, Y.AA. Ma. R. Wang, A. Mirhoseini, T. Darrell, and Gonzalez. Deep mixture of experts via shallow embedding. Uncertainty in artificial intelligence. PMLR, 2020.
[83] 7. Wang, Z. C. Lipton, and Y. Tsvetkow. On negative interference multilingual models: Findings and a meta-learning treatment. arXiv preprint arXiv:2010.03011,, 2020.
[84] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108 109.40 2021.
[85] B. Yang, G. Bender, Q. V. Le, and J. Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. NIPS, 32, 2019.
[86] Z. Yang, Gan, J. Wang, X. Hu, F. Ahmed, Z. Liu, Y. Lu, and L. Wang. Crossing the format boundary of text and boxes: Towards unified vision-language modeling, arXiv preprint arXiv:2111.12085, 2021.
[87] Yu, Z. Wang. V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 2022.
[88] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C, Finn. Gradient surgery for multi-task learning. NIPS, 33:5824-588 5836, 2020.
[89] L. Yuan, D. Chen, Y.L. Chen, N. Codella, X. Dai, J. Gao, H. Hu. X. Huang, B. Li, C. Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.1143,, 2021.
13

---

[90] B. Zhang. A. Bapna, R. Sennrich, and O. Firat. Share or not? learning to schedule language-specilt capacity for multilingual translation. In ICLR, 2020.
[91] 7. Zhang. Y. Shi, C. Yuan, B. Li, P. Wang, W. Hu, and 7.-J. Zha. Object relational graph with teacher- recommended learning for video captioning. In CVPR, pages 13278-13333 13288, 2020.
[92] L. Zhou, H. Palangi, L. Zhang. H. Hu, Corso, and J. Gao. Unified dision-languael pre-training for image captioning and vqa. In AAAI, 2020.
[93] X. Zhu, Zhu, H. Li, Wu, X. Wang, Li, X. Wang, and J. Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. arXiv preprint arXiv:21 2021.
[94] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pages 19-27, 2015.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? (Yes) Our code is released at
wowm:/uunaamnttallissoooooo////a Perceiiee and the used data is publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [No] Most experiments have stable results with little variance.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the open datasets that we use.
(b) Did you mention the license of the assets? [Yes] See Section
(c) Did you include any new assets either in the supplemental material or as a URL? [No] Our code will be released once the paper is accepted.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] See Section 4
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
14

---

(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "vJ8c4KAjsyo", "reviewer": "Reviewer_vm3E", "summary": "This paper proposes a generalist model (ie a model that handles multiple modalities and numerous tasks) based on a Mixture of Experts (MoE). The paper argues that, since current generalist models often share modules across different tasks and modalities, interference among different tasks is responsible for performance degradation when compared to specialist models. To mitigate interference, the proposed method dynamically routes the execution of specific tasks to specialist modules within the MoE. ", "questions": "see Strengths And Weaknesses", "limitations": "see Strengths And Weaknesses", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The proposed approach (a generalist Conditional MoEs) is well motivated. Different tasks can cause interference in gradient directions and lead to sub-optimal performance in individual tasks. Sparse mixture of experts, dynamically allocated based on a task prompt, has the potential to solve this problem while still retaining the ability to perform zero-shot task generalization.\n- The existence of task interference was well-validated through an interesting analysis of image classification, masked language modeling, and captioning. It showed that while image classification and captioning can cooperate when learning lower layers in the model, they interfere with each other at higher layers.\n- The paper proposes several approaches to conditioning the routing mechanism with task-related tokens (ie information that describes the current task and modality). The proposed routing strategies make sense, and they were shown to have an impact on the overall performance. It is was interesting to see that one of the best strategies was to rely on a hard-coded task description (summarized in Table 2). Would new tasks require new descriptions? Would the model benefit from a more generalized task description? Perhaps using learned task embedding, eg using something similar to task2vec ([https://arxiv.org/pdf/1902.03545.pdf](https://arxiv.org/pdf/1902.03545.pdf))?\n- Experimental results (Table 3) show that interference has a detrimental impact on performance (as specialist models always have higher performance than a model which shares all parameters across tasks). The paper also shows that the proposed MoE is able to mitigate the impact of interference, achieving almost the same performance as specialist models on multiple tasks. Having said that, it is still disappointing that there is little evidence of task transfer, as the generalist model does not improve performance in most tasks (except for COCO Captioning B@4). I believe this is still a limitation that should be clearly stated in the paper.\n- Finally, it is impressive that the proposed model was able to achieve competitive performance on several tasks (like object and action recognition, cross-modal retrieval and language understanding with little to no extra data. This ability to do zero-shot or few-shot transfer highlights the benefits of generalist models in general (which the current approach still retains).\n\nIn sum, I believe this is a strong paper. While the ideas of mixture-of-experts and the existence of interference in multi-task learning problems are not new, this paper successfully applies these concepts to generalist models, and achieves good results on several datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XBAEMbgJjju", "reviewer": "Reviewer_LxEk", "summary": "The paper proposes several MoE routing strategies, i.e., token-level, context-level, task-level, modality-level and attribute-level on top of uni-perceiver, to mitigate task interference, showing noticeable improvements. \n ", "questions": "1. Which conditional strategy is exactly adopted in the rest of the experiments? I see statement of `#param is the parameters required during model deployment' in table 4. I wondering if the adopted strategy is data-independent so that we can use only part of the whole parameters to do the inference. \n\n2. What's the training FLOPS of both MoE and non-MoE version of the model? \n\n3. How can such models do zero-shot inference on unseen tasks (in different form, instead of different domain) when task identifier is only id rather than general task representation like instruction? If using attribute condition, the 8-bit attribute seems also less expressive to be general.\n\n4. How is the prompt tuning implemented? If the authors use exactly the same method with prior works, it's recommended to cite them or describe the detail in a simple way.\n\n5. Just curious about how PT_{1%} compares to FT_{1%} and PT_{100%} to FT_{100%}.\n\n6. OFA also reports the result of MS COCO caption using CE optimization only. \n\n\n", "limitations": "The authors have addressed their limitations and potential negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+ Different routing strategies, i.e., token-level, context-level, task-level, modality-level and attribute-level is evaluated. These strategies, though are quite common in their forms, are the first time being evaluated and proved effective in vision-language pretrained models, as far as I know.\n\n\\+ The paper has conducted extensive experiments, including without finetuning (WT), prompt tuning (PT), finetuning (FT), and zeroshot. All showing satisfactory results given its number of params and training data. \n\n\\+ The task interference problem is tested and shown as a clear motivation of using MoE.\n\n\\- The technical novelty is still somehow limited though it demonstrates effectiveness.\n\n\\- Some details of experiments are missing. e.g., It's not clear which conditional strategy is adopted at default, in all the experiments, it's not clear how the prompt tuning is implemented.\n\n\\- The term zero-shot in this paper seems to indicate data of unseen domain, since task id alone does not make it possible to achieve zero-shot inference for an unseen task in a totally different form.  \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OwkX5N01Emb", "reviewer": "Reviewer_RZvn", "summary": "The paper presents an architectural improvement for attention based large scale task models. They start from the uni-perceiver, an attention based model to learn multimodal representations that is amenable to few-shot generalization via a prompting mechanism. The uni-perceiver has a token based modality conditioning and they 1) they introduce more flexible conditioning and 2) use that conditioning as inputs to a gating mechanism applied to the weights attention and MLPs in the uni-perceiver model. This is motivated by insightful but unsurprising experiments showing that in multi-task learning gradients coming from different tasks can have opposite directions thus specializing weights using a gating mechanism is meant to mitigate this effect. They show a broad set of experiments to demonstrate benefits of the new conditioning and the pretraining and generalization benefits of the approach. ", "questions": "- the details on the gating could be improved. how do you train the gating ? is this more expensive in training than in inference ? Do you find collapse there or spurious gating solutions ? do you need to do something prevent these solutions ?\n\n- how many experts are you using ? There must be a tradeoff between the number of experts and optimization ie with many experts less gradients are propagated through the other weights. do you have any insights that might help the readers ?", "limitations": "yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strenghts:\n- the CMoE idea and validation seems well executed\n- the attribute routing seems reasonable and novel\n\nweaknesses:\n- some details are missing (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4O69Xmlb9U", "reviewer": "Reviewer_KJqa", "summary": "This paper proposes to augment the previously proposed Uni-Perceiver model with mixture-of-experts (MoE) routing so that parameters of a model trained on several tasks can specialize by task and data modality. This choice is motivated by appealing to the phenomenon of task-interference, where the contributions to the gradient updates from different tasks for shared parameters do not perfectly align. The paper explores several ways to parameterize the MoE routing, and settles on a strategy that learns to route based on a featurization that encompasses task, token, and modality type. The resulting architecture is evaluated against ablations and strong baselines on several image, video, and text domains.", "questions": "- Does interference (as discussed in Table 1) correlate with task performance? It’s unclear that gradients of different orientations between tasks is evidence of difficulty optimizing their average, as claimed in L34-36.\n\n- Does the proposed MoE block reduce the average interference metric discussed in Table 1 and used to motivate the proposed system? This is used to motivate and justify the claims at several places in the text (e.g. abstract, intro, section 2, sections 4.5), but no evidence is presented for changes to this effect.\n\n- The MoE routing relies on a hand-designed featurization that appears to be unlikely to scale beyond a small number of tasks and datasets (as opposed to the token-level routing, which is very general). How does token-level routing perform outside of the ablation study shown in Table 3?\n\n- The paper does not mention the Perceiver architecture by name, but only to the subsequent Uni-Perceiver architecture. This is historically misleading.\n\n- L298: “we propose Conditional” -> “we propose Conditional MoEs”\n\n- Footnote under L258: \"konw\" -> \"know\"", "limitations": "n/a", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed MoE input featurization appears to typically improve performance when plugged into the Uni-Perceiver model. \n\n- The MoE strategy used is more flexible than strict per-task weight sharing, while relying on similar amounts of knowledge about the input domains and tasks. \n\n- Appendix Table 10 provides a very useful overview of the training protocols of a variety of specialist and generalist models.\n\n**Weaknesses**\n\n- It’s unclear that task interference (as characterized in section 3.1) either limits performance by generalist models or (if it does) that it’s addressed by the proposed MoE mechanism. Task interference is central to the claims and justification of the model, so evidence that this effect is a limiting factor is essential. See **Questions** below for more details. \n\n- The results in Tables 4 and 5 are very hard to interpret, as they contain dozens of numbers with no clear evidence of overall improvement and no comparisons at matched performance in terms of wall-clock training, FLOPs, or other computation or memory measures.\n\n- The proposed model doesn’t appear to strongly outperform any of the baseline models or open up new applications domains (it is applied only on standard tasks that are well-addressed by specialist methods or by other generalist methods that do not appear to require solutions to the proposed task-interference problem). What is the use-case for this model?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept