M$^3$SAT: A Sparsely Activated Transformer for Efficient Multi-Task Learning from Multiple Modalities
Authors: Jie Peng
Tianlong Chen
Ruida Zhou
Jianmin Ji
Yanyong Zhang
Zhangyang Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-task learning
multimodal learning
transformer
mixture of experts

Abstract: Multi-modal multi-task learning (M$^2$TL) aims to discover the implicit correspondences among heterogeneous modalities and tasks, which is common in real-world applications like autonomous driving and robotics control. Current single-model solutions for M$^2$TL usually fall short in several aspects. The shared backbone between the modalities is prone to overfitting the simpler modality, while jointly optimizing the tasks suffers from unstable training due to the gradient conflicts across tasks. On the other hand, designing a separate model for each task and modality can avoid the above problems but leads to prohibitively expensive computation and memory consumption, rendering this approach unrealistic.

In this work, we propose M$^3$SAT, a sparsely activated transformer for efficient M$^2$TL. The proposed framework tailors the mixture-of-experts (MoEs) into both the self-attention and the feed-forward networks (FFN) of a transformer backbone. It adopts the routing policy to assign attention-heads and FFN experts during training, which effectively disentangles the parameter space to prevent training conflicts among diverse modalities and tasks. Meanwhile, disentangled parameter space also restrains the problem of simple modal prone to overfitting. Sparsely activating the transformer also enables efficient computation for each input sample. Through comprehensive evaluation, we demonstrate the effectiveness of our M$^3$SAT: a remarkable performance margin (\textit{e.g.}, $\ge 1.37\%$) is achieved over the dense models with the same computation cost. More importantly, M$^3$SAT can achieve the above performance improvements with a fraction of the computation cost -- our computation is only $1.38\% \sim 53.51\%$ of that of the SOTA methods. Our code will be released upon acceptance.

Under review as a conference paper at ICLR 2023
S A SPARSELY ACTIVATED TRANSFORMER FOR EFFICIENT MULTI-TASK LEARNING FROM MULTIPLE MODALITIES
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-modal multi-task learning (M²TL) aims to discover the implicit correspon- dences among heterogeneous modalities and tasks, which is common real-world applications like autonomous driving and robotics control. Current single-model solutions for M usually fall short in several aspects. The shared backbone between the modalities is prone to overfitting the simpler modality, while jointly optimizing the tasks suffers from unstable training due to the gradient conflicts across tasks. On the other hand, designing separate model for each task and modality can avoid the above problems but leads to prohibitively expensive com- putation and memory consumption, rendering this approach unrealistic.
In this work, we propose SAT, a sparsely activated transformer for efficient M"TL. The proposed framework tailors the mixture-of-experts (MoEs) into both the self-attention and the werward networks (FFN) of transformer back- bone. It adopts the routing policy to assign attention- and FFN experts dur- ing training, which effectively disentangles the parameter space to prevent training conflicts among diverse modalities and tasks. Meanwhile, disentangled parameter space also restrains the problem of simple modal prone to overfitting. Sparsely activating the transformer also enables efficient computation for each input sam- ple. Through comprehensive evaluation, we demonstrate the effectiveness of our SAT: remarkable performance margin (e.g., 2 1.37%) is achieved over the dense models with the same computation cost. More importantly, S can achieve the above performance improvements with fraction of the computation cost - our computation is only 1.38% ~ 53.51% of that of the SOTA methods. Our code will be released upon acceptance.
INTRODUCTION
Recently, multi-modal machine learning models have shown effective in several domains, mainly including image, language and audio understanding Ramesh et al., (2022); Saharia al. (2022); Agrawal et al. (2017); Yang et al. (2016); Wang et al. (2022). As the need of understanding our surroundings keeps rising, new sensing modalities that go beyond these domains need to be deployed and incorporated in multi-modal learning.
To illustrate, let consider an example autonomous vehicle system. Nowadays, autonomous ve- hicles are equipped with different types of sensors to ensure the viable perceptual capability under adverse conditions such rain, haze, and snow. Therefore, performing multi-modal perception by fusing the data from these sensors has become a necessity. For example, [Jannni et al. (2022) uses the eye blink sensor and photoplethysmographh sensor for fatigue detection, [Li et al. (2022) uses RGB camera, LiDAR and millimeter (2220) uses the RGB camera and LiDAR for drivable area detection, and Han et al. (2022) uses the RGB camera and LiDAR for collision avoidance.
radar for 3D detection and tracking, Raguraman & Park
In addition, an autonomous vehicle system usually needs to perform a large number of tasks con- currently, including fatigue detection Nemcova al. (2021), 3D object detection and tracking Li (et al. (2022), lane detection Gao et al. 2019)) and local planning Isele et (2018). etc, which poses challenges the underlying system.
For example, autonomous vehicles usually move speed

---

Under review as a conference paper at ICLR 2023
between 60 120 km/h, forcing most of these tasks to at high frequency (e.g.. 10Hz~60Hz or higher). The fact that autonomous vehicles usually have limited computation resources suggests that each task needs finish within pre-set time, and that cannot afford to load different task models when switching tasks.
Multi-modal multi-task learning (MPTL) Liang al. (2022): Hu & Singh (2021) aims solving multiple multi-model tasks simultaneously with a single model. However, challenges from both multi-modal learning and multi-task learning hinder us from building an effective M model. Firstly, multi-modal networks are often prone to overfitting with different modalities overfitting different rates, and thus naively training them together only sub-optimal Wang al.) (2020). Secondly, training multiple tasks within a single model often results in tasks that compete for modal capacity since the same weights might receive conflicting update Lirections(Chen et al. (200bb) Fifty et al. (202T). Notably, we assume that the intelligent system often only requires small number of tasks simultaneously, and each task only involves a subset of all the modalities. For such a system, the "fully activated" model is heavily redundant and hard to scale. For example, Singh et al. (2022): Hu & Singh (202T) has to activate massive transformer-
network for each task, with each modality using& distinct transformer encoder. Thus, the backbone network grows with the number of modalities and tasks, the inference latency of each task becomes catastrophically long.
To tackle these bottlenecks, propose the Multimodal Multi-task Sparsely Actived Transformer (MSSAT) which organically adapts the mixture of experts (MoE) Riquelme et al. (2021 : Lepikhin et al. (021) for efficient M tasks, as MoE can adaptively divide-and-conque the entire model capacity into smaller sub-models [hazzeer et al. (2017); Kim et al. (2021b). We train the routing policy within our backbone select the subset of experts for each input token. In the training stage, the load and importance balancing loss prevents the feature tokens from being always put into the same expert, and thus distributes the parameter updating of the specific modality to different ex- perts. This can effectively restrain the easy modality from the overfitting problem. Meanwhile, the routing strategy separates the parameter spaces, which can balance feature reuse and avoid training conflicts among tasks. In fact, vanilla MoF already disentangles the parameter spaces of the FFN network; however, we find that these experts with separated parameter spaces are still insufficient to handle multiple multi-modal tasks. Therefore, the SA adopts the MoE into the feed-forward network (FFN) and self-attention modules of the vanilla transformer encoder backbone. By further untangling more parameters into distinct parameter spaces of the transformer backbone, the M achieves better restrains the simpler modalities from overfitting and alleviates the gradient conflic- tions between different tasks. During the inference stage, the S only activates those experts corresponding to the necessary modality/task instead of the entire model. As such, the highly sparse active transformer achieves efficient inference for the specific modality and task.
To verify the effectiveness of our SAT we conduct comprehensive evaluation on MultiBench, a large-scale benchmark spanning more than 10 modalities, and testing for 20 prediction tasks across 6 distinct research areas. Our model surpasses the performance of the state-of of-te-of-the-ar art (SOTA) multi- modal multi-task model on the MultiBench. Meanwhile, our computation cost is 1.38% 53.51% of the computation cost of the current SOTA multi-modal multi-task model on MultiBench.
Our main contributions are outlined below:
We target the problem of efficient multi-modal multi-task learning and propose the first multi-modas multi-task mixture of expert model.
We engage MoE achieve the following three goals: (1) solving the training conflicts among tasks, (2) restraining the easy modality from overfitting, and (3) sparsely activating paths for single-modality and single-task inference.
We demonstrate remarkable performance improvements over dense models with equiv- alent computational cost and outperform current multi-task state-of-thh-aar performance with only 1.38% 53.51% % of their computational cost.
2 RELATED WORK
Multi-modal and Multi-task Learning. There has been a long history of work multi-modal and multi-task learning. On the one hand, most previous efforts on multi-task learning Strezoski et al. (2019); Zamir et al. (2018): Spgaard & Goldberg (2016); /Hashimoto et al. (017) focus
2

---

Under review as a conference paper at ICLR 2023
specific domains modalities, such as language and vision understanding. MaTL Strezoski et al. 2019) enables structured deterministic sampling of multiple for multiple vision tasks. (Søgaard & Goldberg (2016) design an MTL model with bi-RNNS for vi- sion tasks. On the other hand, recent work on multi-modal learning prefers the Transformer-basee model to learning general-purpose models
within a single modal
two three modalities, typically in the language, vision, and audio Ramesh et al.] (2022): Saharia et al.] (2022); Agrawal et al. (2017): Yang et al. (2016); Dai al. (2022). Base on the vanilla text-based Transformer model Vaswani et al.) (2017). many multi- modti extensions typically use full self attenionn over modalities concatenated across the sequence dimension Su et al. (2020); Chen et al. (2020a) or a cross-model attention layer Tan & Bansal (2019); Tsai et al. (2019). Several works such as Perceiver Jaegle et al. (202I), ViT-BERT Li et al. 2021). PolyViT P Likhosherstov et al. T) have investigated the potential of using the same unimodal encoder architecture for different modalities. Moreover, multiple works have endeavored to build a single model that works well on multiple mullti-moda tasks (i.c., multi-modal multi-task learning) SSu et al. (2020); Cho et al. (202T); Hu & Singh (202T): Lu et al. (2019); AAkari et al. 22221. Akbari et al. (2021) introduces a shared model on video, audio, and text data to per- form audio-only, video-only, and image-text retrieval tasks. VLBERT (Su et al. (2020) investigates a simple yet powerful pre-trainable generic representation for visual- visual-linguisic tasks. Unit Hu & Singh (202T) uses single model for several asiionaaddlaanguage tasks. HighMMT [iang et al. (2022) goes beyond the commonly studied language, vision, and audio modalities to relatively more affluent modalities such as tabular, time-series, sensors, graphs, and set data, In addition, High-
MMT investigates single model to process the above modalities for multi-task learning, and each task involves only parts of the above modalities, which is what we are concerned about.
MoE and Conditional Computation. Sparsely activated mixture of expert (MoE) models have recently been used with great effect to both vision Riquelme et al. (202T); Lou et (202T) and lan- guage [eeiikhnn guage[Lepikhin
al. (2021):] Kim et (l.2 models. MoE contains a series of sub-models (i.e.,
experts) and performs the conditional computation an input-dependent fashion. Several pioneer investigations explore MoE for multi-task learning Ma et al. (2018); Aoki et al. (2021); Hazimeh iet al. 2002); Kim et al.] (202 and multi-modal learning Kudugunta et al.] (2021); Mustafa et al. (2022). which related this work. Particularly, Ma et al. (2018); Aoki et al. (2021); Hazimeh et al. (202T) investigate task-specific gating networks to choose different sub-models for each task solve recommendation system Ma et al. (2018), medical signal process (Aoki et al.) (202T), and
digital number recognition Hazish) et al. (2021) multi-task learning. Kim et al. (202 propose single gating network MoE for supporting training large-scale multi-task multilingual models. MMstafa et al. (2022) use single gating network to allocate tokens in
Hazimeh
fashion for multi-modal contrastive learning. |Fedus al. (2022) investigates MoE into the attention layer for NI.P task, Zhu et al. (2022) further explores different types of routers in the FNN layer and the attention layer for multi-modal learning. We integrate the MoE to further raveling parameter spaces of the transformer backbone for the M
METHODOLODY
We first describe the overall architecture of our SAS as shown in Figure | and then present the proposed Sparse MoE design for multi-modal, multi-task learning.
3.1 MULTI-MODAL MULTI-TASK MODEL DESIGN
Input Data Preprocessing. Each modality is treated as sequence. And then the modality- specific Fourier positional encoding and one-hot modality encoding are applied to integrate tem- poral/positional information and modality information into the input sequence of embedding. We refer (i) the details of the processing for each modality sequential data, (ii) the Fourier positional encoding setting for different modalities, and (iii) the one-hot modality encoding to Appendix [..]
Unimodal Encoder. After the input-data pre-processing, we would receive the sequential tokens of different modalities, in which the feature dimension of all modalities is the same. A transformer- based Perceiver Jlock et al. (2021) is adopted to convert each modality sequence to sequences with the same length. Note that only one copy of the Transformer based Perceiver Block is used for all modalities and tasks. Moreover, the process that shares the parameters of unimodal encoder
3

---

Under review as a conference paper at ICLR 2023
Multi-task training
Image
Task
Task
Task 3 Head
Task
Audio
M.SAT Fncoder Layer)
Dense Rncoder Layer
Image
Unimodal Encoder
Spurse FFN
Task
Video
Norm
SAT
Encoder
Layer
Time <eriis
Sparse Moß Sell-Altention
Time -series
Task
Modality
embedding
Table
Image
Audio
Video
Time-series Table
Tokea
Token
Token
Token
Token
Figure Our model first standardizes each input modality into a sequence and uses modality- specific embedding layers capture the modality-speciff information. Then the uni-modal encoder layer converts each sequence to sequences of the same length. We concatenate these modality tokens on the sequence dimension within each task and input them SAT encoder layers for multimodal multi-task learning. These S encoder layers perform efficient modality information fusion, eliminate the training conflicts among tasks, and control easy modality avoid overfitting.
across different modalities also allows to get rid of setting specific modality encoder for each modality. The details of the Transformer-based Perceiver Block can be found in Appendix A.2]
Consecutive Transformer Encoder with MoE. So far, receive the multi-modal tokens for each task, for which the sequence length of each modality is the same, and the feature dimension of each modality token is the same. After concatenating these modality tokens on the sequence dimension within each task, we put these tokens into several consecutive transformer encoder layers. Our proposed SASAT encoder layer and the vanilla transformer dense encoder layer compose these transformer encoder layers. Specifically, the M SAT encoder layer replaces the self-attention layer and the feedforward network (FFN) layer of the dense encoder layer with corresponding sparse MoE layers. The ST encoder layer is introduced in Section 3.2 and the detailed configuration of the SAT layer and other training setups are provided in Appendix A.3
Task-specific Head and Multi-task Learning. Finally, we use linear layer with normalization per task for task- tase-spec learning. Our optimization objective is minimizing weighted sum losses for multiple tasks.
3.2 MULTI-MODAL MULTI-TASK MoE FOR SAS
We first describe the standard MoE, show the proposed Sparse MoE Self-attention, and then present the multi-router version MoE that consists of standard MoE and Sparse MoE Self-attention.
Mixture of Experts Layer. Mixture of Experts (MoE) layer typically consists of a group of N experts f1, .e.... fN together with routing network (also called router) to select appropriate experts. The experts usually use multi-layer perceptrons transformer based models (Riquelme) al. 202T). We inherit the router design from V-MoF (Riquelme et al.! 2021). For input x, the
4

---

Under review as a conference paper at ICLR 2023
SAT
Fnonder
Layer
Spare Man
T Linear
Self-atiention
Sparse MoE
Spete
Equer
Multi-Hou Self-amention
Riyert
F
Eipert
Reak
Key sta
Hatt.
Self-atiention Moł for Value/KeyQQurr
Sparse
Figure 2: The detailed architecture of the SAT Encoder Layer: the Sparse MoE Self-attention and Sparse MoE FFN. The Sparse MoE Self-attention consists of three Self-attention MoE to compute the value, key, and query, respectively. Note that, the expert of the Self-attention MoE is composed of single linear layer. The Sparse MoE FFN is the same as the vanilla MoE layer.
output of MoE layers selects the top K experts through a router R, depicted as below:
K
Σ R(x); fx((),
(1)
k=1
R(x) TopK (so tooofttaa K),
(2)
TopK K) [v if is in the top K elements
(3)
otherwise
where Gate(.) represents the learnable network within the router, for which we employ a single lin- ear layer without bias practice. The so softmazt (tmar(-) and TopK (:, K) work together to set all vector elements to zero except the elements with the largest K values. To avoid always routing the same experts while ignoring others, we employ the load and importance balancing loss following (Shazeer al. (2077). We list the settings of K and N for different tasks group in Appendix A uses the vanilla sparsely activated MoE in the FFN layer.
The SAT
Sparse Self-attention MoE. We first revisit the definition of the original Self-attention layer. The Self-attention layer is mainly used to compute the self-atenntion of input tokens. The scaled-dot product computes the self-attention:
QK"
Attention Q K, V) = softmart
V
(4)
VC
where K, V E SXC are the query, key, and value matrices computed by three linear layers from the input tokens; S and denote the sequence length and the hidden dimension. These three linear layers for computing query, key, and value use the same architecture but different parameters. In our proposed Sparse Self-attention MoE, we integrate MoE into these three linear layers further disentangle parameter spaces, which are displayed on the left side of Figure 2 For each attention MoE:
K
R(x); (x).
(5)
k=1
where these expert candidates f are shared across modalities and tasks. Unlike vanilla MoE, the expert f (-) is a single linear layer where the input and output dimensions are the same as the hidden dimension. Each expert of vanilla MoE is computed with W20 activation Hendrycks & Gimpel (0016). Wi and W2 are two learnable weight matrix. The Sparse MoE Self-attention layer expert is computed with Wa,, where the W is the learnable weight matrix for calculating key, query. and value for self-attention. Note that, unlike Fedus et al. (2022): Zhu al. (2022), three independent routers to router tokens for q. k, and separately.
where 6geiv is the GELU
5

---

Under review as a conference paper at ICLR 2023
Multi-router version MASAT
Encoder Layer
Spare Man
T Linear
Self-atiention
Sparse MoE
Molti-mmater Sparse MoF: FFN
Spunc Schookizrecs
Equat
Dijan
Multi-Hedd Attention
Riyent
Eipert
Multi-router Artention
Value/Key/Query Mułi
Milloooa Valan Salil Rez SA4
Vadi Quer Hit
...
Reasri
Rester
Task/Moodaity
Boar
Specific
Figure 3: In the multi-router version of SAT encoder layer, we allow the Self-attention MoE and the Sparse MoE FFN of our SAT encoder layer to use task sse-sspccfiic router network modality- specific router network. The task-specific router indicates that each task owns its router network, and the modality-specific router indicates that each modality owns its router network.
Task/Modality Specitic Multi-router MoE. We notice several works Ma et al. (2018); Aoki et al. (202T): Hazimeh et al.] (202T): Kim et al. (202Ia) investigate task-specific rout- ing networks for multi-task learning. This paper takes step further propose the task/modality specific multi-router MoE to study the benefits of the multi-router design. Formally, define the output of our MoE layer follows:
K
V/ = - fx(x,,
(6)
k=1
where is the routing network index and can be set as task index or modality index. The ex- pert f-(-) can be either single linear layer (used in the Sparse MoE Self-attention layer) or an FFN layer (used in the Sparse MoE FFN layer). All of these experts are shared between different modalities and tasks. Both the Sparse MoE Self-attention layer and FFN layer can use the task-specific or modality-specioe router S i) the Multi-router S uses modality-specifii routing networks the Sparse MoE Self-attention layer and task-specific routing networks in the Sparse MoE FFN layer. ii) the R-Multi-routee SAS 'reverses' settings of the Multi-router SAT which uses modality- specific routing networks in the Sparse MoE FFN layer and task task-specific routing networks in the Sparse MoE Self-attention layer. Mcanwhile, we also use modality-specific routing net- works (P-Modaliyy-routee MASAT task-specifie routing networks Ta-Task-rotter & along both in the Sparse MoE Self-attention layer and the Sparse MoE FFN layer. The backbone model parameters of the S and these versions of multi-router M do proportion- ally increase if we involve more modalities and tasks in training. We show the details of the task/modality specific multi-router MoE Figure The effects of the multi-router MoE are cluded in Section 4.3 and Appendix B
router. Therefore, we design four versions of multi-
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
To evaluate the proposed method, we conduct experiments on the MultiBench, large-scale multi- modal multi-task benchmark involving more than modalities and testing for 20 prediction tasks across 6 research areas. We choose 7 tasks in MultiBench and train 3 multi-modal multi-task models across combinations these tasks in Table T (please see Appendix [for more experimental details).
Evaluation Metrics. We the standard evaluation metrics provided by MultiBench Liang et al. (2021). Following Vandenhende et al. (2022), we use A to evaluate our M model as the average per task drop with respect to the HighMMT model b over all tasks: A

---

Under review as a conference paper at ICLR 2023
Table 1: We follow the setting of HighMMTT Liang et al. (2022) which uses 3 multimodel multi- task training to evaluate the performance of the SAT. These setups include tasks with different modality inputs, predicting objectives, research areas, and dataset size.
Setting
Dataset
Modalities
Prediction Task
Research Area
Size
PUSH
amage..orce..ggg www.:e.ooceeeecccccc.cc. wwwprioccption.cco.. control
object pose contact
Robotics Robotics
37,990
Small
V&T
imagee.orce.prrorrocccee pttpe//rrre.ppppppppiooeeeellll
147,000
ENRICO
image,set
design interface
HCC
1,460
Medium
PUSH
image, forceeeorreeeree
object pose digit humoor sentiment
Rohotics
37,990
AV-MNIST
image.audio
Multimedia
70,000
YR-FUNNY
vide,
Affective Computing
16,514
MOSEI
text, widco.idddiiii
Affective Computing
22.777
Large
MIMIC
sabie image,audio
ICD-9 codes
Healthcare
36.212
AV-MNIST
digit
Multimedia
70,000
Table 2: We compare the performance of model, HighMMT (the state-of-the-art multi-modal multi-task learning method on the MultiBench benchmark), and all the 20 models implemented in the benchmark for in different training settings. We show that our model outperforms the HighMMT model in most tasks.
setting
V&T1
A(%)
Params (M) Flogs
MulriBench Models HighMMT
0.574-0.290
93.30-93.60
2.03-24.70
5.20-25.08
0.445
96.10
0.00
0.85-0.89
5,14-32.48
96.33
0.25-0.25
2.59-17.38
Medium setting
ENRICO 1
PUSH J.
AV-MNIST
A(%) f
Params (M)
Flops (G)
MultiBench Models HighMMT
44.40-51.00
0.574-0.290
65.10-72.80
21.20-51.70
0.25-314.10
53.10
0.600
68.48
0.00
0.52-0.63
0.95-79.48
Ours
71.58
0.475
71.86
20.19
1.23-1.25
0.41-2.33
Large sctting
UR-FUNNY
MOSEI f
MIMIC 1 AV-MNIST
A(%) f
Params (M)
Flops (G)
MultiRench Models HighMMT
58.30-66.70
76.40-82.10
67.6-68.9
65.1-72.8
0.41-37.7
0.25-10.03
62.00
78.40
65.60
70.60
0.0
0.52-0.52
0.67-1.65
Ours
64.24
79.47
67.91
71.05
2.28
0.76-0.76
0.15-0.53
1 (-1) Mo.i//Mo.. where M, the metrics of task i, and 4, = -1 if a lower value means better performance. M results HighMMT are running by their released code and training configuration. Training each task group takes about 12 24 hours for HighMMT and our S model. Therefore, the performances of these tasks for HighMMT and S that report in this paper the mean of times repetitions. For the min and max performance of MultiBench in Table we report numbers directly from the MultiBench paper.
T
Configuration Details. We display our model overview architecture in Figure and the architec- ture design details of S we proposed in Figure 2 We conduct all of our experiments on the NVIDIA A30 Tensor Core GPU. Please refer to Appendix A.. A..3 for more details on network config- uration and training setup.
4.2 PERFORMANCE COMPARISON S WITH EXISTING MULTIMODEL MODELS
In Table we compare the performance of our model with the current SOTA model High- MMT et al. (022) as well as 20 recent multimodel models that are implemented in Liang al. (021). The results show that our method outperforms the HighMMT on all tasks under all three settings ++12.93%/420. 19%/42.28% M performance, respectively). Notably, the 'ENRICO' performance of SA is even significantly higher (+20.58% single-task performance) than the best performance of MultiBench, which sets a new state-of of-the-rrt result. Meanwhile, the SAT only uses 1.38% 53.51% of the computation resources compared to HighMMT. For the large setting, although the number of parameters ST is larger than HighMMT, the computation resources (Flops) used are still much smaller.

---

Under review as a conference paper at ICLR 2023
Table Comparison of routing networks. To explore the effects of different routing networks, we consider the influences of task-specific routing networks and modality-speciife routing networks in the self-attention layer and the FFN layer separately. We also investigate the combinations between the multi-routing network and the dingle-routing networks in Appendix B
Model
ENRICO f
PUSH L
AV-MNIST 1
A(%)
HighMMT multitask
53.10
0.600
68.48
0.00
M (ours)
71.58
0.475
71.86
20.19
Multi-gate MAS
71.00
0.684
71.03
7.81
R-Multi-gate S
64.38
0.995
71.33
-13.48
P-Modality- SAT
68.72
0.786
70.70
0.54
P-Task-gate SAT
68.38
0.833
70.69
-2.25
Dense Model
65.98
1.342
70.49
-32.14
w/o Attention MoE
69.06
1.227
70.26
-23922
w/o FFN MoE
68.84
0.818
70.94
-1.02
4.3 DETAILED INVESTIGATIONS OF SAT
Ablation Study: single-router vs. multi-router. We notice that earlier works such as Ma et al. (2018); Aoki al. (202T); Hazimeh et al.] (202T); Kim al. (202Ta) have investigated task- specific routing networks in learning the routing policy individually for different tasks in MTL. Therefore, we next explore the following question: What kind of routing network is suitable for M To answer this question, we compare our S with the Multi-router SAT the R-Multi-router M the P-Modality-routee SASAT, and the r-Task-router SAT We also investigate more detailed experiments in Appendix Our final results show that the single rouler better suited for MLLL
with MoE vs. w/o MoE. We further analyze the MoE components of S in 3Jale|3annd Appendix B.I The Dense model is the model that uses the same computation cost with the S but without any MoE components. The w/o Attention MoE leaves out the Sparse MoE Self-attention and uses the original self-attention layer. The w/o FFN MoE replaces the Sparse MoE FFN with the primary FFN layer. Compared with the dense model, S achieves noticeable performance gain (e.g., 2 1.37%), which shows that the M benefits from the MoB.
Ablation Study:
expert number and selection number. For the MoE layer, the number of se- lected experts per token K and the total number of experts N are two of the most significant hyper- parameters. Due to the limited space, we show the detailed performance in Appendix C.2
Ablation Study:
In-Depth Discussion: MTL. We measure the following two metrics explain the reason for obtain- ing MoE successfully from the multi-task learning (MTL) view: the gradient positive sign purity and the inter-task affinity. The gradient positive sign purity Chen et al. (202bb) (GPSP) measures how many positive gradients are presented in a network parameter at any given value. P is bounded by [0, 1]. The value of ' close to 0 or indicates that the gradient conflict of MTL. has less effect the corresponding parameter. In Figure 4 we discretize P into intervals and then count the number of parameters that fall within these intervals. We record the GPSP distribution of the MASAT MASAT without MoE on self-attention, M without MoE on FFN, and the equal computation dense model. The inter-task affinity Zi- defined by Fifty al. (202T) indicates the influence of the parameter update of task i on task j. The higher value of Z;-+j indicates the update on the parameters positive for task j. while lower value of Z;-. indicates that the parameter update is antagonistic for task j. For the medium setting, in the right part of Figure 4 we record the inter-task affinity of the 'ENRICO" task to the 'PUSH' task of the MASAT, the multi-router SASAT, and the equal computation dense model.
Compared with other models, the GPSP of M is accumulated more in intervals [0.6, 0.8] and J0.8, 1.0]. which shows by splitting the parameter space, only fraction of the conflict parameters are running for specific tasks. The inter-task affinity of S and multi-router M is higher than the dense model most of the time, which shows that MoE can restrain the gradient conflict of MTL. For more details on the GPSP and the inter-task affinity, please refer to Appendix C.5 and Appendix C.6
8

---

Under review as a conference paper at ICLR 2023
0.7
Inter: Task Affinity
Parameter Ratio (%)
-0.5
0.33
We Daxie mokk
0.33
-1.5
0.5
IUS-11
-2.5
0.4
-3.5
0.2
-4.5
MSSAT
Malli-mote MASAT
0.1
-5.5
-6.5
0.2.0.4
0.4.0.6
40
70
The Gradient Positive Sign Purity
Training Epochs
Figure 4: The distribution of The Gradient Positive Sign Purity(left), and the inter-task affinity of the 'ENRICO' to the 'PUSH' task (right).
Table 4: The optimal gradient blend for each tasks under different model architectures.
Model
ENRICO
PUSH
AV-MNIST
image
set
image
force
proprioception
control
image
audio
SAS
0.48
0.52
0.00
0.37
0.32
0.31
1.00
0.00)
Dense Model (w/o MoF)
0.61
0.39
0.00
0.36
0.32
0.31
1.00
0.00
w/o Self-attention MoE
0.63
0.37
0.00
0.37
0.32
0.30
1.00
0.00
-w/o FFN MoB
0.75
0.25
0.00
0.37
0.32
0.32
1.00
0.00
multi-gate SASAT
0.71
0.29
0.00
0.35
0.32
0.32
1.00
0.00
In-depth Discussion: multi-modal learning. From the perspective of multi-modal learning, the optimal gradient blend (OGB) defined by Wang et al. (2020) indicates which modality is easily prone to overfitting (the smaller the value, the easier the modality is prone overfitting. For multi-
task with M modalities, the OGB is bounded: w [o, 1] and Σ modality index. The greater the difference between the modality OGB values within a single task, the more serious the overfitting problem for the modalities with smaller OGB values. Table 4 we present the optimal gradient blend of the trained models under different MoE settings. For PUSH and AV-MNIST tasks, the overfitting problem still exists. However, S alleviates the problem in the ENRICO task. For more details on the optimal gradient blend, please see Appendix C.7
M wogh 1, where m is the
In-depth Discussion: Expert distribution. We also explore how routing is distributed across dif- ferent modalities and tasks. Due to the limited space, show the routing distribution under testing data for different modalities and tasks of the medium setting in Appendix C.8
5 CONCLUSION AND LIMITATION
This paper proposes a sparsely active transformer model for efficient multi-modal multi-task learn- ing. By tailoring the mixture-of-experts into both the self-attention and the feed-forward net- works of a transformer backbone, we achieve the following. Firstly, we sparsely active experts in the self-attention and the feed-forward networks in training to restrain easy models from be- ing overfitting and mitigating MTL gradient conflicts. Secondly, given any task and corresponding modalities, we can only activate the sparse 'expert' pathway for efficiency. Comprehensive experi- ments show that the proposed ST surpasses the SOTA with a fraction of the computation cost (+ (+1.933/4200.199 11.22...99/22.889 M performance); our computation cost is only 1.38% 53.51% of the SOTA model. Our experiments on MoE also provide rational perspectives for designing multi-modal multi-task learning neural network architectures.
The limitation of our work is that the proposed SAT ing forward, we will evaluate SASAT on more practical tasks like in-door robots and autonomous vehicles in future work. Also, we expect to expand our model size for larger scale tasks and more kinds modalities in future work.
AT is only evaluated on academic datasets. Mov-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. VQA: visual question answering. Int. Comput. Vis., 123(1):4-31, 2017.
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Bo- qing Gong. VATT: transformers for multimodal self-supervised learning from raw video, audio and text. In Mare Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, viriual, pp. 24206-24221, 2021. URL 0a2p0//0000711111111360264444453
nwwrips.ep... cC/ .html
Raquel Aoki, Frederick Tung, and Gabriel L. Oliveira. Heterogeneous multi-task learning with expert diversity. CoRR, abs/2106.1055 10595, 2021. URL sttpp://arxxv..og//abb//////
10595
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, vol- ume 12375 Lecture Notes in Computer Science, pp. 120-220 Springer. 2020a. doi: 10.1007/ 3-00000057777 URL httttt:s: org 10. 100100007770 00000777778-0-0--87777777
978-3-030-58577-22
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In NeurIPS, 2020b.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying aision-and-languael tasks via text generation. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 1931-1942. PMLR, 2021. URL http: //proceedings.
wwwpsssvlss9 htt.:////chol.ahhhecm html
Yong Dai, Duyu Tang, Liangxin Liu, Minghunn Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi. One model, multiple modalities: A sparsely activated approach for text, sound, image, video and code. CoRR, abs/2205 06126 2022. doi: 10.48550/ arXiv.2205.061
htRt:ttts: /ooi 11. 48550/ 0612 06126
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URLhittss:/ id=YicbF dNTTY
net/ forum?
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-33
2022. URL http::ttp:/mmmmll http://wwww..ppppp/sse/////se///
httpppers//2/221 0998 html
Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identify- ing task groupings for multi-task learning. In NeurlPS, pp.
2021.
Biao Gao, Anran Xu, Yancheng Pan, Xijun Zhao, Wen Yao, and Huijing Zhao. Off-road drivable area extraction using 3d lidar data. In 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 1505- 1511, 2019. doi: 10.1109/VV..001 8814443.1
Yiheng Han, Irvin Haozhe Zhan, Wang Zhao, Jia Pan, Ziyang Zhang, Yaoyuan Wang, and Yong- Jin Liu. Deep reinforcement learning for robot collision avoidance with self-state-atten and sensor fusion. IEEE Robotics and Automation Letters, 7(3):6886-688) 2022. doi: 10.1 109/LRA. 2022.3178791.
10

---

Under review as a conference paper at ICLR 2023
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many- task model: Growing a neural network for multiple NLP tasks. In EMNLP, pp. 1923-1933. Association for Computational Linguistics, 2017.
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed H. Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. In Marc' Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NewrlPS 2021, December 6-14, 2021, virtual,
pp. 29335-223477 2021. URL https:: nwwriss.eeriipcoomm Absttt ceeerrppscc/pppee/200..
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularzze with gaussian eroor linear units. CoRR, abs/1606.08415, 2016. URL
606
08415
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified trans- former. In ICCV, pp. 1429. IEEE, 2021.
David Isele, Reza Rahimi, Akansel Cosgun, Kaushik Subramanian, and Kikuo Fujimura. Navigating occluded intersections with autonomous vehicles using deep reinforcement learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2034-2039. IEEE, 2018.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and João Carreira. Perceiver: General perception with iterative attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learming Research, pp. 4651-4664 PMLR, 2021. WRLLhttpp//proceediii
mlr press / / Jaeglelllll. html
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J. Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In The Tenth International Conference on Learming Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. net fert/forum? id= II -
URL https: /ettps://openrevee
R P Janani, K. Lakshmi Narayanan, R. Santhana Krishnan, Kannan, R. Kabilan, and N. Muthuku- maran. Intelligent drowsiness and illness detection assist system for drivers. In 2022 Second International Conference on Artificial Intelligence and Smart Energy (ICAIS), pp. 1150-1155,
2022. doi: 10. 10.111// 2.2.3311...22.77337555
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andrés Felipe Cruz-Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and ef- ficient moe training for multitask multilingual models. CoRR, abs/2109.10465,, 2021a. URL
htttps://arat www.aaiwwww...aaaaaaaaaaaaoooore abs /wwp///sss... 211.2111...10
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andrés Felipe Cruz-Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and ef- ficient moe training for multitask multilingual models. CoRR, abs/2109, 2021b. URL
bttps://arxiin /rg/ abs / 10099114663
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient in- ference. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pp. 3577-3599. Association for Computational Linguistics, 2021. doi: 10. 10.18653/v1/2021 11163///1/221..nnnnnnng wwwpliingg-emml...... URL https:
7/doi. .w..or//// 10. 08653/v1/2000 find ngs-emn)p.300

---

Under review as a conference paper at ICLR 2023
Michelle A. Lee, Brent Yi, Roberto Martín-Marthn Silvio Savarese, and Jeannette Bohg. Mul- timodal sensor fusion with differentiable filters. In JEEE/RSJ International Conference on In- relligent Robots and Systems, IROS 2020, Las Vegas. NV, USA, October 24, 2020 January URL.
24, 2021, pp. 10451. IEFE, 2020a. doi: 10. 1109//R0S4454442200099445 https: http://di..oggg/// 1ttp://00.11100055555510171 T00//47054453333 2020. 9341579
Michelle A. Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning mul- timodal representations for contact-rich tasks. IEEE Trans. Robotics, 36(3): 59,, 2020b. doi: 10.11099TR022019295555
URL
oto. 1 1109/ TRO
Luis A. Leiva, Asutosh Hota, and Antti Oulasvirta. Enrico: A dataset for topic modeling of mobile UI designs. In Susanne Boll and Simon T. Perrault (eds.), MobileHCI '20: 22nd International Conference on Human- Human-Compun Computer Interaction with Mobile Devices and Services: Expanding the Horizon of Mobile Interaction, Extented Abstracts, Oldenburg, Germany, October 5-9, 2020, pp. 9:1-9:4, ACM, 2020, doi: 10. 10.1155340032444410771 URL http://///o.../1111111111..
3406324 3410710
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL XXHTmbb)
httts:://ppenrevdt ew
Qing Li, Boqing Gong, Yin Cui, Dan Kondratyuk, Xianzhi Du, Ming-Hsuan Yang, and Matthew Brown. Towards a unified foundation model: Jointly pre-training transformers on unpaired images and text. CoRR, abs/21 1 12.070774 2021. URL https:: /ttp://aavvvvv... wwtpv///.gbbbbb/b......m 07074
Yao Li, Jiajun Deng, Yu Zhang, Jianmin Ji, Houqiang Li, and Yanyong Zhang. ezfusion: A close look at the integration of lidar, millimeter wave radar, and camera for accurate 3d object detection and tracking. IEEE Robotics and Automation Letlers, 7(4):1 1et11182-111888 2022. doi: 10.1
1R2.2022.3193465.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A. Lee, Yuke Zhu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multibench: Multiscale benchmarks for multimodal representation learning. In NeurlPS Datasets and Bench- marks, 2021.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Shentong Mo, Dani Yogatama, Louis-PPilipee Morency, and Ruslan Salakhutdinov. Highmmt: Towards modality and task generalization for high- mood-moylltty representation learning. CoRR, abs/2203.01311, 2022.
Valerii Likhosherstov, Anurag Amab, Krzysztof Choromanski, Mario Lucie, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. CoRR, abs/21 1ts/2111.12999. 2021. BtRh:tts::// /atp///aavvvv//aa//////eaa/aiava httpaaavvvv.r.oaaaaae/a wttg///sg//aieeeeee 129933993
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. Cross-token modeling with conditional computation. arXiv preprint arXiv:2109.02008, 2021.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task- aggosticc visiolin-
guistic representations for vision- vision-and-lan and-language tasks. In NeurlPS, pp. 13-23, 2019.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relation- ships in multi-task learning with multi- -ate mixture-of-experts. In Yike Guo and Faisal Farooq (eds.), Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pp. 1930-1939. ACM, 2018. doi: N0.1145/3219819.2222 920000777 URL https: sttps::/doi.orss /oii wttpaa/ogg100....... /tt.://tee////............2/2//2 1tt.///11111111111111333333 3220007
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Mul- timodal contrastive learning with limoe: the language-image mixture of experts. CoRR,
abs/2206.02770, 2022. doi: 10.485500axxx..2220007777
URL huttps://de http:///oo..ogg.../
485 48550/arXiv 2206.02770
12

---

Under review as a conference paper at ICLR 2023
Andrea Nemcova, Veronika Svozilova, Katerina Bucsuházy, Radovan Smīšek, Martin Mezl, Branislav Hesko, Michal Belak, Martin Bilik, Pavel Maxera, Martin Seitl, et al. Multimodal features for detection of driver stress and fatigue. IEEE Transactions Intelligent Transporta- sion Systems, 22(6), 2021.
Sriram Jayachandran Raguraman and Jungme Park. Intelligent drivable area detection system using camera and lidar sensor for autonomous vehicle. In 2020 IEEE International Conference on Elec- Iro Information Technology (EIT), pp. 429-436, 2020.
doi: 10. TTA.://TTT8999.200022200000
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. CoRR, abs/2204.06 125, 2022.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto. Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Perey Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 8583 8595, 2021. URL http:tps::pp/ceettt Abst .ttm
neurips CC/ wwwpppppppppppppppctcc htteer//22hhhhaa /
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sali- mans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffu- sion models with deep language understanding. CoRR, abs/2205. 11487, 2022.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le. Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL net/ wow.mmidd=BlccMDaa/...
httts:://ppereet
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15638-15650, June 2022.
Anders Søgaard and Yoav Goldberg. Deep multi-task learning with low level tasks supervised at lower layers. In ACL (2). The Association for Computer Linguistics, 2016.
Gjorgji Strezoski, Nanne Noord, and Marcel Worring. Many task learning with task routing. In ICCV, pp. 1375-1384. IEEE, 2019.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: pre- training of generic visual-linguistic representations. In ICLR. OpenReview.net, 2020.
Hao Tan and Mohit Bansal. LXMERT: learning cross-modality encoder representations from trans- formers. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing, EMNLP-IJCN 2019, Hong Kong, China, November 3-7, 2019, pp. 0099-51100 Association for Computational Linguistics. 2019. doi: 1/11555/1//D11-155/0 URI. http:: //doi wwwg/ol1/00
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 6569, Association for Computational Linguisties, 2019. doi: 10.18653/v1/pp 1/119 1656. URt.attpee//o....r/01..11155111.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):36143633, 2022. doi: 10.1109/TPAMI.
2021. 005449994
13

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008 2017. URL httppss/pproccedigg..coomm htteee/4435eeeeeeeeeeeeeeeeeeee
wwwppppee//1//aaaeeeeeeeeeeeeeee Abstract html
Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie. Centralnet: A multilayer approach for multimodal fusion. In Laura Leal-Taixé and Stefan Roth (eds.), Computer Vision ECCV 2018 Workshops Munich, Germany, September 8-14, 2018, Proceedings, Part VI, vol- ume 11134 of Lecture Notes in Computer Science, pp. 575-589, Springer. 2018. doi: 10. 1007/
378-3-030-11024--6 11024-6)
URL
44
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In CVPR, pp. 12692-12702. Computer Vision Foundation / IEEE, 2020.
Wenhui Wang, Hangho Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. Image as a foreign language: Beit pretraining for all vision and vasion-language tasks. arXiv preprint arXiv:2208. /0442, 2022.
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Amir Zadeh, Paul Pu Liang. Soujanya Poria, Erik Cambria, and Louis-Philipee Morency. Mul- timodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Iryna Gurevych and Yusuke Miyao (eds.). Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melhourne, Australia, July 15-20, 2018, Volume Long Papers, pp. 9236-2240 Association for Computational Linguistics, 2018. doi: 10.18653/v 1/P18/ 1208. ht http:://caaaaathohocooooo hww.hhhllooo...oo11.1311 1208/
Amir Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, pp. 3712-3722. Com- puter Vision Foundation / IEEE Computer Society, 2018
Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-perceiver-men moe: Learning sparse generalist models with conditional moes, 2022. URL https: wwtg//b/////a/ /200 04674
14

---

Under review as a conference paper at ICLR 2023
A MODEL DETAILS
A.1 PROCESS DATA INTO SEQUENCE
Following the process of Daegle et al. 2022 we first standardize each input into sequence. For each modality Jaegle et al. (2022). we define some hyperparameters (such as max mae_ num freq bands, and base) for the Fourier positional encoding. Fourier transformations get this positional information. For modalities such as text and time-series, they are already sequential data. We apply 1D positional encoding for these modalities x R X xd=, where bm Im dm are the batch-size, sequence length, and input dimension of current modality, respectively. For im- age and similar modalities, we follow the processing procedure of Dosovitskiy et al. (202T). which breaks each input into hm x Wm patches and flattens it as sequence of p regions. We use 2D positional encoding for image and similar modalitics input x E IR Xh.. X xd.., where hm x wm is the number of patches. For image modality, the dm is the number of pixels within a patch. For video and similar modalities, we treat each frame data as the image modality, therefore we apply 3D positional encoding for input I R xlm xhm X xdm, where mm is the number of the frame. In the other modalities, such as table and graph, we treat each element in the table/graph as an element in the sequence and use ID positional encoding.
After transposing inputs into sequence data, now we show the subsequent processing procedure in Algorithm I The 'max ceaax_modaiit_diie _dim' equals to maXme (dm + dym), where dpm the dimension of Fourier positional encoding for the corresponding modality. The one-hot encoding is defined as 6m E IR M, where [M] is the number of all modalities involved.
Algorithm Data Preprocess
x: the input tokens of specific modality
def DataPreprocess(x modality):
# get positional encoding information
# pos_dim: indicates 1 1D/DD/3D positional encoding ooc.poos-ffuuereeenndddmmamaiall
modality.max freq, modality.num freq bands, modality.freq (
# add padding for modalities with smaller input dimension
# max_modality -dmm:: the maximum input dimension overall modalities
# input_dim: the input dimension of current modality
padding-zeros (max_modalittyy sawpalttty.immmmddee coidmmmmattiyipnuutddiii
# modality one-hot encoding
# modality index: the index of current modality
modality_ene
# construct final input
modality imput=concatenateexx padding, enc_pos, modality.encodiimg return cedality_input
A.2 THE UNIMODAL ENCODER
The result of Algorithm is then feeded into the unimodal encoder layer. We display the details of the unimodal encoder layer in Figure 5] The sequence length T of different modalities are different, as T can be tm hm Wm. or Im X hm Wm. However, the cross attention between the input sequence and latent input will convert the sequence length from different modalities into the same value. For example, the input modality sequence is x E RT' xD and the latent input is z E IR NxC After these three linear layer, got K, V I x and Q RRNX Following the scaled-dot product attention:
QK
Attention Q, V) = softmax
V
VC
from which we can know the dimension after the attention is Attention Q K, V) E NXX There- fore, the sequence length of the output depends on the sequence length of the latent input and the
15

---

Under review as a conference paper at ICLR 2023
Cross Attention Layer
T
Modality Specific Input Soquence
Linear
Scaled-dot Product
Self Self-Attente Aff--tttenion
Encoder Layer
Attention
Linear
Linear
#
Outpot
Latent
Linear
Input
Figure 5: The details of the unimodal encoder layer. The D and T are the sequence length and feature dimension of the modallty-speeciff input sequence. The N and are the sequence length and the number of dimensions of latent input. The latent input is the learnable parameters shared across different modalities and tasks.
feature dimension depends on the unimodal encoder's 's hidden size, which is independent of the shape of the input modality sequence. The hidden dimension of the self-attention encoder layer equals to the previous layer's eross-attention layer.
A.3 THE MODEL AND TRAINING SETUPS
We list hyperparameters for the training and the model Table Table and Table [5 for small, medium and large setting, respectively.
B SINGLE-ROUTER V.S. MULTI-ROUTER IN SAS
This paper applies MoE in both the self-attention layer and the FFN layer within vanilla trans- former encoder layer. Moreover, several works Ma al. (2018); Aoki et al. (2021); Hazimeh et al. 2021 Kim et al. (2021a): Kudugunta al. (2021) ) Mustafa et al. (2022); Kim al. (2021a) in- vestigate single-router or multi-router for multi-task learning or multi-modal learning. Therefore, also investigate the multi-router SAT for TL. With those in mind, ask much more significant question:
What kind of MoE structure is appropriate for M SAT M
For our proposed MSSAT, we can use single-router MoE, multi-router MoE, and dense network in both the self-attention and FFN layers, respectively. Meanwhile, the multi-router MoE can also be divided into the modality-specific multi-router MoE and the task-specific multi-router MoE, There- fore, we explore all possible combinations of the above settings the self-attention and FFN layers. We list all explored network architectures in Table
Table 5: All possible MoE design combinations for SAT
Meeblay Speate Mall
Secespall XSI
Style Roaee MEI
anD Mall
Nanhli Spents
F.rsalir Korn SAT
Mela Kore M 3AT
Meentte S.. SUST
Malchonaa HUS. nou PSN She
attenik
Saci: Mell Mark
Alt
R.ME and
We run above network architectures in the medium setting and report the results Table 6 All results reported in Table/]juee the same hyperparameters in Table [4] except for the routing network setting. In particular, the 'Dense Model' is an equal computation dense model where we propose two kinds of equal computation dense model: 'Dense Model 1' uses the transformer encoder layer with double depth and 'Dense Model 2' is 4x wider than the hidden dimension of the transformer encoder layer. To further illustrate our performance gains mainly come from SASAT design,
16

---

Under review as a conference paper at ICLR 2023
Table 6: The results of different MoË router settings in the medium setting.
Model
ENRICO
PUSH
AVINISTT
A1%)
HighMMTT muliiisk
M'SAT
71.58
0.475
71.36
20.19
Multi-ruuer M'SAT
71.00
0.684
71.03
7.81
R-Multicouee M MAT
64.38
71.33
-13.48
Dense
Dense Model
62.36
1,400
71.40
-37.11
MPSAT wio FPN MoE
68.34
0.818
70.94
-1.02
MPSAT wio Alt..
wio
Mulli-Route M"SAT wio Altn. MaE
65.41
1,402
70.08
-36.03
Mulii Router SAT wio ETN MoE
67.35
0.633
71.37
8.54
Muulii-Ruuter M"SAT
Atto Jasi-FFN-SSigle SAT
03.51
0.952
71.02
11.82
Mun-MMrdaliy-FFN-SSing M"SAT
60:52
0.777
71.47
1.94
Atn-Single-FNN-Taa M
Altx-ingle-FFN-WM Modilliy M"SAT
1,088
71.31
-17277
"MModaliy MPSA
65.35
0.786
70.70
0.54
roske-rotter M'SAT
68.38
0.833
70.69
-2.25
Fqual Capacity Model
Table 7: Task performances of different models. M 2/3/4 layers: 2/3/4 transformer encoder layers and replacing with S layer every other layer. P-M 2/3/4 layers: 2/3/4 consec- utive SA layers. S carly/middleelltee-:: 4 transformer encoder layers and replacing the carly/middlle/lte-2 encoder layers with two S layers.
Model
ENRICO
PUSH
AV-NNIST
A(%) 1
HighMMT multitask
53.10
0.600
68.48
0.00
M
71.58
0.475
71.86
20.19
M"SAT 2 Laaeers
70.35
0.992
70.34
-9.92
M Inyers
69.18
0.551
70.32
13.71
M layers
71.46
1,223
70.18
-22.24
P-M"SAT layers
69.63
0.766
71.57
2.64
P.M"SAT' layers
70.78
P-M layers
67.47
0.976
71.68
-10,30
M"SAI early two layer
68.15
0.793
71.19
-0.03
M"SAT middle two layer
73.17
0.884
69.89
-2.49
M layer
72.15
1.374
69.97
-30.33
we construct same capacity model where we x4 the number of attention heads, x8 the dimension of each attention head, and x32 the hidden dimension of the MLP layer.
We find out that the single-router is the best architecture for M The second best architecture using the task-specific router in the self-attention layer and the dense layer the FFN layer. Meanwhile, using the modality-specifff router in the self-attention layer and the task- router the FFN layer also seems like reasonable choice.
For better understanding, we display the architecture of the Multi-Router MASAT and the R- Multi- S in Figure Figure 7 respectively.
B. USING CONSECUTIVE SAS
This section used to illustrate how use consecutive M layer as transformer backbone, and provide more observation about how use M while network is getting deeper.
Our experimental results in Table show:
The performance may not be improved as the number of SAS layers increases.
The location of SASAT matters. Using MASAT in shallow layers helps the most.
17

---

Under review as a conference paper at ICLR 2023
Multi-router SAT
Fnooder Layer
Spare Man FPA
Sparse MoE
T. Lincar
Multi-Head
Artention
Molti-mmater Sparse MoF: FFN
Spunc Schookizrecs
Equat
Multi-Hedd Attention
Rayent
F
Expert
Multi-router Artention Value/Key/Query MuE
Milloooa Valan Salii Rez SA4
Vadi Quen Hit
Reali
Markery
Figure 6: In the Multi-router SAT encoder layer. We use the modality-specinf router in the Self-attention layer and the task sask-specific router in the FFN layer.
R-Mulli-routee
MPSA Ficoder
Layer
Spare
Sparse
Linear
Mulii Head
Norr
Atiention
Multi-router Sparse
Concat
Dapen
Equan
Dapan
Narz
Multi-Head Attention
Exper
Eqpen
Expani
Multi-router Artention Value/KeylQurry
Mabioo
Stucilty
Anzzze
Kez
Quer
Ireic
Reaket
Reake
Figure 7: In the Reverse Multi-router MASAT (R-Multi-router M encoder layer, We use the task-specific router in the Self-attention layer and the modality-specifie router in the FFN layer.
Table Detailed results of parameter and computation cost.
PUSHI
V&T
Small setting
Params (M)
Flops (G)
Params (M)
Flops (G)
MighMMT multitakk
0.89
5.14
0.85
32.48
M
0.27
2.59
0.25
17.38
ENRICO
PUSH
AV-MNIST
Medium setting
Params (M)
Flops
Params (M)
Flops
Params
Flops
HighMMT multitask
0.58
79.48
0.63
21.60
0.52
0.95
SAS
1.23
1.10
1.25
2.33
1.23
0.41
UR-FUNNY
MOSEI
MIMIC
AV-MIIST
Large setting
Parums (M)
Flops (G)
Params (M)
Flops (G)
Params (M)
Flops (G)
Params (M)
Flops (G)
HighMMT multitakk
0.22
1.11
0.52
1.63
0.32
0.67
0.32
0.93
SASAT
0.53
C EXPERIMENTS DETAILS
We show the number parameters and the computation cost of the current SOTA and SAT in
Figure 8
18

---

Under review as a conference paper at ICLR 2023
C.I DATASET
PUSH Lee et al. (2020a), i.e., the MUJOCO PUSH task, is a planar pushing task, in which 7-DoF Panda Franka robot is pushing a circular puck with its end-effector in simulation. We estimate the 2D position of the unknown object on a table surface while the robot intermittently interacts with the object. This dataset contains 1000 training data, 10 validation data, and 100 testing data, where each data point is split into 29 sequences, and each sequence includes 16 consecutive steps.
Lee et al. (2020b) also called "VISION& is real-world robot manipulation dataset that collects visual, force, and robot proprioception data for a peg insertion task. The robot is used to insert the peg into the hole. In this paper, we use this dataset to predict the manipulator weather contact the peg in the next step, which is binary classification task. We follow the setting of MultiBench and 117,600 data points for training and the remaining 29,400 data points for validation and testing.
ENRICO Leiva al.) (2020) includes 20 Android app design categories. Each data point consists of the app screenshot and the view hierarchy. The view hierarchy describes the spatial and structural layout of UI elements of the corresponding screenshot. During training, the view hierarchy is ren- dered as "wireframe", which can be viewed as a form of set data. ENRICO contains 947 data points for training. 219 data points for validation, and 292 data points for testing.
AV-MNIST Vielzeuf et al. (2088) is multimedia dataset that uses audio and image information to predict the digit into one of 10 classes (0-9). This dataset comprises 55,000 training data points, 5,000 validation data points, and 10,000 testing data points.
UR-FUNNY the multi- modal affective computing dataset of humor detection in human speech. Each data point of UR-FUNNY is video with text, visual, and acoustic modalities. We train this dataset to predict whether the current data point makes people fill positive or negative. There are 1,166, 300, and 400 videos in the train, valid, and test data, respectively.
MOSEI Zadeh et al. (2018) is the largest dataset of sentence-level sentiment analysis and emotion recognition in real-world online videos. Each video is annotated for 9 discrete emotions (angry, excited, fear, sad, surprised, frustrated, happy, disappointed, and neutral), and a continuous emotions value (valence, arousal, and dominance). We follow the MultiBench, training this dataset as binary classification task. We use 16,265, 1,869, and 4,643 train, valid, and test data points, respectively.
MIMIC Johnson et al. (016)), i.e., the Medical Information Mart for Intensive Care III, is freely accessible critical care database, which records ICU patient data, including time-series and other demographic variables in the form of tabular numerical data. We use this dataset for binary classi- fication on whether the patient fits any ICD-9 code in group 7 (460-519). The dataset is randomly split into 28,970, 3,621, and 3,621 data points for training, validation, and testing.
For more details of the above datasets, please refer to the Liang et al. (021) and their released website:
http://gghhbbcoom///lgg22MMMuhbn
Results of HighMMT is running by Liang al. (2022) released code:
C.2 THE TOTAL NUMBER OF EXPERTS AND THE NUMBER OF EXPERTS PER SELECTION
We do ablation studies on the number of experts per selection K and the total number of experts N for the medium setting, From Table 9 we can observe that the performance is increase with the number of N. However, increasing the total number of experts requires more memory resources. Increasing the number of experts per selection K can improve performance to some extent. but too larger K will restrain parameters from getting enough training, decreasing performances. The appropriate value of N and K is crucial for M performance.
19

---

Under review as a conference paper at ICLR 2023
Table 9: Ablation studies. Effects of the number K of selected experts per token and the total number N of experts.
Model
ENRICO
PUSH J. AV.MNIST A(S) 1
HighMMT multitask
53,10
0,600
68.48
0.00
2K, 32/V (M'SAI)
71.58
0.475
71.86
20.19
32N
65.41
0.522
71.48
13.49
3K, 32/V
69.98
0.645
71.10
9.36
32NV 32N
67.24
0.782
71.43
0.20
2K, 4N
67.92
1.250
71:33
-25.41
2K, 8.N
2K. 16.V
69.75
0.771
70.45
1.89
Table 10: Concatenate tokens along the batch axis.
Model
ENRICO f
PUSH 4
AV-MNIST f
A(%)
HighMMT multitusk
53.10
0.600
68.48
0.00
S'A
71.58
0.475
71.86
20.19
Concate along batch
64.38
1.174
71.05
-23,57
C.3 FUSION BY CONCATENATE TOKENS ON THE SEQUENCE DIMENSION
Before we imput tokens into our transformer backbone (several consecutive transformer encoder lay- ers), we concatenate tokens on the sequence dimension. Therefore, we can fusion different modal- ities by the attention layer within each transformer encoder layer. To further illustrate that such an operation is necessary. we additional training the same model but concatenate tokens along the batch axis. Our following table shows fuse modalities by concatenating tokens along the sequence axis is positive for our tasks.
Our results in Table 10 show fuse modalities by concatenating tokens along the sequence axis positive for our tasks.
C.4 INDEPENDENT ROUTING POLICY BETWEEN Q, K, AND
Prior works Fedus et al. 2022 Zhu et al. 2022) also apply MoE in the attention layer. However. they all use single router routing tokens for q, k, and simultaneously. We think such design lack flexibility. Therefore, in our MoE attention layer, the router for k, and is separate, which could provide more flexible attention mechanism. In order to support the above statement, we con- duct additional experiments in Table 11 to study the advantage of S v.s. Prior MoF attention design style (q. k, v using the same router in the MoE attention).
C.5 THE GRADIENT POSITIVE SIGN PURITY OF SAS
The Gradient Positive Sign Purity Chen al. as:
2020b) P of single parameter for T tasks is defined
1 (1+
AL
(8)
2
AL,
where AL; is the gradient for the task i. The Gradient Positive Sign Purity is bounded by |0, 1]. which P close to or 0 indicates such parameters suffer less gradient confliction from multi-task training. We the trained model to collect the Gradient Positive Sign Purity of such model. Then we discrete the Gradient Positive Sign Purity value into five intervals of each parameter and count the ratio of parameters in these five intervals.
20

---

Under review as a conference paper at ICLR 2023
Table 11: Using single router routing tokens for 9, k, and simultaneously.
Model
ENRICO
PUSH
AV-MNIST 1
A(%) 1
HighMMT multitask
53.10
0.600
68.48
S'SAT
71.58
0.475
71.86
20.19
gkv share routers
73.51
0.936
69.28
-5.45
0.5
Inter Task Affinity
0.03
0.00
-2.5
Inte-Taak
.35
Makkeren 14 lsuer sats wao polodaltyppe sespssccch MPSAA
00120
30
40
10
80
90
100
10
30
40
60
60
70
80
90
100
Epochs
Training Epochs
Figure 8: The inter-task affinity of the "ENRICO' to the 'AV-MNIST' task (right), and the inter-task affinity of the 'ENRICO" the 'PUSH' task (right). The results reported are the average of three replicates.
C.6 THE TASK AFFINITY OF M
The task affinity Fifty et al. (2021) is defined as follows:
L2(X', 0 85) 21-, L,(X'' 04, 09)
(9)
where X is the training batch time-step f, B is the updated shared parameters after a gradient step with respect to the task i. 03 represents the task j's specific parameters. Considering the imbalance between datasets in the MultiBench, we set the size of X is the training data size of such task t. Therefore, for the medium setting, we collect the task affinity by solitary training the 'PUSH' task for an single epoch, then calculate the loss of 'ENRICO' and 'AV-MNIST' the corresponding training data. We count the task affinity from 'PUSH' to 'ENRICO' and 'AV- MNIST every 10 epochs during training. We display the task affinity changes with training epochs in Figure 8 The task affinity of M SAT and multi-router S is usually higher than the one of the dense model which indicates that the MoE we proposed alleviates the training conflict of MTL.
C.7 THE OPTIMAL GRADIENT BLEND OF SAS
The optimal gradient blend Wang et al. (2020) is used to re-weight the feature of each modality during multi-modal training. The optimal gradient blend will give this modality a small weight for the modality that is easy to prone to overfitting. The weight of each modality is bounded by [o, 1] within a task, and the sum of all modalities for this task 1. Therefore, the gap between different modalities within task indicates that the modality with smaller weight (optimal gradient blend) tends to overfit. We collect the optimal gradient blend of the corresponding trained model determine whether
proposed model can restrain the easy model from overfitting. We use modified version of the optimal gradient blend where the unnormalized optimal gradient blend of modality m defined as:
L T palid
m. to unnoom
(10)
I'm Loaldd
L Brain
where Li is the validation loss after training n epochs only using modality m, and Lm the training loss after training epochs only using modality m. For task i, the final optimal gradient blend we reported is:
up Hunorrm
Wi.m
(11)
72 W
21

---

Under review as a conference paper at ICLR 2023
Table 12: The optimal gradient blend for each tasks under different model architectures.
Model
ENRICO
PUSH
AV-MNIST
image
image
force
proprioception
control
image
audio
SAT
0.48
0.52
0.00
0.37
0.32
0.31
0.00
Dense Model (w/o MoE)
0.61
0.39
0.00
0.36
0.32
0.31
1.00
0.00
w/o
MoE
0,63
0.37
0.00
0.37
0.32
0.30
1.00
0.00
w/o FFN MoB
0.75
0.25
0.00
0.37
0.32
0.32
1.00
0.00
multi-gate M*SAT
0.71
0.29
0.00
0.35
0.32
0.32
1.00
0.00
P-Modality-gaes SASAT
0.73
0.27
0.00
0.37
0.31
0.32
1.00
0.00
P- g-Tak--gate SAT
0.80
0.20
0.00
0.36
0.32
0.31
1.00
0.00
where M is the number of modalities of the task i.
For M the appropriate combination between coddality-speciffe routers and task-specific routers (multi-router S helps each other better than purely using one of them (In Figure 8 and Ta- ble 12 the Inter-Task Affinity and the optimal gradient blend of multi-router M better than models which only use modality-speciie routers (P- (P-Modality-gaal S task-specific routers (P-Task-gate SAT)..
C.8 EXPERT DISTRIBUTION
This section explores how tokens are distributed across different tasks and modalities by the routing policy of the SAT. We show the routing distributions under the testing distribution in Figure 9 Figure [0] and Figure IT] these three settings, our routers work well, and most experts handle modalities and tasks. Meanwhile, several experts focus on specific tasks.
For the large setting, we find out that the routing policy tends route tokens several specific ex- perts, which also successfully proves MTL's MoE separate gradient conflict parameters. Especially for the 'MIMIC" dataset, only experts activate this task.
In Figure
Figure T0 and Figure 11
we also denote the FFN layer as the MLP layer.
22

---

Under review as a conference paper at ICLR 2023
Figure 9: The token distributions of the small setting of the first S layer. The first two show the token distribution of different modalities for the 'PUSH' dataset, and the 'V&T' dataset. The last row shows the token distribution across different tasks within the self-attention key layer, the self-ate attention query layer, the elffaatennionn value layer, and the FFN layer.
23

---

Under review as a conference paper at ICLR 2023
Figure 10: The token distributions of the medium setting of the first S layer. The first three rows show the token distribution of different modalities for the 'ENRICO' dataset, the 'AV-MNIST' dataset, and the 'PUSH' dataset. The last row shows the token distribution across different tasks within the self-attention key layer, the self-attention query layer, the self self-attention value layer, and the FFN layer.
24

---

Under review as a conference paper at ICLR 2023
A.tisburry Lept
Figure 11: The token distributions of the large setting of the first M layer. The first four rows show the token distribution of different modalities for the 'AV-MNIST' dataset, the 'MOSEI dataset, the 'UR-FUNNY' dataset, and the 'MIIIC' dataset. The last row shows the token distribution across different tasks within the self-attention key layer, the self-attention query layer, the self- attention value layer, and the FFN layer.
25

---

Under review as a conference paper at ICLR 2023
Table 13: Table of the modal and training setups on the small setting tasks: PUSH and V&T.
Model Sctup
Name Hyperpurameter
Value
Sequence Lengih Latent
Lileni Dimension
64
Peresiver Unimodal
Head
Encoder
Cross Head Dim
Scll-Atienion Head
Depth
DooS&DDesse Encoder Layer
Scl--Allentio Head
Experts Numhhr
Esperts Number Per Selection
Classification Heads
BatchNorm follow a Linear layer
Inpuz/Ouipu: dimensions
256/32
32W1
Opsimizer
Adam
Learning mite
Learning Schiclialer
N7A
Weight Decay
0.0
Tminning
Load& Importance
Balancing Loss Weight
0.1
Pretrain
N/A
Max, Epoch
Truining loss weight
1000
1.0
Lvva
Batchsize
Loss Function
MSE
CrossEntropy
Image: 128 128x3
Iaput Dimension
MultiBench
Image: 16x32x32
Gripper
Gripper Sensors: 16x7
Depth: x28xx128
Force: 6x332
Propeio: 8
Control:
Action:
Gripper Pos:
Fore:: 32
Image: 3
Pereciver Input Channel Size
Gripper Sensors:
Propeio: 8
Image:
Coniml: 7
Depth: I
Gripper Pos:
Image:
Perceiver Input
Gripper Sensors:
Foree:
Proprio:
Image:
Depch:
Control:
Action:
Dataset
Perzeiver Input mim_freq.hands
Gripper Sensons: 6
Gripper Pos:
Forec:
Image:
Proprio:
Control:
Depch:
Action:
Image:
Perceiver Input max_freq
Gripper Sensors:
Gripper
Proprio:
Foree:
Control: 16x7
Image:
Depth:
26

---

Under review as a conference paper at ICLR 2023
Table 14: Table of the modal and training setups on the medium setting tasks: ENRICO, PUSH and AV-MNIST.
Model Setap
Name of Hyperpanenterr
ENKICO
Valae PUSHH
AV-NNSST
Sequence L Langt of Lanent
12
Latez Dinenaion
64
Perceiver Unimodal
Lioss Alteston Head
Encaderr
Cios Heal Dun
64
Self-Aliembon Head
Head Dim
MessiiDDease
Sell-Allcation Tim
Fncowler Layer
Experts Number
Esperts Number
ButchNon follow layer
Input/Cuiput dimesions
128/20
256/22
Optimizer
Adam
Letening Tile
07001
Learling Scheduler
Woiglr Dicay
(10
Laud Iapolttiee
Training
Bolancing Less Weight
0.08
Pietrain
Training PUSH for EDU epochs link
Max Lppcc
T00
Traiting loss waight
10:0
BCB
Evolluaioon weight
10,0
1.0
Ialchiuze
Crosttunoyy
MSL
Input
Multieench
256x x
Grappa
Gripper
Image: 36x32x32
Coloriess Image: 28
Audio Spectogram:
Parcsiver Input
Channel Size
Sec:
Imagge:
(cut into 16x8 rectangles)
(cst restingles)
Grippe
Image:
Gripper Sensoss:
(cut into 4 squazes)
Control: 7
Andio Spectogram: 256 tcut into 16x squares)
Image: squires)
Gripus Foo:
Perceiver Input Exira Axis
Imager: 2
Gripper Sensory:
Colorless Image: 2 Audio Spectegram: 2
Set:
Image:
Control:
Datsser
(rrpper Fix
Parceives Input Boarss
Image:: 6
Grripee Sensoss: 6
Coloriess Image:: 6 Audio Speciogram:
Set:
Imager
Control:
Fox:
Parcevvr Inpur max_freq
Gripper Sensors
Coloriess Image: Spectegram:
Image:
27

---

Under review as a conference paper at ICLR 2023
Table 15: Table of the modal and training setups on the large setting include tasks: tasks:UR-FEI UR-FUNNY MOSEL, MIMIC, and AV-MNIST.
Midel Seou
Value
Name Hyperpenmeier
URFUNNY
MOSET
MINICC
AV-NNIST
Sequence Length Lalent
Lateni Dimension
64
Perocierr Unimodal Eneader
Cross Alletion Heal
Atientin
Self Head Dim
64
Depth
M3o5&Deese Layer
Alicnbiic Head
Experts Number
16
Experts Number Per Selection
Clasolicaion Heads BatchNorm follow Linear layer
Inpus/Duppt dimensions
192/2
192/2
128/2
128/10
Optimizer
Leaming rale
0.0008
larning Scheclller
NAA
Weight Dacay
Training
Balancing Loss Weight
Imporiance
0.1
Pretraan
N/A
Mas Epuch
100
Triining weight
02
0.9
Evalustion weight
1.0
Batchstze
32
32
Funcctoo
Crossninnyy
Crossaningy
CrossPatinpy
ChosPriropy
MuhtiBenc
20x
Image: S0x
Coloriess Image: 28 2x
Input Dimension
Text: 50x300
Text: 50x.300
Time-series. 2xx
112x 112
Spectogram:
Colorless Image: 16
Percriver Inpul
Audio:
Image: 371
Image: 35
Audio: 74
Text:
Statie:
Time-series:
(cut into 16x 16 squares)
into 4 squanes)
Spectogram:: 256
Perceiver Input Extra.
Andio:
Image:
Text:
Audio:
Image:
Static:
serissseres::
Audio Spectogram:
Colorless Image: 2
Ditoset
Perceiver Input nun_froq.bands
Text 3
Audio:
Text:
Audio:
Image:
Time-strios:
Static:
Colerless Image: 6
Audio Spectugram: 6
Percelver Input max,freg
Audio:
Statie:
Colortess Image:
Text:
Text:
Spectograms:
28


Reviews:
[{"review_id": "rbM0U4nGh1t", "reviewer": "Reviewer_9ufm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes $M^3SAT$ for the multi-modal multi-task learning task, which tailors the mixture-of-experts (MoEs) into both the self-attention and the feed-forward networks (FFN) of a transformer backbone. $M^3SAT$ can prevent training conflicts among diverse modalities and tasks, and restrains the problem of simple modal prone to overfitting.", "summary_of_the_review": "It is useful to reduce computational costs. The authors should further state the novelty of the proposed method and provide more details.\n\n-------------------AFTER REBUTTAL-----------------------\n\nThanks for the response, my concern has been partially addressed. However, I think the novelty of the paper needs further improvement. \n1. I appreciate the authors for explaining in detail the differences between this work and Uni-Perceptionr-MoE. However, these differences are not particularly innovative and do not qualify as a major contribution.\n2. We note that compared to modality-specific routers, task-specific routers are not more effective for the gradient conflict problem in multi-task learning. I think the interpretability of modality-specific routers and task-specific router still needs further clarification.\n\nTherefore, I would keep my rating.", "strength_and_weaknesses": "1. In the multi-modal multi-task learning, authors propose a multi-modal multi-task mixture of expert model, which solves the training conflicts among tasks and restrains the easy modality from overfitting.\n2. The proposed $M^3SAT$ has the remarkable improvement in performance and computational cost.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D7kzS09pbX", "reviewer": "Reviewer_9uX5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on multi-modal multi-task learning. They customize Mixture-of-Experts into the transformer layer to do efficient MTL. They achieve good performance on the HighMMT dataset. ", "summary_of_the_review": "See above.", "strength_and_weaknesses": "My major concern with the paper is the technical novelty. I will illustrate the issue from three aspects: 1. Using MoE to do multi-modality is not new. 2. The technique it uses (MoE attn and MoE Mlp) is exactly the same as previous work. Multi-router MoE is more like a clever trick rather than a solid technical contribution. 3. No much interesting finding in the paper.\n\n\n1. Using MoE to do multi-modality is not new.\n[1, 3] scale up on big model and data and also use MoE mlp to do efficient learning. Both text modality and vision modality have been explored. [2] also, focus on multi-modality and also use experts for each modality. [2] is not exactly the MoE, but the idea that using different experts for different modalities is not new.\n\n2. The technique it uses (MoE attn and MoE Mlp) is exactly the same as previous work. \nMoE MLP is exactly the same as in [1,3]  and many other MoE papers. MoE Attn directly applies moe on qkv in the attention layer, which has been explored by the switch transformer[4]. (in their Tabel 10) and also mention by [1,3] before.  Multi-router MoE is more like a clever trick rather than a solid technical contribution. \nSo I don't think there is enough technical contribution in this paper.\n\n3. Since the technical novelty is restricted, I would expect more interesting findings in experiments. However, it occurs to me that the experiment part is mainly about the performance gain and I don't find some interesting conclusion/insight. Perhaps the writers can illustrate more about that in the rebuttal.\n\nOverall, the paper is more like a direct implementation of some known technique. So I would keep a negative rating for now.\n\n[1] Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts\n[2] VL-BEIT: Generative Vision-Language Pretraining\n[3] Scaling Vision with Sparse Mixture of Experts\n[4] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\n\n\n\n\n-------------------AFTER REBUTTAL-----------------------\n\nI appreciate the detailed response from the authors. I also carefully read the comment from other reviewers.\nHowever, I still hold my original opinion that the paper needs to improve its technical novelty to be accepted as a conference paper.\n\n1. Multi-modality multi-task setting.\nBoth authors and I agree that there are some similar settings in previous papers. I appreciate the author giving a detailed explanation of the differences. However, these are minor differences to me and can not convince me of a major contribution.\n\n2. MoE Mlp.\nBoth authors and I agree it is the same as previous work.\n\n3. MoE Attention and multi-router.\nThe authors explain the difference in rebuttal. I believe some simple techniques could make huge differences but in this model, these contributions are more likely to be incremental rather than a game changers. \n\n4. Insight and some interesting finding on the new task setting with moe.\nThese parts are missing from the rebuttal.\n\nTherefore, i would keep my rating.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AVEAr98DDwd", "reviewer": "Reviewer_rWtA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents M^3SAT, a sparse transformer model based on mixture-of-experts (MOE) for multimodal multi-task learning. Sequences of tokens from different modalities are concatenated together and processed by transformer-based encoder layers that incorporate MOE in the attention and FFN modules. A different decoding head produces the outputs for each task. The method is shown to outperform HighMMT on the MultiBench tasks with less computational cost.", "summary_of_the_review": "Overall, I think this is a good paper that advances the field of multimodal multitask learning.", "strength_and_weaknesses": "Strengths: \n1) The paper is clearly written and shows strong quantitative results on the MultiBench tasks for M^3SAT compared to HighMMT for less computational cost.\n2) The authors also perform thorough ablation studies where they demonstrate the contribution of their proposed encoder.\n3) The authors analyze metrics such as gradient positive sign purity, intertask affinity and optimal gradient blending to support their claims that the method alleviates some of the challenges of multi-task and multimodal learning.\n\nWeaknesses: \n1) For inputs such as videos and audio data that have a temporal component, the feature fusion can be performed along the temporal axis (and is usually done so for many tasks such as speech recognition). Could the authors comment on the decision to instead fuse modalities by concatenating their tokens along the sequence axis, as shown in the Figure 1? Doesn't this remove the temporal correspondence between modalities? Maybe this has a larger impact on tasks other than those in MultiBench.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6omxQymLaZ", "reviewer": "Reviewer_yAsT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed work aims to tackle the multimodal multitask optimization problem with sparsely activated mixture-of-experts models where both self-attention and FFN networks are implemented as routed expert networks. This implementation achieves better disentanglement of parameters which reduces overfitting for modalities that are \"simple\" compared to other modalities in a task. In addition, these models outperform dense counterparts in performance while have lower computation(FLOPS) on several benchmarks.", "summary_of_the_review": "Overall, the proposed method is interesting as applying converting self-attention MOE layers can bring improvements to different types of MTL problems. The method works for some tasks in a MTL setting quite well. However, some parts of the claims are not very clear and how well they will generalize to different tasks/modalities setting is unclear. \n\nLooking forward to the discussion with the authors.", "strength_and_weaknesses": "### Strengths\n- The improvements across the various tasks on MultiBench clearly shows the effectiveness of $M^3SAT$. \n- Authors perform extensive ablation study comparing various routing mechanisms and mixture of expert networks types.\n- From Table 3 ablations, the mixture-of-experts implementation on self-attention clearly shows significant improvements on the evaluated tasks. This is very interesting as standard MoE architectures apply experts to FFN layer and I am curious how this would apply to other MTL problems like MT which have one modality only.\n- Authors ablate thoroughly the different routing mechanisms and it is conclusive from these results that single routing is effective for this setup and tasks.\n\n\n### Weaknesses\n- It is not clear to me why those specific 7 tasks were chosen among 20 tasks from MultiBench.\n- When comparing MoE model with a dense model with similar computation(FLOPs), it is unclear whether the gains are coming from capacity increase or due to efficient parameter space separation due to different experts specializing in different tasks/modalities. One important comparison to disentangle these factors will be comparing MoE with dense models with similar capacity. Of course, the counterargument that MoE can provide larger capacity with lower FLOPs is valid, but for better understanding of where the improvements are coming from it is important to compare similar capacity models. \n- From Table 2, in the `Large setting` of the tasks, the improvements are not as significant as `Medium setting`. It is also unclear how the model capacities are chosen for these different settings. Were the models optimized for a particular setting through sweeping over different values of $N$ and top-$K$? Which other hyper-parameters are significant for these choices?\n- Given that most of the ablations for MTL are done on the `Medium Setting` tasks, where the improvements are significant for the ‘ENRICO’ task, it is unclear whether $M^3SAT$ improves across all types of tasks and settings, and how well it generalizes. For example in Table 4, other tasks like PUSH and AV-MNIST do not show any improvements to the overfitting problem.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject