High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Authors: Paul Pu Liang
Yiwei Lyu
Xiang Fan
Jeffrey Tsaw
Yudong Liu
Shentong Mo
Dani Yogatama
Louis-Philippe Morency
Russ Salakhutdinov
Venue: Accepted by TMLR ()

Abstract: Many real-world problems are inherently multimodal, from the communicative modalities humans use to express social and emotional states such as spoken language, gestures, and paralinguistics to the force, proprioception, and visual sensors ubiquitous on robots. While there has been an explosion of interest in multimodal representation learning, these methods are still largely focused on a small set of modalities, primarily in the language, vision, and audio space. In order to accelerate generalization towards diverse and understudied modalities, this paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. Since adding new models for every new modality or task becomes prohibitively expensive, a critical technical challenge is heterogeneity quantification: how can we measure which modalities encode similar information and interactions in order to permit parameter sharing with previous modalities? This paper proposes two new information theoretic metrics for heterogeneity quantification: (1) modality heterogeneity studies how similar $2$ modalities $\{X_1,X_2\}$ are by measuring how much information can be transferred from $X_1$ to $X_2$, while (2) interaction heterogeneity studies how similarly pairs of modalities $\{X_1,X_2\}, \{X_3,X_4\}$ interact by measuring how much interaction information can be transferred from $\{X_1,X_2\}$ to $\{X_3,X_4\}$. We show the importance of these $2$ proposed metrics in high-modality scenarios as a way to automatically prioritize the fusion of modalities that contain unique information or unique interactions. The result is a single model, HighMMT, that scales up to $10$ modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and $15$ tasks from $5$ different research areas. Not only does HighMMT outperform prior methods on the tradeoff between performance and efficiency, it also demonstrates a crucial scaling behavior: performance continues to improve with each modality added, and it transfers to entirely new modalities and tasks during fine-tuning. We release our code and benchmarks, which we hope will present a unified platform for subsequent theoretical and empirical analysis.

Published in Transactions on Machine Learning Research (05/2023)
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Paul Pu Liang', Yiwei Lyu', Xiang Fan', Jeffrey Tsaw', Yudong Liu', Shentong Mo',
Dani Yogatama', Louis-Philippe Morency', Ruslan Salakhutdinov"
'Carnogic Mellon University, *University of Michigan, *DeepMind
Reviewed OpenReview:
Abstract
Many real-world problems are inherently multimodal, from the communicative modalities humans use such spoken language, gestures, and paralinguistics to the force, proprioception, and visual sensors ubiquitous on robots. While there has been an explosion of interest. in multimodal representation learning, these methods are still largely focused on a small set of modalitics, primarily in the language, vision, and audio space. In order to accolerate generalization towards diverse and understudied modalities, this paper studies efficient representation learning for high-modality scenarios involving large set of diverse modalities. Since adding new models for every new modality task becomes prohibitively expensive, measure which
critical technical challenge is heterogeneity quantification: how modalities encode similor information and interactions in order to permit parameter sharing with previous modalities? This paper proposcs two new information thcoretic metrics for heterogeneity quantification: (1) modalily helerogeneily studies how similar 2 modalities (2) are by measuring how much information can be transferred from X1 to X2, while (2) interaction heterogeneity studies how similarly pairs of modalitics X2), (Xa, X2) interact by measuring how much information can be transferred from fusing {2) to X4). We show the importance of these 2 proposed metrics in high-modality scenarios way to automatically prioritize the fusion of modalities that contain unique information or interactions. The result is single model, HIGHMMT, that scales to 10 modalities (text, image, audio, video, sensors, proprioception, speech, lime-series, sets, and lables) and 15 tasks from 5 research areas. Not only does HIGHMMT outperform prior methods on the tradeoff between performance and efficiency, also demonstrates a crucial scaling behavior: performance continues to improve with each modality added, and it transfers to entirely new modalities and tasks during fine-tuning. We release code and benchmarks at: htttp:://github: wttp:/u....oommmp..../../.a.a MMM
Image-text
Video
Sercasm
Robotic
Human
Design
Disease codes
Introduction
category
manipulation
emetions
Multimodal machine learning brings unique challenges for both computational and theoretical research given the het- erogeneity of various data sources (Liang et al. 2022). While there have been im- pressive advances modeling language, vision, and audio (Agrawal et al.) 2017 Ramesh et al. 2021), advances in sensing technologies have resulted in many real- world platforms such as collphones, smart devices, self-driving cars, healthcare tech- nologies, and robots now integrating a much larger number of sensors such time-series, proprioception, sets, tables, and high-frequency sensors (Frantzidis)
1. Which modalities are different and should be separately processed?
2. Which modality pairs interact differently and should be separately fused?
Language
Image
Video
Sensors Progrioception Speech
Figure 1: Heterogeneity quantification: Efficiently learning from many modalities requires measuring (1) modality heterogeneity: which modalities different and should be separately processed, and (2) interaction heterogeneity: which modality pairs interact differently and should be separately fused. HIGHMMT uses these measurements dynamically group parameters balancing performance and efficiency.

---

Published in Transactions on Machine Learning Research (05/2023)
et al.
2010:
Lee et al. 2019 Leiva et al. 2020 Liang et al. 2021a Belpaeme et al. 2018 Yeong et al. 2021
This new setting of high-madaklly learning involves learning representations over many diverse modality inputs. As more modalities are introduced, adding new model parameters for every new modality or task (Tsai
ict al.
2019
Jayakumar et al. 2020 Lu et al. 2019 bocomes prohibitively expensive and not scalable (Liang
let al. 2022). A critical technical challenge for efficient. high-modality learning, therefore, is helerogeneily quantification how can we measure which modalities encode similar information and similar interactions in order to permit parameter sharing with previous modalities (see Figure | For example, how can one determine whether the same modality encoder can be shared when processing language and speech, that the same fusion network can be shared when fusing human speech and gestures as well as robot visual and force sensors?
In this paper, we propose a principled approach for heterogeneity quantification via modality information transfer, an approach that measures the amount of transferable information from one modality to another. Our first proposed metric, (1) modality heterogeneity studies how similar 2 modalitics (X1,X22 are by measuring how much usable information can be transferred from X1 to X2, and our second metric, (2) interaction heterogeneity studies how similarly 2 modality pairs X2), (X3, interact by measuring how much usable interaction information can be transferred from {2] to (X3,X4): We show the importance of these 2 proposed metrics in high-modality scenarios as a way to automatically prioritize the fusion of modalities that contain unique information or unique interactions, and otherwise sharing parameters across similar modalities displaying similar information or interactions.
Operationalizing these ideas on a suite of 10 modalities, 15 prediction tasks, and 5 research areas, we show how to train a single model, HIGHMMT, that (1) improves the tradeoff between performance and efficiency over task-specific state-of-the-art models (Liang et al. 2021b Jayakumar et al. 2020 and general multimodal
models with full parameter sharing (Jaegle et al.) 2021b Hu and Singh, 2021 Akbari et al. 2021 Reed
et al. 2022), (2) enables moosl transfer by pretraining on source tasks before transferring to new target modalitics and tasks, and (3) is espocially beneficial for low-resource scenarios (less training data and partially-observe modalities). Beyond these empirical results, we believe that our insights on quantifying heterogeneity and information sharing in multimodal models are independently useful for future work.
2 High-Modality Multimodal Transformer
In this section, we describe our overall approach for high-modality representation learning (sce Figure 2 In 12.1 we formalize modality and interaction heterogeneity to understand whether modalities should be processed similarly or differently. Using these insights, $2.2 describes our proposed HIGHMMT model with dynamic parameter sharing based on heterogeneity measurements.
2.1 Measuring Heterogeneity via Modality Information Transfer
We begin our motivation by formalizing two important sources of helerogeneity in multimodal tasks. Firstly, modality heterogeneity occurs because the information present in different modalities often shows diverse qualitics, structures, and reprosentations. Secondly, interaction heterogeneity occurs because different modalities interact differently to give rise to new information when used for task inference. Formalizing and measuring these two sources of heterogeneity results in actionable insights for building multimodal models: measuring modality heterogeneity enables us to answer: should I use the same unimodal model to encode X1 and X₂? Measuring interaction heterogeneity enables us to answer: should I use the same fusion model to fuse X2) and (X3, X,}? We will formalize heterogeneity via modality transfer, an approach that, mcasures the amount of transfcrable information from one modality to another.
Estimating modality heterogeneity via unimodal information transfer. We propose to measure heterogeneity between modalities X1 and X2 via unimodal transfer. Given a task Y defined over X1 and X2, how well docs an unimodal model trained on the task Y) transfer to (X2; Y)? We choose model transfer as our focus of helerogeneity since it is captured at the level of features extracted via representation learning, rather than at the data-level. Even though the input data may be very different (e.g., images from different cameras or paraphrased sentences), effective feature extractors may be able to learn similar representations from them. Furthermore, directly models task-relevance: the degree of heterogeneity depends on the end task, which enables using these heterogeneity measures subsequently for end-task optimization.
2

---

Published in Transactions on Machine Learning Research (05/2023)
1a. Estimate modality heterogeneity via transfer
2a. Compute modality & interaction heterogeneity matrices
3. Heterogeneity aware model across modalities and tasks
1b. Estimate interaction heterogeneity via transfer
2b. Determine parameter clustering
U1 = {U1, U1,UU, Ua}
Ci = {C12, C13, Cas
U2 = {Ua}
C2 = (C2a)
U3- = {U&}
Figure 2: HighMMT workflow: (1) We estimate modality and interaction heterogeneity via modality transfer to determine which modalities should be processed and fused differently. (2) Using the inferred heterogeneity, we determine the optimal grouping of parameters balancing both total performance and parameter efficiency, which (3) informs our design of helerogeneity--wwe ewaaee aware model with dynamic parameter sharing across many modalities and tasks. HIGHMMT enables statistical strength sharing, efficiency, and generalization to new modalities and tasks.
We formalize unimodal transfer the difference in performance between unimodal models trained X1 before transfer to X2, versus those trained directly on X2- Specifically, we represent an unimodal model using modality X2 with parameters 0 as ÿ = fyy22200) 0). For a suitably chosen loss function L(, y), define the loss of a model as Ep(r (1) 8). y) which measures the expected error over the joint distribution p(#2.9) y). To measure transfer, we train 2 models to obtain an approximation of task performance: the first randomly initialized and trained on the target task giving loss L5
C = min Bx(r..) (( ())
(1)
and the second using initialization from model parameters 0, trained on the source task (X1;Y) before fine-training on the target task giving loss Ci-2-
0. arg min E ( ((
(2)
6
L min (p( (f 01).)),
(3)
where 0 0- 0, denotes parameter initialization with 0,. Intuitively, L3 measures the (baseline) task-relevant information in X2, while fi measures the task-relevant information transferrable from X1 to X2. The differences between these 2 losses,
(4) therefore measures the difficulty of transferring a model trained on the source task (X1; Y) to a target task (X2;Y). Y). Note that computing T(X, + X2; Y) only requires the training or fine-tuning of models across the source and larget modalities, which is efficient. In practice, the expectations over p(z1,y) and p(22,1) are approximated using empirical samples from the training set (for model fine-tuning) and validation dataset (for final evaluation of performance).
T(1: X2; x) Ci
What are some properties of T(X, ~ Y); For very different modalities X, and X2, we typically expect. source task YX to contain less usable information for target task (X2;Y), which would imply that 2i=2 2 L3 and therefore T(X, X2;Y) O (i.c., positive distance). This is consistent with work demonstrating ing negative transfer across different modalities (Liang el al. 2021b Tulving and Watkins, 1974 Wang et al. 2019). Under these scenarios, the larger the positive magnitude of T(X, X2; Y), the more different modalities X1 and X2 are in the context of task Y (more difficult to transfer). However, there can also be cases of zero or even positive transfer (i.e., T(X1 + X2; s 0), even in the surprising case of very different modalities (Lu et al. 2021). These cases reinforce the benefits of feature-based approaches measure heterogeneity: while the raw modalities themselves seem very different, they can still be processed by similar models resulting in positive transfer, and should be assigned a difference of Our final heterogeneity measure X2) aggregates the non- non-negative value (to account for positive transfer) of transfer difficulty statistics across both transfer directions X; X2 and X₂ + X1:
d(Xi;X)) = T(X; X2; Y)2 T(X2 X1;Y)20.
(5)
3

---

Published in Transactions on Machine Learning Research (05/2023)
where Zso max(1 x. 0). Under certain assumptions on the modalities and tasks, our modality heterogeneity measure d(x;; X2) is metric: it satisfies non-negalivily: d( d(X1;X) 2 0, with d(X1; =0 when X1 ===3 X2, and symmetry: d(Xi; X2) Xi),;xi), positivity, X1 # X2 implies that X2) > 0, and a relaxed version of the triangle inequality: X() s d() + Xx). However, in the most general case, there may be settings where positivily and the triangle inequality are not satisfied since the exact dynamics of transfer learning still not well understood for general deep networks: positive transfer can happen (which would imply cases of X1 + X2 but ((x;;X) = -), and in practice, the relaxed triangle inequality is satisfied 96% of the time from a real heterogeneity matrix in Figure 5 (see Appendix A.1 for details).
Estimating interaction heterogencity via crossmodal information transfer. We are also interested in interaction heterogeneity: specifically, how differently should I fuse modalities (2) versus (X3, X We therefore extend to crossmodal transfer by comparing the difference in performance between a multimodal model pretrained on (2; Y) before transfer to (X3:Xxx Y), versus those trained directly on the target task (X3,X4; Y). In other words, we measure the difference in loss between
012 = argmin min Ep(m
T2; 0):
(6)
f2--34 = min E Ep {(f (f x1: 012), v):
(7)
and direct training
Czs min E Ext, 21: 0).
(8)
to obtain X Xx) fi--34 Lzz The distance d(X,, X2, X3, X.) after aggregation over tasks and transfer directions estimates the interaction heterogeneity between {X1, X23 and (X3,X,}
Modality and interaction heterogeneity matrix. Finally, we construct modality heterogeneity matrix Mt(tt,) X,) and an interaction heterogeneity matrix (technically 4D-tensor) Mc k) Xx: X²). Determining parameter groupings to balance both total performance and parameter efficiency can be solved via agglomerative hicrarchical clustering where modalities are nodes and heterogencity measurements are edges. The number of clusters k is treated as a hyperparameter dependent on the parameter budget (see Appendix [A.3] for details, and see clustering examples in $3.1). Clustering on the modality heterogeneity matrix My results in a grouping of modalities based on similarity (e.g., U1 (2,, X2) [X3), {X,3), and likewise for the crossmodal matrix Mc (c.g., C1 = {{X1, X2), (x), (x1, ((X2, ((X4, Xo), (Xs, Xa)), and so on.
Computational complexity. In a high-modality setting, suppose are given a suite of modalities and tasks of the form {(X,, Yi), (X1, X3, X2) ...} and so on, where there are a total of M unique modality and task pairs {(X1, (X2, Vi), (X1, (X3, Y2), (X4, Y2), ... In practice, the number of unique (pairwise) interaction and task pairs X2, X3, Y2), (4,, (X,, Y2..... is also O(M), since the maximum number of modalities jointly observed for a task is never above a constant (at most in all real-world datasets, and often 2 or 3). As an example in Figure 5 our experiments involve M- 10 modality and task pairs (across 4 tasks defined on 2, 2,3 and 3 modalities respectively), and 8 (2) (2) +++(3) interaction and task pairs.
The modality heterogeneity matrix for M unique modality and task pairs has ((M - 1)/2 unique entries after removing the upper triangular portion due to symmetry and diagonal entries since d( ((,,, X,) = O. Computing these M(M - 1)/2 entries exactly requires one to first train M unimodal models (to estimate the M & terms) before fine-tuning M(M-1) transfer models (to estimate the M(M-1) Ci-- terms), for a total of M² pre-trained and fine-tuned models. The interaction heterogeneity matrix also requires O models for exact computation. However, we find that key approximation can be made in practice: the heterogeneity matrices are highly structured due to distances approxinately satislying the triangle inequality, which implies that we do not need to compute all entries and instead rely on low-rank reconstruction from partial entries in practice. In our experiments, even using a low-rank approximation of = 3 is sufficient to approximate the entire matrix. This suggests that we do not need to exhaustively measure unimodal and interaction transfer between all modality pairs to enjoy the benefits of our proposed approach. Instead, running a random sample of O(M) pairs of heterogeneity values, and imputing the rest of the heterogeneity matrix, is sufficient in practice. Please an example heterogeneity quantification for real- weal-world datasets in $3.1 and Appendix A.2 for further details.
4

---

Published in Transactions on Machine Learning Research (05/2023)
Language
Shared
Z1
Muttioool
[ 22-11
unimonia enooder
cross attection
unimodial
Z2
Multimori attontion
[2 -n: Zz-1
dbssstr
Prodiction
X3
Shored unimocil choodat
Za
Muttimooal stltaniio
72 7g
2
Scria
Altention Keg,
-poiitrral arozding
(0)
emboddings
Festurex
Attention Queryy
parameter sharing
Latent vectors
(pencoonlyy intiadiod & Iniiable)
Figure 3: HIGHMMT architecture: Given arbitrary modalities, (1) the inputs are standardized into a sequence and padded, (2) modality embeddings and positional encodings added to the input sequence, (3) single shared unimodal Perceiver encoder is applied to all modalities to learn modality-agnostic representations, (4) each pair of unimodal representations is fed through a shared multimodal cross-attention layer to learn multimodal representations, and finally (5) all outpuls are concatenated, balch-normalized, and fed into lask-specific classification heads.
2.2 Capturing Heterogeneity and Homogeneity in HighMMT
Using these insights, modality representation across many modalities and tasks (see Figure 3 Training the HIGHMMT model consists of 2 main steps (sec Figure + (1) homogeneous pre-training of fully shared model across all modalities, before (2) helerogeneilly-wuore fine-turing to respect modality and interaction heterogeneity.
describe architecture for general model HIGHMMT suitable for high-
Homogeneous pre-training, We first design a homogeneous multimodal model fully shared across all modalitics and tasks with the following key components (sec Figure 3 and details in Appendix B
1. Slandardized inpul sequence: We first standardize modalities as a sequence of embeddings, as is already done for sequential data such as text, audio, and time series, and recently adapted for image patches too (Dosovitskiy et al.] 2021). For tables, sets, and graphs we treat each element in the aable/set/graple as an element in the sequence. The end result is a standardized input data Xm of dimension 4m x dm, where im is a modality and task-specific input sequence length, and dm is a modality and task-specific input dimension.
2. Modality-specificc embedding and positional encoding. For each distinct modality m M (which may appear across multiple tasks), we define a one-hot hot modality embedding e. RIM, where [M] the total number of distinct modalities, to identify common modalities across different tasks for information sharing. We also introduce Fourier feature positional encodings Pm E Impmm where dpm is the positional encoding dimension, to capture positional information across each modality. For multimodal tasks where a common dimension is shared across time (e.g,, videos/ / time series), we apply common positional encoding to capture the common time dimension (i.c., the first image frame occurs at the same time as the first word and first audio segment). Finally, the processed modality m is given by concatenating Xm Xmm @ em + Pm @ O,, (i.e., the input sequence, modality embedding, positional encodings and zero-padies (rro-pddding) into standard dimension fm x daut. dau max (dm, |M] dpm) where dm is the channel size of modality m, dpm is the positional encoding size of modality m, and [M] is the modality encoding size (i.c., the total number of involved modalities).
3. Shared unimodal networks. Now that we have standardized all modalities into common format, we design general unimodal encoder with parameters U via Transformer-based Perceiver block (Jaegle et al.) 2021b). Our model recursively trains a latent array Zm of shape dNN dLs (where NNN is the sequence length number of latent vectors and dis is the latent dimension) that is random initialized as 2(()) For each layer L starting with a previously-oopptter
representation z(!-11, we first perform crossaattention from the processed input (Xm of shape tm x dau) to ((L-1) obtaining an intermediate representation (()), before self-attention and layers on Z(L)) resulting in new representation zi(L) for input the next layer:
QoK? Vidrss
((L-1) W WT V.
XT
(L) CROSS AFTENTION(Z1L (((11), ((()))xmm)) softmax m
V. = softmax
X,, Wy.
Vdr.s
(9)
7(L) Wo. W.T. Z(L)T
Qak] Vdus
2(4) SELF ATTENTION (2())) softmax
V. softmax
2(L) Wv,,
(10)
Vdr.
5

---

Published in Transactions on Machine Learning Research (05/2023)
1. Homogeneous Pre-training
2. Heterogeneity-awan aware Fine-tuning
Yi
Y2
Y3
Y4
Y1
Y2
Y3
Y1
Ci
C2
C*
C3
U1
U
U1
U
U
U
U
U
U
U
U
U
W3
Figure 4: HighMMT training involves 2 steps: (1) homogeneous pre-training of fully shared model across all modalities, before (2) heterogeneity-awwrr fine-tuning of modality and interaction parameters in different groups to respect modality and interaction heterogeneity respectively.
with trainable cross-attention parameters Wo. e &RdsssdLe WK, R Wv, Raaitddge and self-attention parameters We, e Rduss*dus WK, Rosssdds .W, Rddssddis, Repeating cross- and self-attention between the latent vector and the input modality summarizes the relationships between modality elements into the latent vector, resulting in a final unimodal representation Zm E IRdLNxdEs, Summarizing all information into a common dI,x x drs latent array regardless the input shape tm x dan results in total runtime only lincar with respect to the size of im and daul which scales to high-modality scenarios.
4. Shared crossmodal networks. To learn multimodal representations, we use a shared Crossmodal Transformer block with parameters C [Tsai et al. 2019 Lu et al., 2019). Given 2 unimodal representations Zi and Za of common shape dLN x dLs learned from unimodal Perceiver encoders, Crossmodal Transformer (CT) block uses crossmodal self-attention by setting the input layer query Q Z1 and keys and values K, V Zz to learn atlention from Z2 to Zi, and separate block to caplure the attention in the opposite direction.
Q VTz
Zi Z.WQ. W. Z
Z CROSS ATTENTION 22) softmax
V/2 softmax
Zo W2
(11)
vdk
and vice-versa for Z1-2, with parameters Wo Wo, Rd WK, WK, e Redesdde Wv, Wv, e Resxsd, This step enables one modality's elements discover bidirectional interactions with another, resulting in final multimodal reprosentation Zumm = =[Z1-2, Z2-1] of shape d 2dg. For each layer, we first perform cross-atiention followed by self-attention and feed-forward functions. For tasks with more than 2 modalities, a Crossmodal Transformer block is applied for each pair of modalities before concatenating all representations.
5. Task-specific classifier and multitask pre-training. Finally, on top of Zmm we use a separate lincar classification layer per task. To enable information sharing across modalities and tasks, homogeneous pre- training is performed across diverse set of datasets in a multitask manner by optimizing a weighted sum of losses over tasks. The result is single set of shared unimodal parameters U* that encodes all modalities, and single set of shared crossmodal parameters C* that captures all pairwise interactions between modality pairs, along with all mopelity-specific embeddings E* and task-specific classifiers T*.
Heterogeneity-wwan fine-tuning. Finally, we account for heterogeneity by grouping unimodal parameters based on modalities that we know to be similar from 2 (e.g., setting U, (U, U2), 12, = (Uz), U, (U1, Us, Uo}), and likewise for the crossmodal parameters (e.g., C = [C12, Cla. Cu.. C2 = (C23, C1s), C3 (C24....)). From Figure these parameter groups are first initialized with the homogencous model U and C* before separate fine-tuning, which results in final parameters U (Ui, Už.... and C modality embeddings E' and task classifiers T are jointly fine-tuned well. Fine-tuning is also performed in multitask manner by optimizing weighted sum of supervised losses across all modalities and tasks.
Cc.... The
3 Experiments
Sctup: this section, we design experiments analyze the multitask, transfer, and generalization capabilities of HIGHMMT. We use large collection of multimodal datasets provided in MultiBench (Liang al. 2021b spanning 10 modalities, 15 prediction tasks, and 5 research areas. We trained multitask models across

---

Published in Transactions on Machine Learning Research (05/2023)
Table 1: We investigate multitask setup to evaluate the performance of HIGHMMT across different modality inputs and prediction objectives. The total size of datasets involved in experiments exceeds 370, 000 and covers diverse modalities, lasks, and research areas.
Datasets ENRICO
Modalities
Prediction task
Research Area
(image,set (text, video, audio)
1, 1,460
design interface
HCI
UR-FUNNY
16, 614
humor sentiment, emotions
Affective Computing
MOSEI
[text, video, audio)
22, 777
Affective Computing
MIMIC
(time-series, table)
36, 212
mortality, ICD-9 codes
Healthcare
PUSH
(image, force, propriaception, control)
37, 990
object pose
Robotics
AV-MNIST
{image, audio)
70, 000
digit
Multimedia
V&T
(image, force, proprioception. depth)
147. 000
contact, robot pose
Robotics
MIMIC t-te series
AV-MNIST image
AV. AV-MNIST audio
UR-FUNNY video
UR-FUNNY audio
UR-FUNNY text
MIMIC table + timeseries
AMMNIST image audio
UR-FUNNY video audio
UR-FUNNY video text
UR-FUNNY audio text
MOSEL video
MOSEI audio
MOSEI text
MOSEI video audio
MOSEI aucio text
MIMIC
MOSEI video
MIMIC table
MIMIC time series
0.4
AV-MNIST image
20
20
MIMIC table timeseries
AV-MNIST audio
4.1
5.0
0.2
AV-MNIST image audio
23
MOSEI video
0.2
20
4.1
MOSEI video audio
0.8
22
MOSEI audio
4.1
MOSEI audio 0.4
video text
MOSEI text
UR-FUNNY video
UR-FUNNY video audio
UR-FUNNY audio
UR-FUNNY video text
0.4
UR-FUNNY text
20
4.6
UR-FUNNY audio text
0.5
0.9
Modality heterogeneity matrix
Interaction heterogeneity matrix
Figure 5: Modality and interaction heterogeneity matrices color coded by distances, with green showing smaller distances and dark red larger distances. We find clear task outliers (AV-MNIST has high difficulty transferring others), and that there is generally more interaction heterogeneity than unimodal heterogeneity. Otherwise, the same modality and modality pairs across different tasks are generally similar each other.
combinations of these datasets (see Table I and Appendix C for details). Overall, the total size of datasets involved in our experiments exceeds 370, 000 and covers diverse modalitios such images, vidco, audio, text, time-series, robolics sensors, sets, and tables, prediction tasks spanning the image-caption matching, robot pose, object pose, robot contact, design interfaces, digits, humor, sentiment, emotions, mortality rate, and ICD-9 codes from the rescarch arcas affective computing, healthcare, multimedia, robotics, and HCI.
3.1 Heterogeneity Measurements and Parameter Groups
We begin with a study of the heterogeneity matrices in Figure and the resulting parameter groups.
Modality heterogeneity: We first notice that the modalities from AV-MNIST only transfer well to each other and has high difficulty transferring to other modalities from the other datasets. The same modality across different tasks is generally similar each other (e.g., text between UR-FUNNY and MOSEI, audio between UR-FUNNY and MOSEI). The text modality in UR-FUNNY scems to be close to most other modalities, and likewise for the tabular modality in MIMIC. It also worth noting that the video and audio modalities are not the most informative in MOSEL, and predictions dominated by language (Zadeh et al. 2017 which may explain their general homogeneity with respect to other modalities.
Interaction heterogeneity: There generally more interaction heterogeneity than unimodal, implying that the interactions between modality pairs tend to be more unique. Again, we notice the general poor transfer from the modality pair (imago+andio) in AV-MNIST to other pairs, and the general strong transfer from (audio+text) in UR-FUNNY the rest, which shows relationship between modality and interaction heterogeneity. We also find that the same modality pairs vrideo+text) and (video+audio)) shows crossmodal
7

---

Published in Transactions on Machine Learning Research (05/2023)
similarity across both datasets they appear in: MOSEI and UR-FUNNY. Finally, while the triplet of crossmodal pairs in MOSEI are quite different from each other, those in UR-FUNNY are more similar.
Using these measurements, Appendix C.2 includes the final groups of parameters obtained after clustering the matrices for different values of k. As an example, for [U] = 3. 3.|C| = 3, k = 6, the groups are U1 = {AV- MNIST image, AV-MNIST audio), ; U2 = (MTMTC table, MOSEI video, MOSEI audio), 43 = (MIMTC timeseries, MOSEI text, UR-FUNNY text, UR-FUNNY video, UR-FUNNY audio), and C₁ = (AV-MNIST image+andio), C2 (MOSEI vidco+audio), and C3 {MIMIC table-timeseries, MOSEI video-textt MOSEI audio+text, UR-FUNNY video+text, UR-FUNNY video, UR-FUNNI audio+text)..
Finally, we observe the low-rank nature of the heterogeneity matrices due to symmetry and approximate triangle inequality, such that even using a low-rank approximation of r 3 is sufficient to approximate the entire matrix. This suggests that we do not need to exhaustively measure unimodal and interaction transfer between all modality pairs to enjoy the benefits of our proposed approach (scc more details in Appendix A.2
3.2 Qualitative Results
We now present our results on the multitask; transfer, and generalization capabilities of HIGHMMT using performance and efficiency metrics. Henceforth, we will refer to the following models:
(1) HighMMT share none refers to individual copies of HIGHMMT models, one for each task.
(2) HighMMT share all refers to one single HIGHMMT model fully shared across all modalities and tasks.
(3) HighMMT refers to the full helerogeneily-awarr HIGHMMT model across all modalities and tasks with learned parameter groupings based on heterogeneity measurements.
Multitask performance and efficiency. In Figure G we summarize the overall tradeoff between performance and efficiency using existing task-specific models and variants of HIGHMMT. The blue dots represent all possible com- binations of task-specific models across multiple datasets (summarized in MultiBench (Liang et al. 2021b), > 10 total combinations) with their overall performance (scaled to a 0- range before averaging across datasets) and overall ef- ficiency (inverted total number of parameters). The red dots represent the state-of-tho-art Pareto front: points that not strictly dominated in both performance and efficiency. In light green, separate single-task HIGHMMT models (share none) already improve parameter efficiency as compared to standard Multimodal Transformers Lu et al.) 2019 Tsai let al.] 2019). In dark green HIGHMMT (share all) trained in homogeneous multitask manner (i.e., with full param- eter sharing across unimodal and multimodal layers within and across tasks), which further pushes forward the Pareto front by improving both performance and efficiency. Finally, in orange, HIGIMMT with heterogeneity-awarr fine-tuning achieves significantly better tradeoffs between performance and efficiency, with efficiency and consistently high perfor- mance across multiple modalitics and tasks.
Performance
1080
1
1005 Efficiency (params) -
2020
1060
All task-specific model
A HighMMT share
combinations (=10,000)
HighMMTT share all
Pareto front
* HighMMT (heterogeneity
Figure 6: Overall tradeoff. HIGHMMT pushes forward the Pareto front of performance and ef- ficiency as compared to all possible (> 10?) com- binations of task-specific models across multiple datasets (Liang el al. 2021b). The x-axis denotes (inverted) total parameters and y-axis denotes per- formance scaled to a 0-1 1 range before averaging across datasets.
The suite of HIGHMMT models is obtained by tuning k, the total number of unimodal and crossmodal parameter groups (i.e., the mum- ber of clusters when clustering heterogeneity matrices). ho can be seen as a hyper-parameter depending on the computational budget, with smaller &o implying more parameter sharing on lower budgets and vice-versa. In Table 2 we show the effect of & on average per- formance and total parameters. We test & in the range (2,4, 4.6,77.0 7.9). with [J] 1, |C| = [U] 3, 3C| J 3, [C| 3, |U| 3, |C| 4, and
Table Tuning the number of parame- ter groups results in controlled tradeoffs between parameters and performance.
Clusters 2 (share all)
Performance Params (M)
6.4
1.07
68.8
1.24
70.1
2.47
7
71.0
3.11
9
71.2
4.23
8

---

Published in Transactions on Machine Learning Research (05/2023)
Table 4: HIGHMMT achieves strong performance on overall performance and efficiency (mean and deviation over 10 runs), sometimes even beating (shown in bold) the task-specific state-of state-of-the-art, especially on the relatively understudied modalities (time-series, robotics sensors, and sets) from the robotics (PUSH, V&T) HCI (ENRICO),
and healthcare (MIMIC) research areas, while using 10x fewer parameters due to parameter sharing and mullitask
learning. SOTA captures the max performance and parameters of more than 20 lask-specific multitooal models: [1]
GRADBLEND [Wang et al. 2020), [2] LF-LSTM (Ding et al. 2022),, [3] LF (Gadzicki et al. 2020), [4] MULT (Taa
et al.
2019
[5] MFAS (Pérez- Rúa et al.
2019), [6] MFM (Yang et al. 2022 and [7] LRTF (Zeng et al.
2022).
Model
E
PUSH
V UR-FUNNY 1 MOSET f
MIMIC 1 68.9 68.9 0.5,3,6,7
AV-MNIST Params (M)
SOTA
51.0 1.40 0.290 + 0.
93.6+ 0.1[3]
66.7 0.3[4] 82.1 + 0.5[4]
72.8 + 0.2
32.3
HIGHMMT
52.7 + 0.6 0.277 + 0.1 96.3 + 0.2
6.42
80.2+ + 0.2
68.2 + 0.3
71.1+ 0.2
3.01
4C 5 respectively where [U], [C] denote the number of unimodal and crossmodal parameter groups. We see a controllable tradeoff: starting with fully shared model and increasing the number of parameter groups, we also see steadily improving performance approaching task-specific state-of-the-art models. Overall, optimizing for performance results in a model as strong as current state-of-the-art models while using 8x fewer total parameters. Optimizing for efficiency results in a model that reaches within 96% of current state-of-the-art performance but using 30x fewer total parameters (mean and deviation over 10 runs).
Positive transfer to new modalities and tasks. HIGHMMT also offers opportunities to study whether we can transfer knowledge between completely dif- ferent modalities and tasks. Starting with the collec- tion of 4 datasets in the order MOSEI, AV-MNIST, MIMIC, and UR-FUNNY ranked by largest dataset size (total of datapoints and memory storage per datapoint), we pre-train a fully-shared HIGHMMT model on 1/2/3 of the 4 tasks before fine-tuning on the fourth task only (e.g,, train on MOSEI and transfer to UR-FUNNT on MOSEI+AV-MNIST then transfer to UR-FUNNY, and on MOSEI+AV- 3
Table 3: Cross-modal few-shot transfer to new modalities and tasks, We train multitask HiGHMMT on 1/2/3 datasets and find that it generalizes few-shot to new modalities and tasks on the 4th dataset, with improved performance over single-task training on the 4th dataset. Cross-modal transfer improves with more pretraining tasks and works best on the smallest target tasks (UR- FUNNY..
Target task
#
Source tasks
UR-FUNNY
MOSEI
MIMIC
AV-MNIST
(no transfer)
6.5
79.0 67.7
70.3+0.4
63.
0.3
0.5
70.5 0.4
2
6.70
0.5
70.5 6..
64.7
79.6 68.466841100 0.6
70.6
MNIST+MIMIC then transfer to UR-FUNNY,
and likewise for transfer to the other datasets (sce full pre-train and transfer combinations in Appendix D.1
From Table 3 we found that on all four combinations of multitask pretraining and fine-tuning, weights learned from other multimodal tasks generalize well to new modalitics and tasks, improving performance over single larget-task training (mean and standard devialion over 10 runs). When we increase the nunber of pretraining datasets, we observe a consistent improvement in fine-tuned target task performance. There is an inverse correlation between target task size and performance improvement: the smallest datasct, UR-FUNNY, benefited the most (+2.4%) from transfer learning from 0 to 3 multitask datasets. This implies that our multimodal
paradigm is useful for low-resource target modalities and tasks.
Finally, compare transfer learning performance across different levels of partial observability. While one would expect the transfer to MIMIC to be the hardest due to its modality set (time-series, table) being completely disjoint from the remaining 3 datasets, we still observe a +0.8% gain as compared to single-task training. Therefore, HIGIMMT can generalize to new modalities and tasks. Unsurprisingl, for datasets with more overlap (e.g., UR-FUNNY with complete overlap in (text, video, audio} with respect to pretraining), we find larger improvements using transfer learning over single-task models (+2.4%).
Comparison with task-specific state-of-the-art. In Table 4 we compare multitask performance and efficiency with task-specific state-of-the-art models. We achieve performance within the range of published models (and usually close to the individual task-specific state-of-the-art) in MultiBench, which tallies more than 20 recent multimodal models in cach task's literature (Liang et al. 2021b). In fact, HIGHMMT even sets new state-of-the-art results on several datasets, especially on the relatively understudied modalities (time-series, force and proprioception sensors, and sets) from the robotics (PUSH, V&T) and HCI (ENRICO) rescarch arcas. On top of strong performance, the main benefit lies in using fewer total parameters as compared to separate task-specific models more than 10x reduction. Since this reduction grows with the number of tasks, our approach is scalable to high-modality scenarios.
9

---

Published in Transactions on Machine Learning Research (05/2023)
Table 5: We conduct in-depth ablation studies and find strong evidence for (1) having separate unimodal and interaction layers, (2) determining parameter sharing via feature transfer, and (3) homogeneous pre-training before
heterogeneity-wwarr fine-tuning into parameter groups (mean and standard deviation over 10 runs).
Model
UR-FUNNY MOSELT MIMIC 1 AV-MNIST
Ave 1
Full model
HIGHMMT
66.2 = 0.4
0.2
68.2 = 0.3
71.1 0.2
71.4 0.3
w/o embeddings
63.0 1.2
07.1 7970
1.2
0.7
69.8 0.3
Architecture
w/o unimodal
57.9 0.3
61.9
63.0
59.5 1.4
60.6 0.7
ablations
w/o crossmodal (Reed et al.
2022
63.8 1.0
7
0.4
70.4 0.5
share none (Llang al. 202
63.7+
79.4+0.4
0.7
0.1
70.2 0.3
share unimodal (Reed
2022
62.5 1.3
79.0+
1.4
7.77
68.8 0.8
Param sharing
share crossmodal (Akbari al. 2021
63.0
7.3
6.3
0.9
69.2 = 0.3
ablations
share all (Singh ct al. 2021
6.7
7.3
1.6
6.6
68.7 0.5
random difference
6..9
7.6
67.6
0.2
70.1 = 0.3
feature difference (Sun et al.
2016
64.0 1.0
7.34
67.9
0.4
70.4 + 0.2
Training ablations
w/o homogeneous pretraining
61.2
78.5
64.8 0.1
0..1
69.9 & 0.1
Partial-observability Observe HIGHMMT performance on partially-observable modality subsets (i.e., target task involving modalities not present in the other tasks): from Table 4 we find that the model performs well on the MIMIC dataset despite its modality set (Lime-series, table} being completely disjoint from the remaining 3 datasets we obtain similar performance across both multitask and single-task models (68.2 + 0.3% vs 68.9 0.5%). We find that HIGHMMT multitask also works on ENRICO datasct HCI (52.7 + 0.6% multilask vs 51.0 + 1.4% single-task) despite it having completely disjoint modality inputs.
Multitask fusion and retrieval. We perform multitask training over multimodal fusion in AV-MNIST and retrieval in CIFAR-ESC. While fusion emphasizes information integration, retrieval focuses on aligning corresponding elements expressed through different views of the data (Liang et al. 2022 Even across these vastly different prediction tasks, we find that multitask training (60.5% retrieval accuracy) improves upon single-task training (58.8%) Not only have the unimodal networks simultaneously processed different modalities, but the crossmodal network has captured correspondences useful for both fusion and retrieval.
3.3 Ablation Studies
In this subsection, we carefully ablate the model architectures, parameter sharing, and training decisions.
Architectural ablations. We first analyze each architectural component of HIGUMMT: (1) w/o embeddings removes the only mopelity-specific component in the model the modality embeddings. We set embeddings for all modalities to be the same to test whether modality-specific component is necessary to capture heterogencity across input data sources, (2) w/o unimodal removes the unimodal encoder and directly applics the cross-attention layer, and w/o crossmodal replaces the crossmodal layer with concalenation of unimodal features and a linear classification layer. The latter resembles the most direct multimodal extension of existing work in sharod unimodal encoders like Perceiver (Jacgle et al.) 2021b), MultiModol (Kaiser et al., 2017); ViT-BERT (Li et al., 2021 or ViT (Likhosherstov et al. 2022). From Table 5 removing any of the 3 components in HIGHMMT results worse performance. The unimodal encoder is particularly important.
Param sharing ablations. We further ablate with respect to possible parameter sharing scttings in HIGHMMT: (1) share none uses separate unimodal and multimodal layers reminiscent of typical single-task
multimodal transformers (Tsai et al. 2019 Lu et al.] 2019 Hendricks et al. 2021), ,, (2-3) share unimodal
(crossmodal) only shares the unimodal (crossmodal) layer during multitask training, (4) share all shares
all parameters without accounting for possible heterogeneity (Reed et al. 2022 (5) random difference
determines & parameter groups randomly rather than via heterogeneity measurements, (6) feature difference
uses fcaturo-lovel divergences on jointly trained unimodal encoders (i.c., [|U(()) U(x)) rather than transfer performance to measure helerogeneity as is commonly done in transfer learning and domain adaptation @Dauéé III 2007 Sun et al. 2016). From Table 5) our proposed heterogeneity- aware parameter grouping results in the best overall performance as compared to fully shared, fully separate, or parameter grouping informed by other heterogeneity measures such as random or feature distance.
Training ablations. Finally, we explore w/o homogeneous pretruining: directly learning a model with parameter groups as selected by our approach as opposed to performing homogencous pro-training before fino-tuning them into parameter groups. From Table 5 we find that this ablation underperforms training parameter groups from scratch overfits to smaller datasets which hurts overall performance.
10

---

Published in Transactions on Machine Learning Research (05/2023)
3.4 Understanding homogeneity and heterogeneity in HighMMT
We now take a deeper empirical analysis better understand HIGHMMT, through parameter overlap and interference experiments. We investigate other model properties and visualizations in Appendix [D4
Starting with trained multitask HIGHMMT, we use a gradient-based method (Han et al. 2020) to determine how much each parameter is involved in a specific task. For each task T and (r) = ( where Me (y(z) is the predicted probability of correct target y by Me given æ as input. In other words, this measures the absolute gradient with respect to when predicting y given x task T. A higher absolute gradient implies "activated" neurons and vice-versa for gradients closer O. This enables us to compute the extent parameter 0 is involved for cach task. The number of tasks given parameter @ is involved in can then be approximated by thresholding and summing up n(0) Σr (0) e I1 (0)) (2), (()))) which returns an integer from 1 to 4. We chose threshold such that parameters are classified as active about half the time on average, which occurs at E 0.2.
Parameter overlap.
parameter e θ in multitask model Me, we compute the involvement
Since we are interested in the level of parameter overlap in the shared unimodal encoder and multimodal layer, we set 0 as these 2 modules and report results in Table 6 There is evidence of significant parameter overlap across unimodal encoders: more than 92% of neurons are involved in at least 3 of the 4 tasks. On the other hand, there is not nearly as much parameter overlap in the multimodal layer: only 10% of neurons are involved in 3 or 4 tasks. Hence, seems like the unimodal encoders Iearn task-agnostic representations, but the subsequent multimodal layers (closer task-specific classifiers) capture more task-specific information. This also reinforces our observation in $3.1 that there is generally more interaction heterogeneity than modality heterogeneity, which suggests using fewer unimodal parameter groups and more crossmodal parameter groups.
Table 6: We find evidence of significant parame- ter overlap across unimodal encoders: 92% of neurons are involved in at least 3 of the 4 lasks, while the multinodal layers are more lask-specifie: only 10% of neurons are involved in 3 or 4 tasks.
Number involved tasks
Component
Unimodal layers
2.8%
5.1%
61.1%
81.1%
Crossmodal layers
48.8%
39.7%
9.9%
1.6%
Parameter interference. Another empirical proof for parameter sharing in multitask models is the phenomenon of parameter interference: what extent do parameters interfere with each other across tasks? We perform an experiment to inves- tigate parameter interference; we pick one task and flip the labels in its training set, train the multitask model on the modified training set, and see how the incorrectly labeled task affects perfor- mance on other lasks. This experiment provides evidence of information sharing: if the multitask model does not share information (i.c., the model learns independent subspaces for each task), then one would not observe negative interference from one noisy dataset. We study negative interference under 3 configurations of training (a) the whole model; (b) only the unimodal encoder, and (c) only the multimodal layer on the flipped training sot..
Table 7: Parameter interference: we observe different performance drops on each task (columns) after training on one task with flipped labels (rows). Training the shared unimodal encoders causes the most harm, which implies that unimodal encoders contain more shared neurons sensitive to task changes. Red for drops greater than 20%, yellow for drops between 10 and 20%, and green for drops below 10%.
(a) Training entire model
Flipped task
UR-FUNNY MOSEI MIMIC AV-MNIST
UR-FUNNY
-21.6
-8.83
-10.6
-57.7
MOSEI MIMIC
-1.07
-9..7
-0.33
-53.2
-3.59
-5.83
-33.1
-37.5
AV-MNIST
-3.50
-1.23
-4.87
-68.9
(b) Only training unimodal encoder UR-FUNNY MOSEI MIMIC AV-MNIST
Flipped task
UR-FUNNY
23.8
-10.1
-12.8
-58.4
MOSEI MIMIC
-5.77
-57.6
21.1
-52.7
-3.03
-3.54
-35.0
-56.3
From Table 7 certain tasks are more affected by negative interference (e.g., AV-MNIST), while some tasks are not influenced as much (c.g., UR- FUNNY). Again, this reflects our heterogeneil measurements in $3.1. where AV-MNIST displays high heterogeneity. Furthermore, performance drops due to training the unimodal encoders are the most significant, which corroborates with our
AV-MNIST
-2.94
-7.82
-53.6
-69.3
(c) Only training multimodal layer UR-FUNNY MOSEI MIMIC AV-MNIST
Flipped task
UR-FUNNY
-25.2
-8.34
-2.67
-8.16
MOSEL
0.47
-59.6
-19.8
-8.19
MIMIC
0.19
-0.76
-35.2
-4.87
AV-MNIST
-1.61
-1.48
-2.23
69.1
11

---

Published in Transactions on Machine Learning Research (05/2023)
parameter overlap and heterogeneity analysis that unimodal encoders contain more entangled parameters which are more sensitive to task changes. On the other hand, multimodal layers contain more disentangled parameters, which results in higher heterogeneity measurements and needs more separate parameter groups.
4 Related Work
Multimodal Transformers have emerged as strong models for representation learning. Building upon the Transformer (Vaswani et al. 2017); multimodal extensions use either full self-attention over modalities
concatenated across the sequence dimension (Li et al. 2019 Sun et al. 2019 Su al. 2020 Chen al.) 2020)
or cross-modal attention layer (Lu ct al.] 2019 Tsai et al. 2019 Tan and Bansal, 2019 and are uscful for
sequential data by automatically aligning and capturing complementary features al diflerent lime-steps Tsai
et al. 2019 Yao and Wan 2020 Lee et al.) 2020c). Self-supervised multimodal pretraining has emerged as an effective way to train these architectures, with the aim of learning representations from large-scale unlabeled multimodal data before transferring to downstream tasks via fine-tuning (Lu et al.) 2019 Li et al. 2019
Su et al. 2020). These pretraining objectives typically consist. of unimodal masked prediction, crossmodal masked prediction, and multimodal alignment prediction (Hendricks al. 2021
Unified encoder for unimodal learning. Several works such as Perceiver (Jaegle et al. 2021a by, MultiModel (Kaiser et al. 2017), VIT-BERT (Li et al. 2021), and PolyViT (Likhosherstovv et al. 2022 have explored the possibility of using the same architecture for different inputs on unimodal tasks (i.c., language,
image, video, or audio-only). The Transformer architecture has emerged as a popular choice due to its
suitability for serialized inputs such as text (Devlin et al. 2019 images (Dosovitskiy et al. 2021), video (Sun) ct al.) 2019), and time-scries data (Lim et al.] (022), a phenomenon further observed by Lu et al. (2021) where a single Transformer pretrained on text transfers to sequence modeling and image classification. While these serve as building blocks in our model, our focus is on a general-purpose multimodal model for multitask and transfer learning across different subsets of modalities rather than unimodal tasks.
Multimodal multitask and transfer learning. There have also been several attempts to build a single
model that works well on a suite of multimodal tasks (Li et al.) 2019: Lu et al. 2019 Su et al. 2020 Cho et al.) 2021 Reed et al. 2022). For example, UniT (Hu and Singh 2021), VLBERT (Su et al. 2020), ViLBERT (Lu et al.]
| 2019). and VL-T5 (Cho et al., 2021) are all unifying models for vision-ande lision-and-language
tasks. VATT (Akbari et al. 2021) jointly trains shared model on video, audio, and text data to perform audio-only, video-only, and imago-text retrieval tasks. FLAVA (Singh et al. 2021] found that pretraining shared model with unpaired images, unpaired text, and imagee-exx pairs results in strong performance on image-only, text-only, and image-text multimodal tasks, while while(Reed Reed et al. (2022) scales up a single Transformer model for image, text, and decision-making tasks. However, all of these train a single model for all tasks, without investigating how heterogeneity can necessitate partial parameter sharing. On the transfer side, while more research has focused on transfer within the same modality with external information (Socher el al. 2013 Dunnmon et al. 2019: Xing et al., 2019 Zadeh et al. 2020). Liang et al. (221cc) is the only work that studies transfer to completely new modalitics. However, they require paired data collection and modality-speciftt modeling. Our work goes beyond the commonly studied language, vision, and audio modalities to relatively understudied ones (e.g,, tabular data, time-series, sensors, graphs, and set data). Furthermore, we show the possibility of generalizing to new modality subsets. Finally, our work also complements studies of transfer learning in a single modality (Standley et al. 2020 Wu et al. 2022 Zamir et al. 2018). where insights from task heterogeneity have informed multitask approaches, as well as multisensor fusion in various domaïns such as healthcare [Muhammad et al. 2021) and robotics (Suzuki and Matsuo 2022 Taniguchi et al., 2023).
5 Conclusion
We propose an information transfer approach for estimating modality and interaction heterogeneity, key component towards automatically determining which modalities should be processed and fused jointly for efficient representation learning in high-modality scenarios. Our resulting model, HIGHMMT dynamically determines the optimal parameter groupings balancing total performance and parameter efficiency, simultane- ously achieves strong results on modalities (text, image, vidco, audio, time-scries, sensors, tables, and sets) and tasks from different research areas, and transfers to new modalities and tasks during fine-tuning, We release our code and benchmarks which we hope will present unified platform for subsequent analysis.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Acknowledgements
This material is based upon work partially supported by Meta, National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards R01MH125740, R01MH132225, RO1MH096951 and R21MH130767. PPL is partially supported by a Facebook PhD Fellowship and a Carnegie Mellon University's ss Center for Machine Learning and Health Fellowship. RS is supported in part by ONR N000141812861, ONR N00014231220 and DARPA/APUR
FA87502321015. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF, NIH, Meta, Carnegie Mellon University's 's Center for Machine Learning and Health, ONR, DARPA, or AFRI, and official endorsement should be inferred. We are extremcly grateful to the action editor and reviewers for their helpful discussions and feedback. Finally, we would also like to acknowledge NVIDIA's 's GPU support.
References
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. VQA: Visual question answering. International Journal of Computer Vision, 2017.
Hassan Akbari, Liangzhe Yuan, Rai Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. arXio preprint 1rXi:2104.1111 11178 2021.
Paras Malik Amisha, Monika Pathania, and Vyas Kumar Rathaur. Overview of artificial intelligence in medicine. Journal of family medicine and primary care, 8(7):2328, 2019.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Phils-Phlliipe Morency. Multimodal machine learning: survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41 (1(2):233-44 443, 2018.
Jonathan Baxter. A bayesian/inforre itffrmaini theoretic model of learning to learn via multiple task sampling. Machine leurning, 28 (1):7-39, 1997.
Tony Belpaeme, James Kennedy, Aditi Ramachandran, Brian Scassellati, and Fumihide Tanaka. Social robots for education: A review. Science robotics, 3(21), 2018
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In E'ropeon conference computer vision, pages 104-120, Springer, 2020.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and- lision-and-lannuuae tasks via text generation. In ICML, 2021.
Hal Daumé III. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguisfics, pages 256-263, 2007.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshus Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium User Interface Software and Technology, pages 845 845-854, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers language understanding, In NAACL-HLT (1), 2019.
Ning Ding, Sheng-wei Tian, and Long Yu. A multimodal fusion method for sarcasm detection based on late fusion. Multimedia Tools and Applications, 81(6):8597-861 2022.
Alan Dix, Janet Finlay, Gregory Abowd, and Russell Beale. Human-computer interaction. Harlow wa, 2000.
Alexcy Dasovitskiy, Lucas Beyer, Alexander Kolesníkov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers image recognition scale. ICLR, 2021.
Jared Dunmmon, Alexander Ratner, Nishith Khandwala, Khaled Suab, Matthew Markert, Hersh Sagreiya, Roger E. Goldman, Christopher Lee-Messer, Matthew P. Lungren, Daniel L. Rubin, and Christopher Ré. Cross-modal data programming enables rapid medical machine learning. CofH, 1ss1990311100. 2019. URL http:://axxlv..rr/abb////e wwtp///g///g/ooo.//////// 11101
Christos A Frantzidis, Charalampos Bratsas, Manousos A Klados, Evdokimos Konstantinidis, Chrysa Lithari, Ana B Vivas, Christos Papadelis, Fleni Kaldoudi, Costas Pappas, and Panagiotis D Bamidis. On the classification of emotional biosignals evoked while viewing affective pictures: an integrated data-mining-basee approach for healtheare applications. IEEE Transactions on Information Technology in Biomedicine, 14(2):309 -188 2010.
Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche. Early vs late fusion in multimodal convolutional neural networks. In 2020 IEFF 23rd international conference on information fusion (FUSTON), pages 1-6. IEFF, 2020.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Xisochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. Proceedings of the 58th Annual Mesting of the Association for Computational Linguistics, pages 5553-5563, 2020.
Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md lftekhar Tanveer, Louis-Philippe Morency, and Mohammed Ehsan Hoque. Ur-funny: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 2046 2056, 2019.
Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Baptiste Alayrac, and Aida Nematzadeh. Decoupling the role of data, attention, and losses multimodal transformers. arXiv preprint (XXx:2102000520) 2021.
Ronghang Hu and Amanpreet Singh. Transformer all you need: Multimodal multitask learning with unified transformer. arXiv preprint arXiv:2102. 2021.
Andrew Jacgle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, al. Pereciver io: A general architecture for structured imputs & outputs. arXiv preprint 2x1.::1107.14795 2021a.
Audrew Jaegle, Felix Gimeno, Andrew Brock, Audrew Ziserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arxiv preprint arXiv:8103.03800 2021b.
Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicatiie interactions and where to find them. In International Conference Learning Representations, 2020. URL https:///penreviexs
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Tea Anthony Celi, and Roger G Mark. Mimic-lii, freely accessible critical care database. Scientific data, 3 (1):1-9, 2016.
Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. Xiu preprint xxxix1106 706.05137.. 2017.
Douwe Kiela, Suvral Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint ww.Xxx1999..0m 02950, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from thny images. 2009.
Yann LeCun, Léon Botton, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. 1998.
Proceedings of the IEEE, (6(11))2278-28
Michelle A Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannelte Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International Conference on Robotics and Automation (ICRA), pages 8943-8950. IEEE, 2019.
Michelle A Lee, Brent Yi, Roberto Martin-Martin, Silvio Savarese, and Jeannette Bohg. Multimodal sensor fusion with differentiable filters. IROS, 2020a.
Michelle A Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3):582 596, 2020b.
Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song. Parameter efficient multimodal transformers for vidco representation learning. In International Conference on Learning Representations, 2020c.
Luis A Leiva, Asutosh Hota, and Antti Oulasvirta. Enrico: dataset for topic modeling of mobile designs. 22nd International Conference on Humon- Coniuuterr Interaction with Mobile Devices and Services (MobileHCI Extended Abstructs), 2020.
R Gary Leonard and George Doddington. Tidigits speech corpus. Tezas Instruments, Inc, 1993.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: simple and performant baseline for vision and language. Xiv proprint orXie:1908.0088 03557, 2019.
Qing T.i, Boging Gong, Yin Cui, Dan Kondratyuk, Xianshi Du, Ming- Hsuan Vang, and Matthew Brown. Towards a unified foundation model: Jointly pre-training transformers on unpaired images and text. arxio preprint orXi::11 12.07074, 2021.
Paul Pu Liang, Terrance Liu, Anna Cai, Michal Muszyuski, Ryo Ishii, Nicholas Allen, Randy Aucrbach, David Brent, Ruslan Salakhutdinov, and Kcuis-Philippps Morency. Learning language and multimodal privacy-preservinn markers of mood from mobile data. In ACI,/IJCNLP (1), 2021a.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle Lee, Yuke Zhu, al. Multibench: Multiscale benchmarks for multimodal representation learning. In NewrIPS Dotosets and Benchmarks Track, 2021b.
Paul Pu Liang, Peter Wu, Liu Ziyin, Louis-Phliipe Morency, and Ruslan Salakhutdinov. Cross-modal generalization: Learning in low resouree modalities via meta-alignment. In Procoedings of the 29th ACM International Conference on Multimedie, pages 2689, 2021c.
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends multimodal machine learning: Principles, challenges, and open questions. arvvv preprint orXio:2209.034p0 2022.
Valerii Likhosherstov, Mostafa Dehghani, Amurag Arna,, Kraysztof Marcin Choromanski, Mario Lucic, Yi Tay, and Adrian
Weller. Polyvit: Co-training vision transformers on images, videos and audio, 2022. URL htttpp:://operreveee.eee///oo//o id-9r4_ 7GxTInS.
Bryan Lim, Sercan Ô Ank, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 2021.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: pretraining task-agnostic visiolinguistic representations for vision-and-languugg tasks. In Proccedings of the 33rd International Conference on Neural Information Processing Systems, pages 13 23, 2019.
Kevin Lu, Aditya Grover, Picter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines, arXiv preprint xix::2103.05247, 2021.
George A. Miller. Wordnet: A lexical database for english. Commaun ACM, 38(11):39 41, November 1995. ISSN 0001-0782.
Ghulam Muhammad, Fatima Alshehri, Fakhri Karray, Abdulmotaleb El Saddik, Mansour Alsulaiman, and Tiago H Falk. A comprehensive survey on multimodal medical signals fusion for smart healthcare systems. Information Fusion, 76:355-377 335, 2021.
Juan-Manuel Pérez-Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Jurie. Mfas: Multimodal fusion architectare search. In Proceedings of the IEEE Conference on computer vision and pottern recognittio, pages 6966 6975, 2019.
Rosalind W Picard. Affective compading. MIT press, 2000.
Karol Piezak. Esc: Dataset for environmental sound classification. In Proceedliggs of the 23rd ACM international conference Multimedia,, pages 1015-1018, 2015.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tlya Sutskever. Zero-shot. text-to-image generation. erXie preprint orXiv:8102. 12092, 2021.
Scott Roed, Konrad Zolua, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikow, Gabriel Barth-Maron, Mai Gimenez,, Yury Sulaky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXio preprint arXiv:2205.061 061775 2022.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint orXiv:1706. 05098, 2017.
Amanprect Singh, Ronghang Hu, Vedamuj Goswami, Guillaume Couairon, Wajeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. arXiv preprint arXiv:2112.04482 2021.
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D Manning, and Andrew Y Ng, Zero-shot learning through cross-modal transfer. arXiv preprint arXie:1011.366. 2013.
Karthik Sridharan and Sham Kakade. An information theoretic framework for multi-view learning, Conference on Learning Theory, 2008
Trevor Standley, Amir Zamir, Dawn Chen, Leouidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference Machine Learning, pages 0120-9132. PMT.R, 2020.
Weijie Su, Xizhou Zhn, Yue Cao, Bin Ld, Lewei Lai, Furu Wel, and Jifeng Dai. VI-bert: Pre-training of generic visual- lingulste representations. In International Conference on Learning Representatons, 2020. URL httpaap:://geeereieveeeeeee///ee SygXPaEYvH!
Baochen Sun, Jiashi Feng, and Kate Sacuko. Return of frustratingly casy domain adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for vidco and language representation learning, In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 7464-7473 7473, 2019.
Masahiro Suzuki and Yutaka Matsuo. A survey of multimodal deep generative models. Advanced Robotics, 36(5-)):261122) 2022.
15

---

Published in Transactions on Machine Learning Research (05/2023)
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empérical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 5100-5111, 2019.
Tadahiro Taniguchi, Shingo Murata, Masahiro Suzuki, Dimitri Ognibene, Pablo Lanillos, Emre Ugur, Lorenzo Jamone, Tomoaki Nakamura, Alejandra Ciria, Bruno Lars, et al. World models and predictive coding for cognitive and developmental robotics: Frontiers and challenges. arXiv preprint or Xiv: 230.00532,,, 2023.
Yao-Hung Hubert Tsai, Shaojie Bai; Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558-6569, 2019.
Endel Tulving and Michael Watkins. On negative transfer: Effects of testing one list on the recall another. Journal of Verbal Learning and Verbal Behavior, 13(2):181-193, 1974.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Valentin Vielzeuf, Alexis Lechervy, Stéphane Patenx, and Frédéric Jurie. Centralnet: a multilayer approach for multimodal fusion, 2018.
Welyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 12695-12705, 2020.
Zirui Wang, Zihang Dai, Baruabás Póczos, and Jaime Carbonell. Characterizing and avoiding uegative transfer. In Proccedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11293-11302, 2019.
Xuetong Wu, Jonathan H. Manton, Uwe Aickelin, and Jingge Zhu An information-theorrde analysis for transfer learning: Error bounds and applications. arXiv preprint urXie:2207.0623 2022.
Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O. Pinheiro. Adaptive cross-modal few-shot, learning. In H. Wallach, H. Larochelle, A. Beygeksimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, 2019. URL http::/paperss..etece////// nips. ctpppppp///11...1 aaappppr//31-aaattvveeeee aatps-aaapttiterrseeeeee cenpiiiveerros-mmmaall sert---fe--ss--errrree pdf
Dingkang Yang, Shual Huang, Haopeng Kuang, Yangtao Du, and Lihua Zhang. Disentangled representation learning for multimodal emotion recognition. In Proczelings of the 30th ACM International Conference on Mullimedia, pages 1642-1651, 2022.
Shaowei Yao and Xiaojun Wan. Multinodal transformer for multimodal machine translation. In Proceedings of the 58th Annual Meeting of the Association for Compitational Lánguisties, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/vv http/////220ccccco 1a..0..al-maaii acl-main.0000. URL https: www.wwwwwwwweee.eeeeeeeeeeoweeee arg/anthology/022 acl-main.4000
De Jong Yeong, Gustavo Velasco- Hernandez, John Barry, Joseph Walsh, et al. Sensor and sensor fusion technology in autonomous vehicles: A review. Sensors, 21 (6):2140, 2021.
Amir Zadeh, Minghal Chen, Soujanya Poria, Erik Cambria, and Louis Phill-p Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Fomirrical Methods in Natural Language Processing, pages 1103-1114, 2017.
Amir Zadeh, Paul Liang, and Louis-Philippe Morency. Foundations of multimodal co-learning. Information Fusion, 64: 193, 2020.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency, Multimodal language analysis in the wild: Cmu-mosel dataset and interpretable dynamic fusion graph. In ACI, 2018.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712 3722, 2018.
Haijin Zeng, Jize Xue, Hiēp Q Luong, and Wilfried Philips. Multimodal core tensor factorization and applications to low-rank tensor completion. IEEE Transactions on Multimedio, 2022.
Yu Zhang and Qiang Yang. A survey multi-task learning. IEFE Transactions Knowledge and Data Engineering, 2021.
16

---

Published in Transactions on Machine Learning Research (05/2023)
Appendix
A Measuring Heterogeneity via Modality Information Transfer
A.1 Modality and interaction heterogeneity measures
We formalize unimodal transfer the difference in performance between unimodal models trained on X before transfer to X2, versus those trained directly on X2- We represent unimodal model using X₂ with parameters 0 as ÿ = f () and define the loss of a model as Ep( (2)(2)2 0). which measures the expected error over the joint distribution p(z2. y) under suitably chosen loss function E(g.y). To mcasure transfer, we train 2 models: the first randomly initialized and trained on the target task giving loss L2.
C min ((r
(12)
and the second using initialization from model parameters 0, trained on the source task (X1;Y) before fine-training on the target task giving loss Li-
0. arg min E am (((
(13)
8
fi a (o E (f(() 0 f1).
(14)
where 0 + 0₁ denotes parameter initialization with 0j. The differences between these 2 losses,
T(Xi - (x;1>x2;31 fi-2--4--000 L2z
(15)
therefore measures the difficulty of transferring a model trained on the source task (X to target task (X2; Y). Our final heterogeneity measure ((x1:X2) aggregates the non-negative value (to account for positive transfer) of transfer difficulty across both transfer directions X1 X2 and X₂ X1:
d(X1;X2) = T(X: X2; T(X2 + X1;Y)20-
(16)
where z20 max(x, 0). We show that under certain assumptions on the modalities and tasks, our modality heterogeneity measure X2) a metric:
Non-negativity: For any X1, X21
d(X1:X2)= =T{x + x2: - X1; 0,
(17)
and when Xi = X2,
d(X1;X2) T(X, + X2; T(X2 =000000000000
(18)
Symmetry: For any X2,
d(Xi:X,) =(X, + 2 X1;))20 =T(X² + +1;Y))a0 + T(Xi - X2;X)20
(19)
(20)
=d((2:X1)
(21)
Positivity: In general, this heterogeneity measure may not satisfy positivity. In cases of positive transfer where using very different modalities pretraining data still help downstream tasks, we have that fi-2 fit (or fi--1 & Li) and as a result T(X, X2; Y) < O (or T(X; + X1; < 0) and d()1;X2) 0 even # X2, This distance metric satisfies positivity if we assume that only negative transfer exists:
when +
Assumption 1. (Negative transfer) For all i,j, we have that Limz 2 In other words, for i + j, the loss on a lask is always the lowest when directly trained model for that modality and task, rother than pretraining on a different modality. When i = j, we have that Li Lz with equality.
Under Assumption [ for all X1 X2, Li-2> L5 and LJ-1 > Ci, so X() T(X1 + X2;Y)=s + T(X2 + Y):Y)20 O, so positivity is satisfied. However, we would like to emphasize that even when positivity not satisfied due to positive transfer, this is actually a benefit which tells us exactly when we are permitted to share parameters for different input modalities, since d(X1; X2) is zero or small even though X, X2. From
17

---

Published in Transactions on Machine Learning Research (05/2023)
the real heterogencity matrices in Figure 5 do find several examples of X1 X2 where X() is zero and this implies that we can safely share parameters across different. modalities.
Relaxed triangle inequality: We will show a relaxed version of the triangle inequality (Sidharan and Kakade 2008): for all i.j.k, k there exists a positive constant c2 1 such that
d(X;;x)) S c(d(X;))) + x dxX);
(22)
To show the triangle inequality we will first. need the following assumptions:
Assumption 2. (Intermediate negative transfer) For all i,j,k, we have that Di 2 fi-k- Intermediate negative transfer extends negative transfer: adding another pretraining task can only increase the loss. Assumption 3. (Proxy lask) For all i,j, ki there exists a positive constant À >> 0 such that
C 2 -
(23)
This assumption implies that we can use the differences in performance when transferred to a prozy task k (L.e., - to estimate the differences in performance on the original source task j (t.e., fi-;- -L;) ). Certainly these 2 quantities will not be exactly the same due the different nature of modalities j and k. but we assume that the relative differences are bounded below by consland X.
Under these assumptions, we will first show that the triangle inequality holds for both transfer directions, i.e.,
Ci - C s Sc - C3 Cj- 1 Li)
(24)
and
C 175 s0 Lig fi - Li)
(25)
which would then imply
X(X
(26)
=T(X: Xgi Y): >0 T(X; X1;Y)
(27)
= - C3 Ci-ni L;
(by neg. transfer)
(28)
sci(C,) -c; + f (.) + 02 02( - -
LJ
(29)
sdft + -j L7 fip-j- - Lij fiji -()
(c= max(ci, 02))
(30)
=c(T(X, X5; Xj;Y)20 T(X; X;; Y)0 T(X; Xx; Y)20 T(Xe 823
(by neg, transfer)
(31)
=c{((X;;g)) Xx))
(32)
To show each direction, we have that
G/-k f s finj-k - fix
(by intermediate negalive transfer)
(33)
fi -fij-t fij-x Li
(34)
s 11/\(r - -2,) ) + Lg Li
(by proxy task)
(35)
sc(( Ljj + +ij - (i)
(for a = max((11121))
(36)
and the other direction is symmetric. Since C1 = 1// 2 1 we have that the final constant == max (2) 1.
In practice, the negative transfer and intermediate negative transfer assumptions may not hold. In fact, our experiments do show evidences of positive transfer and intermediate positive transfer: pretraining very different modalities and tasks can help downstream modalities and tasks, and that this improvement grows are more pretraining modalities and tasks are used (see Table 3 Nevertheless, we find that the exact triangle inequality is approxinately satisfied 87% (unimodal) and 86% (interaction) of the time from the heterogeneity matrices in Figure 5 and the relaxed version of the triangle inequality for constant c= 5.0 is satisfied 96% (unimodal) and 95% (interaction) of the time. This is also a contributing factor the fact that the heterogeneity matrices have low-rank structure that enables their approximation with only rank 3 matrices.
18

---

Published in Transactions on Machine Learning Research (05/2023)
Singular values of Mu
Singular values of Mc
50
magnitude
magnitude
30-
20-
10
10
number singular values
number of singular values
Figure 7: Distribution of singular values of the heterogeneity matrices My and Mc. The top 3 singular values are sufficient to explain 97% of the variance, so low-rank approximation = is sufficient to approximate the entire matrix. This suggests that simply lesting random set of O(M) unimodal and interaction transfer values is sufficient. to enjoy the benefits of our proposed approach.
A.2 Modality and interaction heterogeneity matrix
In Figure 7 show some experiments demonstrating the low-rank nature of the heterogeneity matrices My and Mc. The top 3 singular values are sufficient to explain 2 97% of the variance, such that even using a low-rank approximation of r 3 is sufficient to approximate the entire matrix. For example, we can test a random set of O(M) unimodal and interaction transfer values, and approximate the heterogeneity matrix M = Σ d.vn as an outer product of individual basis vectors u; and vi, where h a smaller nunber than the uctual dimension of M. This suggests that we do not need lo exhaustively measure unimodal and interaction transfer between all modality pairs to enjoy the benefits of our proposed approach.
A.3 Determining parameter groupings
We balance both total performance and parameter efficiency via agglomerative hierarchical clustering where modalities are nodes and heterogeneity measurements are edges. The number of clusters &t is treated hyperparameter dependent on the parameter budget. Clustering on the modality heterogeneity matrix My results in grouping of modalities based similarity (e.g., U1 (21,X2, X2,,22 [X3) (Xs)), and likewise for the interaction matrix Mc (c.g., C1 = (1X,, X2). (X,, X3), (X4. ([X4,Xa) Xo). (X5, X, and so on. How can we choose the number of clusters k? Note that if A is equal to the total number of modalities (or modality pairs) then it reduces to having separate models for each modality (and interaction), while & == 1 implies using a single model for all modalities and interactions. is therefore most suitably seen as a 'parameter budget' that one would like to control for efficiency. In our experiments, we explored range of k giving rise to suite of models across controlled trade-offs between performance and efficiency (see Figure 0
((X2. (x11X.x33.000 C33.C3
B HighMMT Details
At a high level, HIGHMMT includes the following components: (1) the inputs are standardized into sequence and padded, (2) the perceiver inpul processing adds modality-specific modality embeddings and positional encodings to the serialized raw input; (3) the processed input from each modality is fed into shared unimodal perceiver encoder; (4) cach pair unimodal pereeiver output (unimodal representations) is fed through a shared crossmodal transformer layer twice (the first time with one modality as query and the other as context, and the second time vice versa); (5) finally, all outputs from multimodal layers are concatenated, batch-normalized form a multimodal representation, and fed through a task-specific classification head to make a prediction. Figure is an illustration of the high-level architecture.
B.1 Perceiver Input Processing
We follow the data processing pipeline in the GitHub implementation for multimodal perceivers: https; For each modality, we must specify in
19

---

Published in Transactions on Machine Learning Research (05/2023)
Language
All Fow lerk poany
Video
X1
unimodal encoder
Shared
Z1
Muttimdda crose attention
12 22-01]
X2
Shared uniodal encoder
Z2
Muttmodal croas- stterittertioon
[2 Z3-11
m1am
task cassiter
Prediction
Audio
X3
Shared unimodal encoder
Za
Muttimodal cross attentttnntonn
[22-13 23
21 (0)
zerin posiiionll encoding
0
Attention Kay, Value Attention Query
Modalty embeddings
parameter sharing
23
0)
Features
Latent vactors
Iaadomly initialized trainable
Figure 8: General architecture of HIGHMMT: Given arbitrary modalities, (1) the inputs are standardized into a sequence and paddod, (2) modality embeddings and positional encodings are added to the scrialized raw input, (3) a single shared unimodal Perceiver encoder is applied to all modalities to learn general- representations regardless of the specific input modality, (4) each pair of unimodal representations fed through a shared multimodal cross-attention layer twice (the first time with one modality as query and the other as context, and the second time vice versa) to learn general multimodal representations regardless of the input modalities and lask, and finally (5) all outputs from cross-allention layers are concatenated, mormalze,,, and fed through a task-specific classification head to make a prediction. The unimodal encoders and multimodal layers are shared across tasks during multitask learning to enable statistical strength sharing, parameter efficiency, and quick generalization across diverse modalities and tasks.
advance the channel size (i.e., embedding size) and how many extra dimensions there are other than the channel/embeddding dimension.
The modality embedding is just a one-hot vector denoting the index of the current modality, and the size of the vector is equal to the total number of modalities involved. This embedding layer identifies common modalities across different tasks to enable sharing of information. For example, the modality embedding of the image sequence for a video classification task will be shared with that of an input (static) image for an image and text ceesstion-answeingg task.
We also specify few hyperparameters (such as num num_freq bands and max max_freq) for the Fourier transformation used in the posilional encoding. The positional encoding represents where this embedding is at through Fourier transformations (so if there is 1 extra dimension, then the positional encoding will encode the ID position of cach embodding: if there are 2 extra dimensions, then the positional encoding will encode the 2D position of each embedding). The positional encoding length can vary for each modality depending on the number of extra dimensions and the Fourier transformation hyperparameters.
The total embedding size of the processed output will be equal to dau max dpm [M]), where M is the set of all modalities involved, dm is the channel size of modality m, dpm is the positional encoding size of modality m, and |M] is the modality encoding size (i.e., the total number of involved modalities). When processing each modality, we concatenate the input channels, the positional encoding, and the modality encoding along the haaneelemmbeddie total embedding size dau- As result, all modalities will be processed to have the same embedding size dau- We also flatten all non-embedding dimensions so the processed input will always have shape fm x dall where Im is modality-specific sequence length, and dau is the common embedding dimension.
axis before adding zero-padding along this axis to match a desired
For example, during multitask learning in the large setting (4 datasets involved: UR-FUNNY, MOSEL, MIMIC, and AV-MNIST), dau = 337 (because the image modality from UR-FUNNY has a channel size of 371, ionalional encoding size of 7, and modality encoding size of 9). When processing the colorless image modality from AVMNIST (7 x 7 x 16), we have a channel size of 16, positional encoding size of 26, and modality encoding size of 9, so the processod output will be 49 x 387 where the first 16 dimensions along the last dimension represent 16 raw input dimensions, the next 336 dimensions are padded zeroes, the next 26 dimensions are positional encodings, and the final 9 dimensions are modality encodings.
20

---

Published in Transactions on Machine Learning Research (05/2023)
Note that during this entire processing step all procedures are programmatic and there are no trainable parameters involved.
B.2 Unimodal Perceiver Encoder
Now that we have standardized all modality inputs into a common representation, we follow the Perceiver architecture (Jaegle et al. 2021b) to perform modality and task-agnostic representation learning from each input modality. Starting with latent array of shape dL, dzs (array size configurable as a hyperparameterr where dLN is the number of latent vectors and dls is the latent dimension) with trainable initialization, for each layer, we first perform cross-atlention on the latent array using the processed input array (of shape fm x duut as context. Cross-attention between the latent vector and the input modality sequence learns relationships between elements in each modality, resulting in unimodal contextualized representations. The resulting latent array then goes through a latent transformer (with self-attention and feed-forward layers). We repeat this architecture for each layer within the encoder. The main advantage of this Perceiver encoder is that it can encode the input into a common dLN x dLs latent array regardless of the input shape fmm x dail, and the total runtime is linear with respect to the size of tm which scales to high-modality scenarios. Note that only one copy of a unimodal Transformer (Perceiver) block is used to encode all modalities simultaneously, which enables statistical strength sharing and general-purpose representation learning regardless of the specific input modality.
B.3 Crossmodal Transformer layer
To learn modality and lask-agnostic multimodal representations, we use multiple layers of a general-purpose Crossmodal Transformer block (Tsai et al.] 2019 Lu et al. 2019). Given 2 unimodal representations Z₁ and Ze of common shape dn x dr. learned from unimodal Perceiver encoders, a Crossmodal Transformer (CT) block uses crossmodal self-attention by setting the input layer query Q = Z1 and keys and values V Zz to learn attention from modality 1 to modality and separate block to capture the attention in the opposite direction. This step enables one modality's sequence elements to discover correspondences in another. A Crossmodal Transformer block using Zi to allend to Z2 (and vice-versa) results in a multimodal representation Zmm = [Z1-2, Z2-1]= [CT(Z1, 22), C2 21)]. For each layer, we first perform cross-attention followed by self-attention and food-forward functions. In the end, we only take the last dys-dimensional vector out of the dy,N' x dds final latent array as the output of this module. For lasks with more than 2 modalities, a Crossmodal Transformer block is applied for each pair of modalities before concatenating all multimodal representations. Again, only one copy of a multimodal layer is used on all tasks to learn general representations regardless of the input modalities and task.
B.4 Task-specific classifiers
Since each task may have different number of modalities and output classes, we create a separate classification head for each task. For each classification head, it concatenates all outputs of the Crossmodal Transformer layer (so 2-modality tasks have concatenated size of 2d.s. 3-modality tasks have concatenated size of 6dgs, etc), performs batch-normallzzation, and feeds the normalized multimodal representation 7mm into a linear layer that maps to the logits for this task. This classification layer composcs individual correspondences learned within and across modalities to form final prediction.
B.5 Homogeneous multitask pre-training
Since each task has different number of training batches, not all tasks will be involved in each training step. We arrange the tasks to be included in each training step such that more tasks will be trained simultancously towards the end of an epoch. For example, if task A has 300 training batches, task B has 200 training batches, and task C' has 100 training batches, then for the first 100 training steps in an epoch, only task A will be usod; then for the next 100 steps both A and B will be used; and for the last 100 steps, all three tasks will be used. This approach tends to work better than including all tasks in all steps via uniform batch sampling because the task with fewer training batches tends to overfit in the latter approach.
21

---

Published in Transactions on Machine Learning Research (05/2023)
Within each training step, we compute the losses of the batch from each task used and compute the gradient using a weighted sum of the losses. The weights are part of the hyperparameters that we can tune to ensure balanced training. Then we update the model using the computed gradients.
We compute validation performance after each epoch for each task, and aggregate validation performances across all tasks (this necessary because different tasks are measured differently, sometimes bigger is better, sometimes smaller is better). When all tasks are accuracy-base (such as the large setting), we just weigh them equally. Then we report test performance on the checkpoint with the highest aggregated validation performance.
The result from homogeneous multitask pre-training is a set of modality embeddings, common unimodal and crossmodal parameters U and C", and individual task classifiers.
B.6 Heterogeneity-awan fine-tuning
We account for heterogeneity by grouping unimodal parameters based on modalities that we know to be similar from $2.1 (e.g., setting U₁ = (U1,U2) U2 = {Ua}, U3 = Us, Uo}), and likewise for the crossmodal parameters (c.g., C1 CIa,,Cia C1.). {C23, C1s), C3 (C24,...)). These groups of parameters are first initialized with the homogeneous model U and C* before separale fine-tuning, which results in final parameters U {UJ, ... and C C..... The modality embeddings and task classifiers are jointly fine-tuned well.
B.7 Transfer learning details
If we are trying to transfer from tasks B, C) to D, initially start with randomly initialized HIGHMMT model that defines modality embeddings for all modalities in (A, B,C, as well as a classification head for each. Then, we pretrain the model using multitask learning on (A, B,C) using the same procedure as before. After saving a good checkpoint as measured by aggregated validation performance on pretraining tasks (A, B, C); we finetune the trained model on target task D. The modality and tasks in (A, B, C) present during multitask pretraining can be very different from those encountered in D during fine-tuning.
B.8 Few-shot multitask learning details
We also investigated few-shot learning using limited labeled data a target task. When we perform few-shot learning on task D with the help of tasks {A, B,C), we jointly train (4, B, C, D} together in the multitask manner as before, but since we don't care about the performance of our model on auxiliary tasks {A, B, C), we assign a higher weight to the losses on task D and keep track of the best validation performance on D when selecting checkpoints.
C Experimental Setup
In this section, we provide additional details on the experimental setup to analyze the multitask, transfer, and generalization capabilitics of HIGHMMT.
C.1
Setup
We use large collection of multimodal datascts provided in the standardized and public MultiBench benchmark (Liang et al.) 20211). This benchmark spans 15 real-world datasets, 10 modalities, 20 prediction tasks, and 6 research areas. Each of these datasets requires model to learn basic representations of features in each modality and aggregate complementary information across multiple modalities to make a prediction.
A Affective computing involves understanding our natural display of multimodal signals spanning language (spoken words), visual (facial expressions, gestures), and acoustic (prosody, speech tone) in order to predict human affective states (emotions, sentiment, and personalitics) (Picard (200)). We test on 2 datascts involving fusing language, video, and audio time-series data to predict sentiment and emotions (MOSEI (Zadeh et al, 2018)) as well humor
MammorUURFFOOR (UR-FUNNY (Haaan et al. 2019
22

---

Published in Transactions on Machine Learning Research (05/2023)
Healthcare: Medical decision-making often involves integrating multiple sensory readings from instruments such as lab tests, imaging reports, and patient-doctor conversations (Amisha et al. 2019 We experiment with the large-scale MIMIC dataset (Johnson et al. 2016) which records ICU patient data including time-series data measured every hour and other demographic variables in the form of tabular numerical data. These are used to predict the disease ICD-9 code and mortality rale.
Modern robot systems are equipped with multiple sensors in order to capture complementary signals useful for holistic decision-making We test on the large-scale MujoCo PUSH (Lee et al. 2020a) and V&T Vision&&Touch) (Lee et al. 2020b) datasets which record the manipulation of simulated and real robotic arms equipped with visual (RGB and depth), force, and proprioception sensors. In MuJoCo Push, the goal is to predict the pose of the object being pushed by the robot end-effector. In Vision & Touch, the goal is to predict action-conditional learning objectives that capture forward dynamics of the different modalities (contact prediction and robot end-effector pose). Both of these datasots include visual, sensor, and contact data collected from real robotics and published at top robotics conferences. Multisensor fusion in robotics is especially important due to the large number of sensors, requirements for robust learning from noisy sensors, and limited data compared to tasks involving images, text, and audio. From our experiments, our multimodal multitask approach does indeed perform well on these initial robotics tasks, and we do hope to deploy some of these models on real-world robotics as future work.
Robotics:
Human Computer Interaction (HCI) studies the design of computer technology and interactive interfaces
between humans and computers (Dix et al. 2000). We use the ENRICO dataset (Deka et al. 2017 Leiva et al.) 2020g of Android app screens (consisting of an image as well as a set of apps and their locations) categorized by their design motifs and collected for data-driven design applications such as design search, user interface (UI) layout generation, UI code generation, and user interaction modeling.
Multimedia: A sigmificant body of research in multimodal learning has been fueled by the large availability of multimedia data (language, image, video, and audio) on the internet. We experiment on 2 largo-scale multimedia datasets with varying sizes and levels of difficulty: (1) AV-MNIST (Vielzeuf et al. 2018 is assembled from images of handwritten digits (TeCun et al.] 1998) and audio samples of spoken digits Leonard and Doddington 1993), and (2) CIFAR-ESS (Liang et al.) 2021c) is an image-audio retrieval dataset. To construct CIFAR- ESC, we follow Liang et al. (2021ch and combine 100 classes from CIFAR-100 and 10 classes from CIFAR-10 (rizhevsky el al. 2009 Lo form 110 image classes, as well as 50 audio classes from ESC-50 (Piczak 2015). To bridge these two modalities with partially related label spaces, we define 17 shared classes across the 2 datasets for weak concept alignment. These clusters are obtained by mapping similar classes between the datasets using similarities from WordNet (Miller 1995) and text cooccurrence.
Multitask setup: We trainod multitask models across combinations of the aforementioned datasets. Each multitask setup is designed to include tasks with different modality inputs and prediction objectives.
1. Small: PUSH, V&T: 2 tasks in the same research area (robotics) but with different modality inputs: (image, force, proprioception, control} and (image, force, proprioception, depth} respectively. Furthermore, each robot's 's sensor readings come from different robot-dependent sensors.
2. Medium:
ENRICO, PUSH, AV-MNIST across 3 domains (multimedia, HCI, and robotics) with different modalities: scmaag,sct),, (image, propriccpooi,,,,,,nnnn control), and (image, audio]. audio..
UR-FUNNY, MOSEI, MIMTC, and AV-MNIST, across 3 domains (affective computing, healthcare, and multimedia), again with different modalities: (text, video, audio) for the first 2 tasks with different format of preprocessed embeddings of video and audio, (timo-scrics, table), and (image, audio).
3. Large:
We summarize these experimental settings in Table I Overall, the total size of datasets involved in our experiments exceeds 370, 000 and covers diverse modalities such as time-series, various robotics sensors, sets, and tables, as well as multiple research areas and prediction tasks from affective computing, healthcare, multimedia, robotics, and HCI.
23

---

Published in Transactions on Machine Learning Research (05/2023)
C.2 Hyperparameters, parameter groupings, and training details
We list lypeepaaaaeeters used throughout our models in Table Table g and Table 10 for small, medium, and large multitask settings respectively. Code is also included in the supplementary material for reproducibility,
We train a suite of HIGHMMT models obtained by tuning k, the number of parameter groups (i.c., the number of clusters when clustering heterogeneity matrices). & can be seen as a hyper-parameter depending on the computational budget, with smaller & implying more parameter sharing on lower budgets and vice-versa. We test k in the range {2, 4, 6, 7, 7,9), with |U] | - 1, |U| 3, |C| 1, |0] 3, |C] 3, |U] = 3,|C| 4, and [U] [ 4 5 respectively where [U],[C] denote the number of unimodal and crossmodal parameter groups.
More specifically, we choose which modalities to share parameters by selecting the high value entries in the transfer matrix. When ] 3, the first set of unimodal parameters are shared betweeen AV-MNIST image and AV-MNIST audio; the second set of parameter is shared between MIMIC table, MOSEI-video and MOSEI-audio, the third set of parameters are shared across the rest of the modalities. When [T] = 4, the first set of unimodal parameters are shared betweeen AV-MNIST image and AV-MNIST audio; the second set of parameter is shared between MIMIC table and MOSEI-video; the third set of parameters are shared between MIMIC time series and UR-FUNNY video, while the last set of parameters are shared between UR-FUNNY text and MOSEI-text.
When [C] 3, we use two distinct sets of crossmodal parameters for AV-MNIST (image/audio) and MOSEI (video/ audio) respectively, while the same crossmodal parameters are shared among the rest the tasks. When |C] =4, we share one set between MOSEI (vidco/audio) and UR-FUNNY (audio/text)); one set between UR-FUNNY (video/text) and UR-FUNNY (video/audio): another distinct set for AV-MNIST (image/audio)) and finally set of parameters for the rest. Finally, |C| = is similar to |C] 4 except that train different. set of crossmodal parameters for MOSEI (audio/text), and another new set of parameters shared between UR-FUNNY (video/text) and UR-FUNNY (audio-text)..
24

---

Published in Transactions on Machine Learning Research (05/2023)
Table 8: Table of hyperparameters for multitask prediction the small setting involving PUSH, V&T: 2 tasks in the same research area (robolics) but with different modality inputs: {image, force, proprioception, control) and (image, force, proprioception, depth} respectively, and readings come from different robot-dependent sensors.
Values
Part of Model
Hyperparameter
PUSH
V&T
Depth
Num Latents
20
Latent Dim
64
Unimodal Perceiver
Cross Attention Ieaads
Encoder
Latent Self-Attention Ileads
Cross Head Dim
64
Latent Head Dim
Num Latent Blocks Per Layer
Depth
Num Latents
20
Latent Dim
Multimodal
Cross Attention Heads
Cross-Attention
Latent Self-Attentin Heads
Layer
Cross Head Dim
64
Latent Head Dim
64
Num Latent Blocks Per Layer
Classification Heads
Input/output dimensions
756/32
1280/1
BRatchNorm-LLinnse
Optimizer
Adam
Learning rate
0.0005
Weight decay
Training loss weights
100.0
1.0
Batchsize
18
64
Evaluation weights
100.0
1.0
Image: 128x128x3
Gripper Pos: 16x3
Force: 6x32
Original MultiBench
Gripper Sensors: 16x7
Proprio:
Input Dimensions
Image: 16x32x32
Depth: 128x128
Control: 16x7
Action: 4
Image:
Gripper Pos: 3
Force: 32
Perceiver Input
Gripper Sensors:
Proprio: 8
Channel Size
Image:
Depth:
Control:
Training
Action: 4
Image: 2
Gripper Pos:
Force:
Perceiver Input
Gripper Sensors: 1
Proprio: 1
Extra Axis
Image:
Depth:
Control:
Action: 1
Image:
Gripper Pos:
Force:
Perceiver Input Num freq bands
Gripper Sensors: 6
Proprio: 6
Image:
Depth:
Control:
Action: 6
Image:
Gripper Pos: 1
Force:
Perceiver Input Max Max_freq
Gripper Sensors:
Proprio:
Image:
Depth:
Control:
Action: 1
Shared Modality Encoding
N/A
25

---

Published in Transactions on Machine Learning Research (05/2023)
Table 9: Table of lyperparameters for multitask prediction on the medium setting involving AV-MNIST, ENRICO and PUSH: tasks across 3 domains (multimedia, HCI, and affective computing), again with vastly different modality scts: (image, audio), (imago, sct), and (image, force, proprioception, control) for each task.
Values
Part Model
Hyperparameter
AV-MNIST
ENRICO
PUSH
Depth
Num Latents
12
Latent Dim
64
Unimodal Perceiver
Cross Attention Heads
1
Encoder
Latent Self- Attention Heads
8
Cross Head Dim
64
Latent Head Dim
64
Num Latent Blocks Per Layer
Depth
Num Latents
12
Latent
64
Multimodal
Cross Attention Heads
1
Cross- Attentin
Latent Self- Attention Heads
8
Layer
Cross Head Dim
64
Latent Head Dim
64
Num Latent Blocks Per Layer
1
Classification Heads
Input/output dimensions
128/10
128/20
768/2
[BatchNorm+LLeeg
Optimizer
Adam
Learning rate
0.001
Weight decay
0.0
Training loss weights
0.8
1.0
1.1
Batchsize
32
32
32
Evaluation weights
Gripper Pos: 16x3
Colorless Image: 28x28
Original MultiBench
Image: 256x 128x3
Gripper Sensors: 16x7
Audio Spectogram:
Input Dimensions
Set: 256x
Image: 16x32x32
112x112
Control: 16x7
Colorless Image:
Gripper Pos:
Training
Perceiver Input
(cut into 4xd squares)
Image: 3
Gripper Sensors: 7
Channel Size
Audio Spectogram: 256
Image:
(cut into 16x16 squares)
Control:
Gripper Pos:
Perceiver Input Extra Axis
Colortess Image: 2
Image: 2
Gripper Sensors: 1
Audio Spectogram:
Set: 2
Image: 3
Control:
Gripper Pos: 6
Perceiver Input Num freq bands
Colorless Image: 6
Image:
Gripper Sensors: 6
Audio Spectogram:
Set: 6
Image: 6
Control:
Gripper Pos: 1
Perceiver Input Max freq
Colorless Image:
Image:
Gripper Sensors:
Audio Spectogram:
Set: 1
Image: 1
Control:
Shared Modality Encoding
N/A
26

---

Published in Transactions on Machine Learning Research (05/2023)
Table 10: Table of hyperparameters for multitask prediction on the large setting involving MIMIC, AV- MNIST, MOSEI and UR-FUNNY: 4 tasks across 3 domains (healthcare, multimedia, and affective comput- ing), again with vastly different modality sets: (time-series, table), (image, audio), and {text, video, video, for the final 2 tasks with different format of preprocessed embeddings of video and audio.
Values
Part of Model
Hypcrparamoter
MIMIC
AV-MNIST
MOSEI
UR-FUNNY
Depth
Num Latents
20
Latent Dim
64
Unimodal Perceiver
Cross Attention Heads
Encoder
Latent Self-Attention Heads
Cross Head Dim
64
Latent Head Dim
64
Num Latent Blocks Per Layer
Depth
Num Latents
20
Latent Dim
Multimodal
Cross Attention Heads
4
Cross-Attention
Latent Self-Attention Heads
Layer
Cross Head Dim
64
Latent Head Dim
64
Num Latent Blocks Per Layer
Classification Heads
Input/output dimensions
128/2
128/10 10
384/2
384/2
(BatchNorm- +Linear)
Optimizer
Adam
Learning rate
0.0008
Weight decay
0.001
Training loss weights
1.2
0.9
1.1
1.5
Batchsize
20
40
32
32
Evaluation weights
1
Colorless Image: 28x28
Image: 50x35
Image: 20x371
Original MultiBench
Static: 5
Audio Spectogram: 112x112
Audio: 50x74
Audio: 20x81
Input Dimensions
Timeseries: 24x12
Text: 50x300
Text: 50x300
Colorless Image: 16
Image: 35
Image: 371
Perceiver Input
Statie: 1
(cut into 4x1 squares)
Training
Audio: 74
Audio: 81
Channel Size
Timeseries: 1
Audio Spectogram: 256
Text: 300
Text: 300
(cut into 16x16 squares)
Image:
Image:
Perceiver Input
Static: 1
Colorless Image: 2
Audio:
Audio:
Extra Axis
Timeseries:
Audio Spectogram: 2
Text: 1
Text:
Image: 3
Image: 3
Perceiver Input
Statie: 6
Colorless Image:
Audio:
Audio:
Num Num_freq hands
Timeseries: 6
Audio Spectogram: 6
Text: 3
Text: 3
Image:
Image:
Perceiver Input Max freq
Static: 1
Colorless Image:
Audio:
Audio:
Timeseries: 1
Audio Spectogram: 1
Text:
Text:
Shared Modality Encoding
The text modality from MOSEI and UR-FUNNY are shared.
27

---

Published in Transactions on Machine Learning Research (05/2023)
Table 11: We train multitask HIGHMMT on 1/2/3 datasets and find that it generalizes to new modalities and tasks on the 4th dataset, with improved performance over single-task training on the 4th dataset. 0 source lasks implies transferring randomly initialized parameters, which is equivalent to single-task training on the target task. Cross-modal transfer improves with the number of pretraining tasks and works best on the smallest target tasks (UR-FUNNY).
Target task UR-FUNNY
Source tasks
(no transfer)
63.1 0.5
MOSEI
63.5 0.5 64.0 0.7 64.7 1 0.4
MOSEI AV-MNIST
MOSEI AV-MNIST MIMIC
Target task MOSEI
Source tasks
(no transter)
79.0 79.2 1.3 79.3 + 0.5
AV-MNIST
AV-MNIST + MIMIC
AV-MNIST + MIMIC + UR-FUNNY
79.6 + 0.6
Target task MIMIC
Source tasks
0 (no transfer)
67.7 0.6
MOSEI
67.9 + 0.5 68.0 + 0.8 68.4 0.6
MOSEI + AV-MNIST
MOSEI AV-NNIST UR-FUNNY
Target task AV-MNIST
Source tasks
transfer)
70.3
MOSEI
70.5 0.4
MOSEI + MIMIC
70.5 4 0.4
MOSEI + MIMIC + UR-FUNNY
70.6 0.4
D
Additional Results
In this section, we detail additional experimental results that support the multitask, transfer, and generalization capabilities of HIGHMMT.
D.1 Generalization to new modalities and tasks
HIGHMMT also offers opportunities to study whether we can transfer knowledge between completely different lasks and modalities. On the large setting, we first pretrained a model on 0/1/2/3 of the four tasks before fine-tuning on the fourth task only. We show these full results in Table IT On all four target tasks, proposed multitask pretraining and finc-tuning paradigm improves performance over single target-task training. Therefore, weights learned from other multimodal tasks indeed generalize well to new modalities and tasks. We further analyze this transfer learning phenomenon by studying the following research questions:
Effect of pretraining datasets. When we vary the number of pretraining datasets, we observe a consistent improvement on fine-tuned target task performance across all datasets. This effect is particularly pronounced on the UR-FUNNY target task, which shows the biggest improvement using pretrained parameters from to multitask datasets. This implies that HIGHMMT learns more generalizable multinodal features more tasks are involved in multitask training.
Effect of target dataset size. improvement: the smallest dataset, UR-FUNNY, benefited the most (+2.4%) from transfer learning, This implies that this multimodal pretraining -iw.iinnngnn--tuunnggg paradigm is useful for improving performance for low-resource target modalities and tasks.
We observed an inverse correlation between target task size and performance
Effect of transfer modalities. We compare transfer learning performance across different levels of partial observability. While would expect transfer to the MTMIC dataset. to be the hardest due to its modality
28

---

Published in Transactions on Machine Learning Research (05/2023)
63.6
UR-FUNNY
B0
MOSEI
69
MIMIC
71
AV-NNIST
63.2
63.0
63.4
79
test accuracy
68
67
test accuracy
70
69
62.8
78
66
68
8 62.6
77
65
2 62.2
02.4
single-task
single-task
64
single-task
67
single-task
62.0
multitask
76
multitask
63
multitask
66
multitask
61.90 20 30 40 60 70 80 90100 % labeled data
740 20 30 40 50 60 70 B0 90100 % labeled data
10
40
80 90100
610
20 30 40 50 60 70 80 90100 % labeled data
% labeled data
Figure 9: Few-shot results on new modalities and tasks. Multimodal multitask training using HIGHMMT learns more generalizable representations which improves performance across all ranges of data. The x-axis shows the percentage of labeled data used during training.
Table 12: tasks.
Multitask HIGHMMT also enables training a single model for both multimodal fusion and retrieval
Model
AV-MNIST
CIFAR-ESC 1 Params (M) $
HIGHMMT
70.4
58.8
1.04
HIGHMMT multitask
70.4
60.5
0.52
set (time-series, table} being completely disjoint from the remaining 3 datasets, we still observe a +0.8% gain as compared to single-task training. Therefore, HIGHMMT can generalize to new modalities and tasks. Unsurprisingly, for datasets with more overlap in modality sets (e.g. UR-FUNNY with complete overlap in [text, video, audio} as compared to the other datasets used for pretraining, we find larger improvements using transfer learning over single-task models (+2.4%).
Comparisons to unimodal transfer. Recent work has explored the possibility of transferring Transformer representations trained in one modality to another Lu et al. (202I); Kiela et al. (2019). Our transfer experiments also corroborate these findings in a multimodal setting with promising results on new modalities and tasks, especially involving real-world, smaller, and noisier datasets such as those involving human videos (MOSET and UR-FUNNY), medical data (MTMIC), or real and simulated robots (PUSH and V&T).
D.2 Few-shot learning
HIGHMMT offers opportunities for statistical strength sharing across tasks. We test this hypothesis in the few-shot learning scenario, by evaluating whether multitask information sharing can improve performance on low-rcsource target tasks. We compare a single-task HIGHMMT trained only on a percentage of labeled training data in the target task with multitask HIGHMMT trained on the same percentage p (during multitask training we prioritize performance of the target task over others). By varying p ε [0.1, 1.0], we plot the performance under few-shot settings in Figure g We find that multitask training is consistently better across all ranges of data, which supports the fact that more generalizable representations across modalities and tasks are learned in HIGHMMT. The main takeaway is that if it is too difficult to collect data in a target domain, collecting data from different domain and using a shared multimodal model is an alternative approach for improving performance.
D.3 Multitask fusion and retrieval
To assess task generalization, we train multitask models over fusion in AV-MNIST and retrieval in CIFAR- ESC. While fusion emphasizes information integration from complementary data sources, retrieval focuses aligning corresponding elements expressed through different views of the data (Baltrušaitis et al. 2018 Table 12 shows the full results of this experiment: even across vastly different multimodal prediction tasks, we find that multitask training (60.5% retrieval accuracy) improves upon single-task training (58.8% accuracy), while performance on the AV-MNIST fusion tasks is similar for both single-task and multitask learning. Not only have the unimodal encoders simultancously processod different modalities, the multimodal attention layer has also learned to capture correspondences useful for both fusion and retrieval, while halving the total number of parameters required as compared to task-specific modeling.
29

---

Published in Transactions on Machine Learning Research (05/2023)
UR-FUNNY text
UR-FUNNY audio
UR-FUNNY video
032
091
420
525
0.0
0.2
12
D.22
030
MOSEI text
MOSEI audio
MOSEI video
A.1
D.12
400
301
502
D04
Figure 10: Visualizations of attention patterns learned by the unimodal encoders across different modalities in the UR-FUNNY and MOSEI datasets, which consist of text, audio, and video modalities aligned in time. We find that there are some common attention patterns across tasks, which implics that the shared unimodal encoders have learned common information across tasks.
D.4 Understanding HighMMT
In this subsection, we analyze why this general model achieves strong results in multitask, transfer, and few-shot settings. Based on prior work in multitask learning (Caruana 1997 Ruder 2017 [Zhang and Yang 2021), we set up two possible hypotheses: (1) improved generalization and (2) improved regularization. We supplement the results in the main paper with additional visualizations and comparisons in this subsection.
D.4.1 Hypothesis 1: Improved generalization
Investigating parameter sharing. In which components of the HIGUMMT model is parameter sharing important?
We further study the importance of parameter sharing in HIGHMMT. From the ablation studies in Table using separate parameters for either unimodal multimodal layers results in worse performance. The full model with completely separate unimodal and multimodal layers is reminiscent of typical single-task multimodal transformers (Tsai et al. 2019 Lu et al. 2019 Hendricks et al. (021)) trained separately for each task. We show that HIGHMMT maintains competitive performance (with slightly better performance on several datasets) due statistical strength sharing, while also reducing parameters by 6x due to sharing of unimodal encoders and multimodal layers across tasks.
Furthermore, surprisingly find that removing the modality-specine embedding layer results in only slightly worse performance (70.3 to 69.6 average score). This implies that the shared unimodal encoder has learned generalizable fcature extractors that can encode heterogencous modalities even without modality identifier.
Visualization of attention patterns. How do the shared unimodal encoders attend to modality-specific tokens?
Given that parameter sharing seems be useful for performance and efficiency, we aim to better visualize the nature of information sharing in the attention layers of unimodal encoders. We perform inference
30

---

Published in Transactions on Machine Learning Research (05/2023)
FUNNY
MOSFI
singlis
74-
singfe
04-
70
62-
se-
Accurac)
70-
Accuroi
Accuracy &
1.. 16.
66-
gfterlane Dart sel gleitaae vald tasik tram muliibask valid
R.
single Look yolid
sarghelash trans
60-
singletask volid
srggllank trats
mubliskk tram
62-
mal tack train
mublitkk voli
muEilook volld
35
10
20
40
50
6
10 20
40 Training epochs
30
50
m
10
20
30
60
50
6
10
40 Training epochs
20
30
50
Training epochs
Training epochs
Figure 11: Multitask models converge as fast but overfit less (a smaller gap between train and valid accuracies) vs single-task models, which implics that multitask training helps to regularize the joint parameters and reduces overfitting on the target task.
trained multitask HIGHMMT model on the test data of the large multitask setting, and average the attention patterns across test datapoints for each datasct. Following Lu et al. (2021), the average attention pattern provides information on general inductive biases captured by the unimodal encoders and enables us to make holistic conclusions rather than comparing attention maps on individual datapoints.
From Figure 10 we actually find that a common attention pattern emerges across modalities and tasks. First. looking across modalities in the same dataset, we find that the model captures common temporal patterns at the same time steps, which makes sense since the 3 modalities are time-aligned in a vidco. The attention patterns are quite similar which implies that the same attention strategy can often work well across different modalities and tasks. This could be an explanation of why our model is able to perform multiple tasks simultaneously using shared parameters in attention layers.
It is also interesting to see how the model automatically learns to "divide up work" amongst its 20 latent tokens (numbered (-19): the latent tokens 10 10-17 13 typically all focus on the region about two-thirds after the start of the input sequence, while latent tokens 1 and 3 always focuses on the region about one-third from the start. Certain tokens (3 and 18) seem to learn oscillating attention patterns, and certain pairs of tokens learn complementary attention patterns (e-g., 4, 5, and 6 atlend one after the other). There are also some latent tokens that more evenly attend to the whole input sequence, such as latent tokens 9 and 17, which can be seen as "summary" tokens. This shows that the perceiver- encoder is able to divide up its limited latent space well to capture important information both in specific time-steps and contextual information across all the time-steps, thus creating a holistic representation of the input using much smaller set of latent variables.
D.4.2 Hypothesis 2: Improved regularization
In parallel to improved generalization, another line of research has focused on the regularization effects of multitask learning. Baxter (1997) showed that multitask parameter sharing reduces the risk of overfitting on the original task by forcing the model to learn across multiple tasks. We study the following regularization effects:
Training dynamics. In Figure 11 we traced the train and valid accuracies across 60 training epochs (with multitask training in the large setting). The training process of HIGIMMT converges at about the same rate between single-task and multitask learning, but the multitask model overfits less (a smaller gap between training and valid accuracies). This implies that multitask training helps to regularize the joint parameters and alleviates their overfitting on the target task.
Task weights. We found that optimizing simple weighted sum of loss functions over all tasks was sufficient to obtain strong multitask performance. Instead of assigning uniform weights to each task, sometimes we found it helpful to set the weight higher for more challenging datasets during HIGHMMT multitask training. We show some examples of this phenomenon in Figure 12] where multitask performance can sometimes be sensitive to weights especially when prediction objectives are of different scales (i.e., MSE vs accuracy). This supports the regularization argument where carefully tuned weighted auxiliary objectives encouraging the model to also fit other auxiliary tasks can help improve performance on a target task. However, doing so would not achieve the best performance on auxiliary tasks.
31

---

Published in Transactions on Machine Learning Research (05/2023)
MujoCo Push
multhask single task
90.2
multitesk single task
test acrurary
96.1
5ast los
0.32
96.0
0.30
95.9
10
15
25
30 weight
45
50
10
20
30
40
60 weight
70
80
100
Figure 12: Multitask performance sometimes be sensitive to task weights especially when prediction objectives are of different scales (i.e., MSE for PUSH vs accuracy for V&T), in manner similar to how carefully-tuned regularization terms help in training models.
D.5 Summary of main take-away messages
In conchusion, we designed a general multimodal multitask model for high-modality (a large set of diverse modalities) and partially-obserrable (each task only defined on a small subset of modalities) scenarios. Our approach relies on training for multitask and transfer learning: multitask learning with shared unimodal and multimodal layers enables stable parameter counts (addressing scalability) and cross- cross-modal modal transfer learning enables information sharing across modalities and tasks (addressing partial observability). Through extensive set of experiments and analysis, we summarize our main take-away messages as follows:
1. Standardized multitask modeling. We train a single multitask HIGHMMT model for numerous high-modality and partially-observable multimodal tasks (across 10 modalities, 15 prediction tasks, and research arcas), achieving strong performance while roducing total paramoter counts. We believe that standardized modeling leads to a smaller set of architectural decisions, enables transfer to understudied modalities and tasks, and present a unified platform for subsequent theoretical and empirical analysis.
2. Cross-moda transfer to new modalities and tasks. Multitask HIGHMMT enables cross-modal information transfer by pretraining on source multimodal tasks before transferring to completely new larget modalities and lasks. Involving more tasks during pretraining improves performance, and gains are more apparent when fine-tuning on low-resource target tasks. This finding can supplement current pretrain-finctune paradigms typically performed on the same modality (c.g., text-only or imago-only), and encourage research in more general multimodal pretraining over high-modality settings before fine-tuning only partial subset of all observed modalities.
3. Tradeoff between performance and efficiency. Multitask HIGHMMT improves the tradeoff between performance and efficiency over task-specific state-of-the-are models especially in low-resource scenarios (less training data and partially-observabee modalities). Coupled with the relatively fewer architectural decisions and gencralization to understudied modalitics and tasks, we believe that multitask HIGHMMT and similar architectures should be a starting point for future research.
4. Few-shot multitask learning. Multitask information sharing can improve performance on low-resource target tasks with limited labeled training data. Therefore, if it is too difficult to collect data in target domain, collecting data from different domain and using shared multinodal model is an alternative approach for improving performance.
5. Information sharing. Finally, our analysis reveals surprising insights regarding the nature of information sharing in multimodal and multitask models, which may be of independent interest. Specifically, there are both generalization and regularization effects at play:
On the generalization side, information sharing is present across modalities and tasks, but at different levels across shared unimodal and multimodal layers. Information sharing enables strong multitask performance even under partial-observabiitty and generalization to new modalities and tasks.
While using modality embeddings achieves the best performance, there only minor drop when removing them, which implies that shared unimodal encoders can learn generalizable feature extractors even without modality identifier.
On the regularization side, well-tuned regularization weights yield training dynamics that display less overfitting on target tasks as compared to single-task learning.
32


Reviews:
[{"review_id": "6LwsFNDRQz", "reviewer": "Reviewer_mTD3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper provides a new method that improves the fusion of multimodal information based on solid theoretical background.\n+ The proposed method is evaluated using a multimodal dataset and showed significant improvement.  \n\n- The performance is only evaluated on classification tasks on a static dataset.\n- The applicability to the real data in robotics has not been explored.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel multimodal representation learning method that quantifies multimodal and interaction heterogeneity using ¥Nu-usable information. The experiment on a multimodal dataset shows that the proposed HighMMT method can efficiently fuse multimodal data.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, the paper is well-structured and well-written, and the study is well-motivated. However, there are a few minor issues that could be addressed.\n\n\n1. \"High modality\nI am not sure that \"high-modality\" is an appropriate term to describe a situation where a robot has a large number of sensor modalities. I have never seen the term before. To be honest, the term gives me a very different understanding of the existence of a large number of modalities. If it is not a keyword accepted by the community, it will be better to rephrase the term.\n\n2. Realistic robotics scenario\nThe author often mentions the scenario of \"robotics\".\nIn robotics, the target task is not only label estimation, but also control and so on. Also, important heterogeneity is the temporal gap of data, data sparsity, unpredictable missing data, and so on. A robot needs to update its knowledge by interacting with the environment. \n\nI think the paper is also an important contribution to the robotics community. However, there is still a big gap between multimodal information fusion in real robots and the task tested in the paper. Therefore, if the authors provide some more concrete ideas to bridge the gap between machine learning and the robotics community, it will attract more robotics researchers to read and cite this paper.\n\nAlso, I think this research is very relevant to world models and predictive coding in robotics, which emphasizes sensorimotor interaction with the robotic world. If possible, please discuss the applicability of the proposed method to world models. \n\nWorld models and predictive coding for cognitive and developmental robotics: Frontiers and Challenges\nhttps://arxiv.org/abs/2301.05832\n\n3. Systematic review of Multimodal representation learning\n \nAs the authors mention, there are many ways to integrate multimodal information. It might be nice for readers to be given a systematic overview of multimodal fusion or explicit references to review papers. The following might be a good candidate for this.\n\n\nSuzuki, Masahiro and Yutaka Matsuo. \"A review of multimodal deep generative models\". Advanced Robotics 36.5-6 (2022): 261-278.\nhttps://www.tandfonline.com/doi/abs/10.1080/01691864.2022.2035253", "broader_impact_concerns": "Not applicable.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UPrG9isF5s", "reviewer": "Reviewer_Yccr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- **Method is new**\n- **Interest to the community**: building efficient multi-modal algorithms is of interest to the community. This paper shows one way of doing it by using learning what modalities to parameter tie.\n- **Code is available**\n- **Empirical claims are generally supported** by evidence. There are some exceptions that I highlight in \"requested changes\"\n\n# Weaknesses\n- **Lack of clarity and unnecessarily complicated discussion of the proposed metrics** I generally really like the concept of V-information but for this paper, it has two main issues: \n    1. **It is an overly complicated repackaging of a much more common term** Indeed, given that you assume that the marginal V-entropy is constant and that you do not use any properties of V-information, all you do is look at the standard loss of a predictor (eg cross-entropy loss of a classifier). So instead of saying that you approximate a difference of V-information terms and spending one page defining that, it would be more correct/concise/simpler if you said what you actually did: \"we consider how much pretraining on one modality helps training on other\".\n    2. **Your use of V-information is imprecise or wrong** E.g. \n        - What do you mean by $\\mathcal{V}(X_1)$? The family of networks is independent of the initialization and so your equation 3 is always 0, unless you condition the family on the exact optimization pipeline (algorithm/epochs/...). Is that what you meant? If that's the case then one of your properties is wrong (see below).\n       - Wrong property $d(X_1;X_2) \\geq 0$ with equality if $X_1=X_2$. Assuming that you define $\\mathcal{V}$ as I did above then it is neither true that $d(X_1;X_2) \\geq 0$ nor that $d(X_1;X_1) = 0$. Indeed, if you condition $\\mathcal{V}$ on the number of steps then training from a better initialization will actually improve performance because you essentially train for longer. I.e. you will nearly always have $d(X_1;X_1) < 0$.\n       - I did not think much about the triangle inequality but it likely doesn't hold for the same reason as non-negativity. \n       - More generally please provide more precise definitions for your main metrics (in particular $\\mathcal{V}(X_1)$) and at least provide some evidence (ideally a proof) for properties that you claim.\n- **Missing discussions on computational efficiency** I might have missed it but I do not see any discussion on the computational efficiency of your metrics. If I understood correctly, for the unimodal case you compute the heterogeneity measure $d(X_1;X_2)$ for every pair of the modalities which requires training M^2 + M models (M corresponds to the unimodal models and M^2 to the unimodal models initialized from another modality). Am I missing something? if not your method scales badly with the number of modalities. Regardless you should talk about how your method scales with M (both for uni and cross-modal).\n- **Missing an essential baseline** If I understood correctly that you need to train M^2 + M unimodal models (see above), then there is a more efficient baseline that I think is more natural and likely performs better. Namely to first train M unimodal models and then all M choose 2 models trained on pairs of modalities. For a total of (M^2-M)/2 models. This is more efficient and likely better given that it is closer to how the models will be used. Please clarify if I missed why such a baseline would not be applicable.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to learn a single model that can perform a wide variety of multi-modal tasks (eg a prediction task that depends on audio and video inputs). Following previous work, they first use a transformer that takes as input a sequence of (modality-specific) tokens and they then combine the outputs of that unimodal transformer using crossmodal attention. The main novelty is that they try to automatically learn which (single or pairs of) modalities should be processed using the shared weights. Specifically, the main contributions of the papers are:\n- decide whether to process two modalities with shared weights, if training on one modality after training on the other helps. \n- decide whether to process two pairs modalities with shared weights, if training on one pair after training on the other helps.\n- show experimental gains (caveat: I do not know this benchmark )", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**\n- **Misleading bold numbers** Your bolding in tables is not consistent inter and even intra-tables (let alone with the common approach). Table 2 you say that you bold your method's numbers if you are beating SOTA but then you should not bold UR-FUNNY and MIMIC. Furthermore such bolding is very misleading, please use the common approach of bolding the number that is statistically significantly better than others (ie don't bold the aforementioned datasets but bold SOTA for MOSEI and AV-MNIST). In table 3 you just bold your method (eg MOSEI w/o cross-modal performs better). Please Bold again statistically significantly better methods (or at least the best method). Please do so in all your tables (eg also table 12).\n- **Making your metric and use of V information more precise/correct** see above.\n- **Discussing how your method scales with M** see above.\n- **Adding the baseline** if it is applicable.\n\nI will update my review of \"Claims And Evidence\" if you at least change the first two points.\n\n**Suggested**\n- As suggested above I would actually completely drop the use of V-information given that you never use any of its properties and drop the V marginal entropy. I.e. you use a metric that could be explained in much simpler terms to the ML community without losing anything. This is a major change, which is not critical but would really improve the clarity of the paper.\n\n**Minor**:\n- page 3: \"be invariant\" -> \"decrease\"\n- page 4: \"metric space\" -> \"metric\" or \"distance function\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RB4AGitP1o", "reviewer": "Reviewer_tic5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work addresses interesting topics, not so well explored in the multimodal learning literature, as, for instance, the \"similarity\" between different modalities and the contribution of diverse multimodal data/tasks in the performance of a very different new task, but also there are several shortcomings.\nIn general, the paper is far from being clear, it is difficult to catch the technical aspects of the work, just for the way it is presented. The wording is in some cases hard to understand and hard to figure out the message authors want to convey. Moreover, some of these contributions are not properly justified, and it is difficult to find an accurate, convincing and clear evidence of the stated claims.\n\nFirst of all, as per the proposal of the information theoretic metrics for heterogeneity measurements of the modalities, I haven't understood clearly the meaning of the interaction heterogeneity, that is, the transfer capability of a pair of modalities to another pair. While it is natural to assess the heterogeneity between 2 modalities, this concept of \"interaction\" is not explained, and indeed I found difficult to understand the utility. One could also ask why modality pairs and also triple or more modalities. \nThe second contribution relates the proposal of a single model able to manage the large amount of multimodal data and tasks, namely, 10 modalities, 15 tasks in different research areas. While the use of a \"single\" model is surely an interesting aspect, the main paper develops without knowing the type of model proposed, other than it is likely a transformer-like system; the actual architecture is only presented in the Appendix, so one attempts to understand everything without knowing how this model manages different data (size, cardinality) and tasks, and their combination, how information flows, etc.. Moreover, apart some contributions (not original) about the information theoretical metrics, it is hard to find any theoretical ground justifying the approach: this is not necessarily a big issue, but accompanied with the pure procedural approach, made the proposed methodology a bit weaker.\n\nMore specifically, the contribution about the metrics is firstly reported, it derives from a variation of mutual information ($\\mathcal{V}$-usable information) presented in former works, exploited here to assess the similarity/difference (called heterogeneity) between 2 modalities (and pairs of modalities). The derived theory is presented cast to the problem at hand, but the conclusion derives from an empirical observation, not corroborated by any other support, but just saying that experiments evidence that $\\mathcal{V}(X_1)$ and $\\mathcal{V}$ are both expressive enough to allow a simplification in formula (3). Moreover, the introduction of the distance $d$ is claimed to respect the metric rules (non-negativity, symmetry, and triangle inequality), but this is also not demonstrated.\nThe same reasoning is applied to interaction heterogeneity measurements, but still issues of clarity are present.\n\nThis analysis is then used to define heterogeneity among the modalities and determine parameters' grouping in the subsequent model bia hierarchical clustering, which is not well addressed too, just reporting that the number of clusters $k$ is a hyper-parameter depending of the computational budget. hence, the proposed model presented in the Appendix is subject to homogeneous pre-training across all modalities followed by an heterogeneity-aware fine tuning. \nThe explanation for the pre-training stage goes through a set of steps addressing the standardization of the input sequences (across modalities) and further data management and training of unimodal and cross-modal networks up to final classification stage, which can be task-specific or multitask. The heterogeneity-aware fine tuning looks for the similarity of the modalities and pairs of modalities and groups accordingly the networks' parameters, and, starting from the homogeneous pre-trained models, separately fine tune the model.\nNot so clear explanations and justifications are given for all these steps as it is likely evident from my description above. The procedural stages are not better described, which results difficult from a reader to fully catch why this procedure is followed. This is probably true for readers not particularly skilled in Transformers, but I think that also Transformers experts can find problems to fully understand the methodology adopted as reported in the paper.\n\nAs per the experimental session, I found similar issues. For instance, datasets are quoted without explaining what they are, leaving such description still in the Appendix. Some comments are here reported but it resulted difficult to figure out well since the datasets and tasks were not explained in advance. In short, the overall section suffers from the same problems of clarity present also in the previous sections, as well as in claims not clearly supported by the results.\nFor instance, Table 1 shows some quantitative performance on the transfer to new tasks, but the improvement is in some cases not so evident, given all the expectations of the method (it is claimed about \"strong\" evidence, but it is indeed not so strong). \nTable 2 presents other results wrt the sate of the art, without actually mentioning the works they are comparing to. \nAlso the ablation studies are not so clearly explained and figures are sometimes contradictory (see Table 3, e.g., direct training re HighMMT results). Finally, all results are given as single performance, but it'd safer to validate the model with more runs per single experiment and report results with average and standard deviation (variance) figures.\n\nState of the art is quoted at the end of the paper, but it results difficult to cast the proposed work in relation to it, and, as mentioned above, comparisons in terms of performance is not well addressed.\n\nThe Appendix improves a bit the situation but still issues about clarity remain. For example, Fig. 7  shows the HighMMT architecture but it's unknown where the latent vector in input comes from. Other claims are reported in other sections in the appendix but they are not well corroborated by evidence or justified properly, e.g., the low-rank nature of the modality and interaction heterogeneity matrices, the use of Fourier transformation for positional encoding.\nMoreover, going into more details regarding the procedure, specifically the data standardization step, since all data should be normalized across modalities taking the max dimensions, it results that AV-MNIST data are very low dimensional and require zero-padding of 336 out of 387 elements: this sounds a bit odd to me, and would require better investigation. Other specific steps of the training procedure would require more explanation and, likely, further ablation analyses.\nFinally, Fig. 10 reports the visualization of attention patterns learned by the unimodal encoders across different modalities and data, and it's claimed that such visualizations evidence common (same position) patterns across tasks, but this finding is not so straightforward to me: given the variety of the data modalities and tasks, I'd expect that each encoder catches an attention pattern for each modality which is not necessarily in the same position of other data modalities, rather, I'd expect different patterns in dependence of the specific modality and task.\n\nIn the end, poor paper organization (including both technical stuff and experiments), clarity of the exposition, claims not always corroborated by evidence, and sometimes too strong, poor theoretical ground, and also specific technical issues affect the quality of this work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the multimodal learning topic by the standpoint of knowledge transfer, i.e., trying to investigate how several combinations of (even very different) data modalities and tasks behave so to improve generalization and transfer capacity to new diverse modality/task. This work specifically considers under-investigated modalities (such as, e.g., tabular data) and proposes to measure the heterogeneity between the modalities, so to assess the similarity of the encoded information as well as of the interaction, in order to allow parameter sharing. Hence, two information theoretic metrics to measure modality and interaction heterogeneity are proposed, in order to quantify, let's say, the \"affinity\" (in terms of information transfer) between two modalities and pairs of modalities (for interaction).\nSubsequently, a new model, named HighMMT, is presented, able to deal with diverse combination of modalities and tasks, regardless the type of data, cardinality, and other peculiar characteristics of the data and of the tasks. A large experimental analysis is reported considering the MultiBench benchmark, which considers text, images, audio, video, non-visual sensors, proprioceptive data, speech, time series, sets and tables, showing promising results in terms of performance (the model provides higher accuracies as long as new modalities are added, and transfer well to new modalities and tasks during fine tuning) and efficiency (scales well due to the parameter sharing).\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main issues regard clarity and organization of the content of the work. Meaningful information is in the appendix and should be included in the main paper, but especially, practically all the paper should be rewritten in order to clarify and justify the proposed method and the several stages. \nMany of the specific issues have been reported above, and should be referred for the paper revision.\n\nAFTER REBUTTAL\n\nI believe that the paper, even after authors' revision, is still affected by notable problems to be solved, first of all regarding the organization and clarity of the work, which cannot only be fixed by adding paragraphs here and there along the manuscript.\nFor example, the modality and interaction heterogeneity measures seen under the perspective of information transfer are not enough convincing: they both look very similar considering the difference of the losses between standard training with random init weights and training using init weights taken from former pre-training with the other modality, and this is valid for both single modalities and pair of modalities. I don't see here a convincing justification of such an approach and, moreover, the method is combinatorial with the number of modalities, so poorly scalable.\n\nAssumption of distances as metrics only valid for some percentage of the time remains fuzzy.\n\nExperimental results suffer of the same problem: despite the explanation is a bit better, I still deem that given the complexity of the method is difficult to understand which its actual contributions, and results do not seem significantly better in a statistical sense.\n\nOverall, I acknowledge that authors made a lot of work in proposing this methodology, but the way it is presented should be much improved to be made understandable, convincing and acceptable for publication.\n", "broader_impact_concerns": "A section about  Broader Impact Statement is not present, but I don't see ethical concerns for this work, at least no more than other machine/deep learning work of the same flavor.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper underwent a significant revision based on the comments from reviewer Yccr, who proposed that the authors discard the original formulation of the method based on $\mathcal{V}$-information, and reviewer tic5, who called out many issues with the organization of the paper (too much important information relegated to appendices), clarity (better motivation for training procedure, better explanation of ablations), more substantiation of the approximate metric properties of the heterogeneity measures, and clearer relation to SOTA results.

Following the revision and discussion, reviewer tic5 still had the same concerns, while the other two reviewers were satisfied. I therefore carefully read the revised paper myself and went point-by-point through the issues raised by reviewer tic5. In my judgement, the revised paper does adequately address the concerns raised. While I am not following reviewer tic5's final recommendation, I thank this reviewer for the careful and thorough review that helped to greatly improve the quality of the paper, and to reviewer Yccr for recommending a reframing of the heterogeneity measures without recourse to $\mathcal{V}$-information.

However, I did find a small number of issues listed below that should be addressed in the final version of the paper.

- Figure 3 and Equations 9 and 10: it is not ideal to use "Z" for both the latent vectors used by the perceiver and the output of the unimodal encoder without some notation to indicate that Z is the result of an unrolled recurrence.
- Table 3: It would be helpful to remind the reader that these are few-shot transfer results (as stated in the reply to Reviewer tic5). Also, in the case of pretraining on one task, there are three tasks to choose from, and likewise when pretraining on two tasks there are three task pairs to choose from. What choices were made for the results in Table 3?
- The paper should explain somewhere that means and standard deviations in tables are based on 10 runs.
- In the caption to Fig. 11 (in Appendix), remind reader that the modalities are time-aligned. Also fix typo: "common attention patters" -> "common attention patterns"

Decision: Accept (poster)