Training Transitive and Commutative Multimodal Transformers with LoReTTa
Authors: Manuel Tran
Yashin Dicente Cid
Amal Lahiani
Fabian J Theis
Tingying Peng
Eldad Klaiman
Venue: NeurIPS 2023 poster (2023)
Keywords: generative pre-training
causal modeling
masked modeling
commutative modeling
transitive modeling
multimodal learning

Abstract: Training multimodal foundation models is challenging due to the limited availability of multimodal datasets. While many public datasets pair images with text, few combine images with audio or text with audio. Even rarer are datasets that align all three modalities at once. Critical domains such as healthcare, infrastructure, or transportation are particularly affected by missing modalities. This makes it difficult to integrate all modalities into a large pre-trained neural network that can be used out-of-the-box or fine-tuned for different downstream tasks. We introduce LoReTTa ($\textbf{L}$inking m$\textbf{O}$dalities with a t$\textbf{R}$ansitive and commutativ$\textbf{E}$ pre-$\textbf{T}$raining s$\textbf{T}$r$\textbf{A}$tegy) to address this understudied problem. Our self-supervised framework unifies causal modeling and masked modeling with the rules of commutativity and transitivity. This allows us to transition within and between modalities. As a result, our pre-trained models are better at exploring the true underlying joint probability distribution. Given a dataset containing only the disjoint combinations $(A, B)$ and $(B, C)$, LoReTTa can model the relation $A \leftrightarrow C$ with $A \leftrightarrow B \leftrightarrow C$. In particular, we show that a transformer pre-trained with LoReTTa can handle any mixture of modalities at inference time, including the never-seen pair $(A, C)$ and the triplet $(A, B, C)$. We extensively evaluate our approach on a synthetic, medical, and reinforcement learning dataset. Across different domains, our universal multimodal transformer consistently outperforms strong baselines such as GPT, BERT, and CLIP on tasks involving the missing modality tuple.

Training Transitive and Commutative Multimodal Transformers with LoReTTa
Manuel Tran 13,4
Yashin Dicente Cid?
Amal Lahiani'
Fabian Theis
Tingying Pengl.*
Eldad Klaiman 1.*
Roche Diagnostics GmbH, "Roche Diagnostics S.L. Technical University of Munich, 'Helmmoltz Munich
Abstract
Training multimodal foundation models is challenging due to the limited availability of multimodal datasets. While many public datasets pair images with text, few combine images with audio text with audio. Even rarer datasets that align all three modalities at once. Critical domains such as healthcare, infrastructure, transportation are particularly affected by missing modalities. This makes it difficult to integrate all modalities into a large pre-trained neural network that can be used out-of-the-box fine-tuned for different downstream tasks. We introduce LoReTTa (Linking mOdalities with (Ransitive and commutativE pre-Traiine Training sTrAtegy) address this understudied problem. Our sele-vuue framework unifies causal modeling and masked modeling with the rules of commutativity and transitivity. This allows us transition within and between modalities. As result, our pre-trained models are better at exploring the true underlying joint probability distribution. Given dataset containing only the disjoint combinations (A, B) and [B, C".. LoReTTa can model the relation + C' with A + B ++ C. In particular, we show that transformer pre-trained with LoReTTa can handle any mixture of modalities at inference time, including the never-seen pair (A, C) and the triplet (A, B, C)). We extensively evaluate our approach on a synthetic, medical, and reinforcement learning dataset. Across different domains, universal multimodal transformer consistently outperforms strong baselines such as GPT, BERT, and CLIP tasks involving the missing modality tuple.
1 Introduction
Integrating multiple modalities, such as images, text, or speech, promises to provide more compre- hensive and holistic understanding of otherwise intractable problems. By leveraging the unique and complementary strengths of each data type, this approach results in a more synergistic representation of the underlying phenomena enabling deep learning systems excel and adapt real-world scenarios. Thus, there has been impressive progress the field of multimodal learning in recent years. While most research focuses on multimodal datasets where all data points have all modalities. there has also been work on missing modalities [44, 45]. However, all these models and methods rely heavily on at least subset of samples where all modality combinations In practice, it is not easy to obtain such datasets when we consider more than two modalities, e.g. three modalities. In the worst case, we might end up with dataset containing only modalities A and B, and another disjoint dataset containing modalities B and C. Thus, the combination A and C
still present (Figure 1).
Equul contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) All Modality Combinations Present
(b) All Modality Combinations Sometimes Present
(c) Some Modality Combinations Never Present
Dataset With Modality A
B Dataset With Modality
Dataset With Modality C
Figure 1:
Venn diagrams showing the relationship between datasets with different modalities A, B, and C. Overlapping datasets indicate that the dataset contains samples with aligned modalities (i.e., audio, image, and text files belonging to the same concept). While recent work has mostly focused on datasets where least some samples have all modality combinations available (a. b), we investigate the case where some modality combinations, e.g. (A, C²) and (A, B, C), are missing entirely (c).
unconditionally missing. we asked ourselves, "Is possible to pre-train a multimodal neural network that can handle any combination of modalities at inference time? Optimally, the model will achieve better downstream performance the more modality combinations are given. This includes the never-seen combinations (A, C²) and (A, B, C).
To the best of our knowledge, this scenario has never been considered in the literature. We propose a novel transitive pre-trained transformer (Figure 2) that can deal with such situations. LoReTTTa (Linking mOdalities with tRansitive and commutativ E pre- Tra-nrainnn s'TrAtegy) can link modalities and transition between them using generative pre-traine trainininn (predicting the next token), the commutative property (A, B) (B, A). and the transitive relation (A - B3 A (B + C) = (A + C). We show theoretically and experimentally that by learning the relationship within and between tokens from different data distributions, LoReTTa is well suited for training multimodal models.
This has far-reaching implications for safety-critical domains such as healthcare, infrastructure, and transportation. medicine, for example, it already difficult to obtain large dataset with single modality to train powerful foundation models such as those used in general computer vision and natural language processing. Finding dataset with But combining them is important because it is common practice to look at different types of data, such as radiological images, genetic sequences, and microscopic slides, to determine the optimal treatment. Similarly, infrastructure monitoring models or automated driving systems require input from variety of sensors. However, all modality combinations might be present the same training set. With LoReTTa, we demonstrate unique way to address this challenging and under-researched problem.
modalities even challenging.
Related work
With the introduction of transformers [63], the fields of natural language processing (NLP) and computer vision (CV) have experienced significant shift in performance and capabilities. These models excel a wide range of complex tasks, including machine translation, text summarization, question answering, sentiment analysis, natural language generation, image classification, image captioning, semantic segmentation, object detection, and face recognition achieving state-of-the-a performance in each area [40]. Transformers have also opened up new possibilities for multimodal learning due to their lack of inductive bias. They can process and fuse different types of data, such as audio, image, video, and text, using self- and cross-attention mechanisms. As result, transformers are considered to be universal, modality-agnostee learners [43, 22].
However, their lack of inductive bias comes at a price. Unlike other neural network architectures, transformers make few assumptions about the structure of the data [18]. This makes them highly expressive, but they also require large amount of data to learn effectively and generalize well [28]. Consequently, supersupe learning (SSI.) on large amounts unlabeled data has been proposed for transformers. In NLP, this strategy has been very successful, with methods such as masked

---

Causal Modeling (Switch Pairs for Commutative Modeling)
(b) Causal Masked Modeling
B2
BG
Transtormer
Transformer
Transtormer
AD
A1
A2
A3
B0
11
B2
B3
BQ
B1
B2
E3
CO
C1
C3
0
M
5
M
3
(c) + Transitive Modeling
(d) Before & After Transitive Pre-Training
Otten Available:
A
Tralining
A0
A3
A
A
B
Transformer
Always Missing:
A
C
AD
8
C
B
Inference
Transformer
A0
A1
A2
A3
B0
B1
B2
B3
use as targets
use as inputs
predicted
8
B
Figure 2: LoReTTa consists of two novel self self-suuer strategies: commutative and transitive modeling. (a) In commutative modeling, we apply causal modeling in a commutative manner to generate modality A from B and modality B from A given the aligned input sample (A, B). For the aligned but disjoint data point (B', C), we apply the same technique. (b) To ensure that the model learns bidirectional relations between the input tokens, pre-training called causal masked modeling. (c, d) Next, we use transitive pre-training to learn any missing conditional joint distributions. The idea is simple. We randomly select sample and use the linking modality B to predict the missing modality C, which is then used to reconstruct the existing modality A. The last step is crucial because ensures that all modalities are properly aligned.
apply modified variant of generative
modeling and causal modeling used to train large language models (LLMs) such BERT [23] and GPT [49] variants [42, 46).
Recently, attempts have been made to unify both approaches through frameworks like prefix mod- eling [53], permutation modeling [71], causal masked modeling [1], or unified language learn- ing [59]. When applied to image classification, token-based pre- pre-training methods [8, 15] have demon- strated their ability to outperform established computer vision approaches such as SimCLR [17] and DINO [13]. As result, these sequence modeling techniques have been successfully adapted and extended a wide range of domains, encompassing programming [16), biochemistry [41], and reinforcement learning [35].
Adding or combining different modalities to the training process could open up three possibilities: positive transfer, cross-modal understanding, and cross-modal generation. Positive transfer occurs when learning in one modality helps to improve performance another modality. For example, has been shown that aligning images with text leads to better classification performance on ImageNet [51]. Cross-modal understanding refers to the explanation of one modality in terms of the other. Models with this capability can describe image using text [4] or infer protein structures from sequences [30]. With cross- mooss-modal generation, one can even generate one modality using the other. Examples are text-to-image generation [54] and text-to-music generation [3]. Many modern multimodal foundation models are based on either contrastive learning [26]. masked modeling [66], or causal modeling [72].
In this paper, we present novel pre-training method that combines recent advances in self-supervised learning into unified framework. LoReTTa uses causal modeling for generative tasks and masked modeling for discriminative tasks. It extends both approaches by using commutative and transitive modeling to seamlessly integrate and impute modalities. This allows us learn expressive features and relationships within multimodal datasets. The pre-trained models can be used for various downstream tasks such as infilling, classification, survival prediction, or cross-modal generation. The most similar to our work is the RNN-based model MCTN [47]. However, it can only be used as classifier, does not model all combinations of modalities, and requires modalities to be aligned and present at training time. LoReTTa, on the other hand, does not need such strong assumptions.
3

---

3 Methods
Model architecture: LoReTTa (Figure 2) uses the autoregressive transformer decoder architecture described in Radford et al. [50]. This model includes a modified initialization, pre-normalization, and GELU activation. In addition, we choose RMSNorm [74] over LayerNorm [7]. as in Rae et al. [52] and Brown et al. [12]. We also replace the StandardAAtennton [63] with FlashAttention [21] to reduce memory usage and runtime- similar to Touvron et al. [61]. Following Chowdhery et al. [19], train without dropout and do not use biases in any of the dense kernels layer norms. While any general sequence model can work for next token prediction, we choose the transformer [63] in particular for its simplicity and scalability. Theoretically, however, our method also works with other backbones, such as the recently proposed RWKV [10] and Hyena [48].
Data representation: Since transformers take a sequence of tokens input, we need to tokenize our data first. We use common methods such as raw byte encoding [56, 29, 15], learned lookup tables [33, 58, 57], and vector quantization [62, 55, 54, 3]. After tokenization, we a parameterized embedding function to project all tokens into a common feature space. In addition, we add modality- specific tokens to indicate the beginning and end of a modality. We also learn modality-specile absolute position encodings that are added to the feature vectors. The representations for each modality then concatenated, but only if they belong the same object/subject. For example, if we have an image-audio pair or an image-text pair describing the same concept, we would concatenate their token sequences and feed them into the transformer. However, if only one modality is available, we use only that modality. In summary, our pre-processing step is described below:
We tokenize and project all samples b, c into a common vector space, obtaining the token embeddings [ai,-... un], [b,,... bm). and [ci,.... C.]]
If a and are aligned, then we concatenate them as s = [a1.... aj bi..................... bm] or s [bi..... bm a1,..., ar), which ensures commutativity.
Pre-training strategy: We train the transformer using causal language modeling. Given sequence of tokens $1.L and parameters 0, the data is modeled using the chain rule of probability
I.
log logpo(s1, sp) =
Σ
log (s 11-1)
Thus, the task of the model is to predict the next token given the sequence of previous tokens. As shown in GATO [56] and PALM-E [24], this objective is modality agnostic. The systems even learn rich internal world models [38, 36). We ensure commutativity by randomly mixing the order of concatenation when appending tokens from two modalities to an input sequence, as described above. This allows the model to infer modality A from B and modality B from A. We call this technique commutative modeling. Interestingly, this simple modification has not been used in recent multimodal foundation models.
Since generative pre-trained transformers only consider information from the left and ignore in- formation from the right, they are known to lack bidirectional context [65], which is crucial for many downstream tasks [53]. Thus, we combine masked modeling (GPT-style models) and causal modeling (BERT-style models) into one framework. This recently developed approach is known in the literature under different names: fill-i--teemmidlle (FIM) [9], causal masked modeling (CM3) [1, 25), hybrid unidirectional modeling (HybUni) [5]. span corruption + language modeling (SCLM) [59]. Conceptually, the idea is to randomly mask out a span (or spans) of tokens, replace them with sentinel tokens, and move the original tokens to the end. This change does not alter the training process based on Equation I; however, it now allows the model to use the information to the right of the mask to predict the masked positions, which is not possible with traditional unidirectional causal modeling. The hybrid view not only leads to better performance in downstream tasks after fine-tuning but also retains the utility of language modeling [5, 59]. With CM3Leon [73], causal masked modeling has been scaled up to large vision- langua- models capable generating and filling both text and images.
Transitive modeling: We now extend several language learning paradigms (masked, prefix, causal, causal masked) with our new technique, which we call transitive modeling. To do this, we return to the problem formulation outlined in Section 1. In particular, it requires us to combine modalities A and

---

C. given only the disjoint combinations (A, B) and (B', C). We propose to randomly sample data points (a,b), (b. a), co),),, or (c. b') during training and apply transitive modeling. For simplicity, we will consider the pair (a, b). Since the model is trained to predict modality C' from B. we can infer the pseudo data point é from Now we train the transformer to predict point á using the newly inferred sample è in way that minimizes the loss with respect to the original sample a, ensuring that all our modalities are aligned (Figure 2c). In this way, we model the conditional distribution of the missing pair (C, A) which, due to commutativity, is equal to (A, C). This idea has some conceptual similarities to cycle consistency. However, our method is much more general since it can be applied to many combinations of modalities as long as there is a linking modality, see Figure 2d.
In fact, upon closer inspection, transitive modeling quite different from cycle consistency, The most popular version of cycle consistency was introduced in CycleGAN [75]. Given an input a in domain/modality A, one wants to generate an aligned output in domain/modality B. The model then computes discriminative loss D on b and reconstruction loss L on the original a and the predicted à. A different version of cycle consistency was used in MCTN [47]] Given aligned inputs (a, b) from modalities and B, a is used to predict b, and the predicted b is used to predict à. Then the reconstruction loss L is applied to compare a and â as well as b and b. Transitive modeling, on the other hand, starts with the aligned modalities (a, b) and (b',c) it uses b predict è and é predict à, the reconstruction loss then compares a and â. Since we use commutative modeling, the other direction starting from a different sample (b', c) is also learned. In summary, we have the following three approaches:
CycleGAN: Given a, model - b - à and calculate T(a,a) + D(i).
MCTN: Given (a, b), model a -> + à and calculate L(a, a) + L(b, 8). LoReTTa: Given (a, b), model b + -> â and calculate Ta,,a).
Thus, transitive modeling does not simply add another step to cycle consistency by extending b to - b - + à. In fact, we model b - è + â. Intuitively, this ensures that the model does not "cheat" by memorizing the input a in order to reconstruct a. We also do not use cycle loss, because a is not used as an input and output. Thus, our loss L(a, â) is not cyclic with respect to the model's inputs. However, we still make sure that the predicted modality is consistent with the data as a whole. This has a similar flavor to cycle consistency, but as can be seen above, it is not the same. In summary, LoRe'TTa uses masked modeling to transition within modalities, causal modeling to transition between modalities, and vice versa thanks to commutative modeling. Moreover, not all modality combinations need to be available at training time, we use transitive modeling to predict the missing modalities and connect them to the original data. In our experiments, we show that this approach allows us to learn very rich multimodal feature representations.
4 Theoretical analysis
We consider a dataset with modalities X2,.... X to be aligned if there exists a subset S sampled from a joint probability distribution P(X1, XN; Y) [37]. Here, Y represents the label space. This form of alignment, where only labels are shared across modalities, is called weak alignment [14]. Conversely, when there is functional relationship at the level of physical objects, we consider the alignment to be strong [14]. The definition of strong and weak alignment also extends to datasets where do not have all modality combinations once. For example, during data collection, we may recover only the paired samples (I1, x22 and (x22, a2),, where 32 and wt do not belong to the same object/subject. Thus, we have datasets of the form (X,, X2) and (X2,X3) instead of (X1, X2, Xa). This is different from the case where the dataset still contains some samples of the form (x 22, 23), which was considered in previous work. To model the unseen relationship (X,.X4 Xs) in a self-supervised way, we cannot simply use techniques like masked modeling or causal modeling. The neural network would have to implicitly learn
p(11,
p(a = p(x1)
by deriving p(a11 m2) = Ex, p(x1, x2, 23, 13) and p(11, x2, x3) = p(2g|12, 2n )p(2)) jn )p(r)) from the chain rule of probability. Empirically [6, 39, 69], both the unidirectional and bidirectional
5

---

Table I: Perplexity on the SVL-MNIST test set. The subscript describes the pre-training datasets with image (I), text (T), or speech (A), while the blue colors highlight the unseen modality pairs.
Training
Testing
T
A
(I, T)
(L,
(T,
CM
1.55
CM3
2.05
(M3
18.63
C2M3
1.62
2.28
1.76
C2M3
1.61
20.09
4.42
C2M3
2.25
19.85
10.68
C2M3
1.64
2.19
21.37
1.76
4.39
30.88
C2M3 (LA)
1.65
2.18
19.69
1.76
104.27
11.26
CJM (A.))
1.64
2.19
23.45
9.70
4.66
11.92
LoReTTaID (LA)
1.56
2.17
18.77
1.72
3.14
11.21
LOReTTaCDD (TA)
1.59
2.10
18.95
1,70
5,04
10.03
LoReTTaAAD.AN
1.60
2.14
20.35
3.13
4.36
10.65
C2M3
1.55
2.35
19.43
1.72
4.25
10.55
approaches fail this task even within one modality because it is difficult to model p(zz3 [22 .1) due to the missing set (X1, X2, X3). This where our proposed transitive modeling paradigm comes in.
Suppose we have a dataset of the form (X1, X2), (X2, Xs)..... (XN-1, Xx). where X₃ and X₃ are not aligned when & # I. By applying causal modeling all pairs,
learn the transition X1 + X2 - + XN-1= - X Adding our proposed commutative modeling, we get an even stronger bidirectional relation: Xi + X2 + + XN-1 XN. Now, let us interpret each modality as node on a high-dimensional graph. This gives us a connected graph (minimum spanning tree) where we can walk from one modality to any other modality. Thus, given real data point x,, we can use transitive modeling to sample pseudo data points 2, by transitioning from 2; to zy. Having sampled many pairs (Ix,, 2 we get a new dataset (X,, X2). It can be used again in the generative pre-training process to learn the new transition X, ++ Xy. We apply this idea to all and j. In the end, we learn the conditional distribution between all samples and obtain a fully connected graph. This idea holds for more complicated modality combinations as long as there is bridging modality, ie., we have a spanning tree (see Figure 2d). We can also sample all missing tuples (T),.... 31). Thus, LoReTTTa allows us to approximate any missing joint probability distribution P(X X;) by sampling and training with the missing modality combinations.
Ideally, there would be no generalization error after pre-training. In practice, however, we would not be able to accurately recover the ground truth of the missing modality if were available. For the datasets X2) and (X2, Xs). the learned generative model would make the following errors:
(1)) 21 12,1,
f(22/=1) = x2 + E1,2 f(12/03) = x2 + 63,2 f(13(02) = 23 + e2,3
where all eij vector-valued error terms. Without loss of generality, we do not keep track of the sign of the error, since we can always write e/,, -1- e.j. Assuming that is differentiable and using the multivariate Taylor expansion, obtain
r3
(2)) e1,2) + 62,3 = f(zalf(z2lz)) (e
+ 62,3, (4)
is the Jacobian of f. By iteratively applying the Taylor rule, a more general error propagation formula is obtained for longer chains. Intuitively, errors from the first transitions are amplified more and more. However, because we use the predicted vg to reconstruct #1, we can actually bound the + 62,3. Recall that we have ground truth 21 for
where is
emor term 61,3 == Jy (zal/ ( which the following is true
#1 = f(ailza) +3,1
[ [f + +2,3) + E3.1
2122 + 22.3) +
(5)

---

Table 2: After pre-training, classifier is trained on the frozen models. We analyze the classification accuracy in a few-shot scenario with 100 samples per class over three trials on SVL-MNIST. The subscript denotes the pre-training datasets with image (I), text (T), or speech (A) modalities, while the blue colors highlight the results on the unseen modality tuple during pre-training.
Training
Testing (L
I
T
A
(A
(T,
(L. T,T)
CM
79.7
()
57.0+0.5
57.0
(M
8..3
C2M3aT)
82.9
61.910.7
89.7
83.8
79.9
89.9
C2M3
61.1
79.5+0.2
85.2
C2M3
83.
59.8+0.1
80.7=0.1
87.9
89.3
71.2=0.4
87.340.4
C2M
81.7+0.1
63.2+0.6
78.5+0.6
89.1
72.9=0.2
84.8
83.240.4
AJ,
80.8+0.4
63.9
82.84m.7
77.8
89.8_n.1
88.140.5
89.940.5
LoReT (AA)
82.7
62.8
82.5
88.5
89.7
84.0.0.2
90.7
LoReTTa(ID, (TA)
81.2+0.9
63.3+o.4
81.020.5
89. 89.022.3
80.920.6
85.520.5
87.840.3
LoReTTBANDAA
40.3
42.
84.2
83.0
90.4
89.0
91.640.7
80.3+0.4
58.3+0.4
79.3
85.2 85.2+0.2
85.3=0.4
82.5
88.540.2
Since we minimizing the reconstruction error of æ given à3 33 e1,3 during backpropagation, the error terms 61,2 and 62.3 must be bounded. And since they make up the error term E1,3, it too must be bounded, as long f and y are.
Experimental evaluation
We empirically analyze LoReTTa on a constructed dataset, real-world medical dataset, and an offline reinforcement learning dataset with three modalities. For the downstream tasks, we choose object classification, survival prediction, and cross-mode translation, respectively. With these three tasks, we cover both discriminative and generative problems. We assess the former through linear probing and the latter in zero-shot scenario. This allows to directly evaluate the quality of the learned features. The order of the experiments is as follows: First, we perform an ablation study on the synthetic dataset to analyze the effect of transitive modeling on different modality combinations. Second, we compare LoRe'TTa with other methods such as masked (BERT), causal (GPT), and contrastive (CLIP) modeling on the real-world medical dataset. Third, we compare our model with a state-of-the-an autoregressive cross-modal transformer on the offline gaming dataset. In the Appendix, we list all the optimization hyperparameters and model configurations. We also publish the pseudocode and data processing pipeline.
Evaluation protocol:
SVL-MNIST:
We test our method on custom dataset derived from real-world data that includes speech (A), vision (I), and language (T) modalities representing 10 different categories. The speech dataset features about 40,000 spectrograms from AudioMNIST [31], the vision dataset comprises 70,000 images from MNIST [34], and the language dataset consists of 130,000 documents from WineReviews [60]. We link the datasets via their labels 0-9. This approach results in weakly aligned modalities. For the sake of clarity, we will refer to this dataset Speech- Lision-Langugge-MNIST (SVL-MNIST). We randomly split the dataset such that they have exactly the same relationship. as shown in Figure 1c. In particular, our bimodal datasets (A, 1), (A, T), and (I, T) each have 12,000 non-overlapping samples. All remaining samples are part of the unimodal datasets A, I, and
The medical dataset contains omics (c.g., genomics, proteomics, and transcrip- tomics) sequencing values from The Cancer Genome Atlas Program (TCGA) [67]. We select the three largest subsets, which include messenger ribonucleic acid (mRNA). micro-ribonucleic acid (miRNA), and reverse-phase protein array (RPPA) from up to ~ 11,000 patients. We align the dataset at the patient level and obtain approximately 7,000 patients with all three modalities. To simulate setting with missing modality combinations, we split the dataset into two subsets (mRNA. miRNA) and (mRNA, RPPA) with non-overlapping 3,500 samples each. We choose mRNA as the linking modality because is one of the most widely available genetic modalities. TCGA is quite unique in that it is the only publicly available medical dataset with multiple aligned modalities.
TCGA-OMICSS
7

---

MUGEN-GANT As a large-scale and multimodal dataset, we choose MUGEN- GAME [27] with 375,000 naturally aligned video (V), audio (A), and text (T) samples collected by reinforcement learning agent in the closed-worde world platform game CoinRun. Mugen, the main character, walks, jumps, collects coins, kills monsters, climbs ladders, and dies - triggering various sound effects that are overlaid with background music. All of their adventures are recorded in three-second video clips and audio tracks. A group of human narrators provide rich text descriptions for each scene, We use the most important modality in video games, video, as the linking modality and consider the disjoint datasets (V, A) and (V, T) in our final experiments.
Implementation details:
For the first two experiments, we use transformer decoder with 8 layers and 8 attention heads, with an embedding dimension of 512 and query, key, and value dimensions of 64 each. This compact architecture has proven effective in various tasks and has been widely used in the literature [23, 28, 36). We apply a single parameterized embedder to encode tokens and add modality-specte
learned position embeddings [54, 56, 2]. For optimization, we choose the AdamW algorithm with a learning rate of 6e-4, weight decay factor of 0.1, and a gradient clipping of 1. The learning rate undergoes a 10-fold decay using cosine annealing and a linear warm-up during the first couple hundred steps. For the third experiment, we use exactly the same training hyperparameters and model configurations as in the baseline [27]. In particular, the transformer has 12 layers, 8 attention heads, and an embedding dimension of 768. The query, key, and value dimensions are each 96.
Due to the low dimensionality of the SVL-MNIST dataset, we directly encode the raw byte streams by binning the continuous values dictionary of 256 each. TCGA- OMICS data can be very high-dimensional and must be handled differently. For example, an mRNA expression array typically contains more than 20,500 entries. Therefore, we first reduce the dimensions by a factor of 10 using PCA and then bin the values to a dictionary of 1,000 (similar to recent work [11, 70, 20J). The final input sizes are 2,048, 64, and 16, respectively - retaining their original relative lengths. The high-dimensional video and audio samples in MUGEN-GAME are tokenized identically to Hayes et al. [27] using their pre-trained VQ- encoders.
Tokenization scheme:
Results
6.1 SVL-MNIST
In the first experiment, we train one model for each modality (A, I, and T) using causal masked modeling (CM3). We then train four additional models using commutative causal masked modeling (C2M3) the bimodal datasets (A, I). (A, T), and (I, T), as well as the trimodal dataset (A, I, T). These seven models serve as unimodal, multimodal, and "upper" baselines, respectively. Then, using C2M3 with T as the linking modality, we train simultaneously on both (I, TT and (A, T), and repeat this process for all modality combinations. Next, we initialize LoReTTa with the C2M3 model weights to explicitly merge the bimodal datasets via the linking modality. After pre-training, we freeze all models and evaluate them in low-data regime with only 1,000 labeled samples (100 per class). We perform three runs, each with newly randomized subset, and report the classification accuracy along with the perplexity on the test set. Note that we only train the models on the modalities that the model saw during pre-training. This also means that during probing, the linear classifier will see some samples of the unseen modality pair. However, the pre- rre-trinne backbone itself did not see them. In this way, we can analyze how the model combines previously unseen mixture of input modalities (modality combinations).
When pre-trained only on the pairs (T, 1) and (T, A), LoReTTa achieves remarkably low perplexity of 5.04 on the unseen pair (I, A), as shown in Table 1, significantly outperforming the baseline non-transe non-transitive model trained with C2M3, which has a perplexity of 104.27. This trend extends to other combinations as well. For example, the model trained on (A. 1) and (A. T) and evaluated on (I, T) has a perplexity of 3.13 (ToReTTa) compared 9.70 (C2M3). Similarly, training on (I, T) and (I, A) and evaluating on (T, A) yields perplexity of 11.21 (LoReTTa) versus 30.88 (C2M3). Importantly. LoReTTa consistently achieves similar perplexities for unseen and seen modality combinations. demonstrating its ability to effectively leam the missing distribution. A similar trend appears in the downstream task (Table 2). In almost all cases, LoReTTa consistently outperforms C2M3 on the unimodal datasets A, I, and T, as well as the bimodal datasets (A, 1), (A, T), or (I, T). In nutshell, if any of the modality pairs are missing, training model with TTRe is recommended because uniquely integrates these modalities and improves accuracy compared to any single-modality model.
8

---

Table 3: After pre-training a Cox proportional hazards model is trained on the extracted features from TCGA-OMICS. We use mixtures of mRNA (M), miRNA (I), and RPPA (R) as inputs. Blue highlights the c-index on the unseen tuples, and the underline highlights the second- value.
Training
Testing
M
1
R
(M, I)
(M, R)
(I. R)
(M, I, R)
BERT
0.592
0.621
0.594
0.595
0.613
0.594
0.610
T-BERT
0.573
0.618
0.626
0.580
0.591
0.623
0.608
GPT
0.575
0.590
0.611
0.585
0.576
0.606
0.588
T-GPT
0.616
0.607
0.599
0.620
0.619
0.611
0.622
CLIP
0.561
0.603
0.587
0.610
0.600
0.623
0.612
L-CLIP
0.588
0.615
0.587
0.620
0.599
0.614
0.620
C2M3
0.621
0.599
0.599
0.624
0.620
0.571
0.624
LoReTTa
0.652
0.623
0.563
0.660
0.652
0.623
0.657
C2MB
0.665
0.635
0.645
0.643
0.671
0.650
0.620
In fact, providing a few samples with three modalities during linear probing further increases the classification score of models pre- pre-trained with LoReTTa. Interestingly. the "upper" bascline trained on all aligned modalities often lags behind LoReTTa despite favorable perplexity scores. This strongly suggests some kind of negative transfer or modality competition that LoReTTTa is able to overcome.
6.2 TCGA-OMICS
We pre-train our transformer with C2M3 and initialize LoReTTa with the weights of the C2M3 model, as we did in our SVL-MNIST experiments. For GPT, we simply use the algorithm of C2M3, but disable causal masking and commutative switching. BERT uses the same transformer architecture; we simply change the unidirectional attention mask to bidirectional attention mask. To directly demonstrate the benefit of transitive modeling, we extend both GPT and BERT using this technique. The resulting models are named T-GPT and T-BERT. The -Lsed model consists of 3 encoders (each using our same transformer architecture) and a contrastive loss [32] to align the features. After pre-training
extract the embeddings and fit a Cox proportional hazards model with elastic net penalty for each modality combination using all available labels. In out of 7 test sets, the LoReTTa pre-trained transformer achieves the highest c-index (Table 3). Most importantly, this includes the unseen modality pair (miRNA, RPPA) and the triplet (mRNA, miRNA, RPPA). The only frameworks par with LoReTTaa T-BERT and CLIP the unseen dataset (miRNA, RPPA). On the unseen trimodal dataset, T-GPT and CLIP give good results but are far behind LoReTTa. Overall, transitive modeling mostly improves the performance of all models. Notably, ToReTTa and CLIP are the only strategies that consistently allow for positive or neutral transfer, which is not always the case with other frameworks. major drawback of contrastive learning is that it requires a separate encoder for each additional modality. In addition, CLIP- based approaches only excel at large batch sizes. Here, we compare models with the same computational constraints. We spent our limited resources on additional computing capacity. This allows us to scale up the CLIPP experiment by increasing the batch size by a factor of and the number of training steps by a factor of L.CLIPP consistently improves the c-index of CLIP but requires 3x more VRAM than LoReTTa. In contrast to the SVL-MNIST experiments, the 3-modal "upper-bound model suffers less from negative transfer or modality competition and achieves the best result for the majority of modality combinations.
6.3 MUGEN-GAME
Recently, there has been a growing interest in cross-modal generation. While much research has focused on text-to-video or video-to-text translation, other cross-modal tasks such as video-to-audio, audio-to-vidde to-video, text-to-audin or audio-to-text have not been investigated as thoroughly. In our large-scale experiment, we address the latter task. We use video as the linking modality and consider the disjoint datasets (video, audio) and (video, text). Splitting the training set equally gives us about 187,50 87,500 pairs in each set. We use the same optimization hyperparameters and model architecture as the state-of-the-ar upper baseline MMGPT [27] to train our version of (MM)GPT and also LoReTTa. As seen in our TCGA-OMICS experiment, BERT, unsurprisingly [5, 64), fails to generate long-range coherent samples (see results with mRNA). Therefore, we do not use BERT in this experiment. We

---

Table 4: Both GPT and LoReTTa are trained on disjoint (video, audio) and (video, text) pairs on MUGEN-GAME. We then evaluate the models on the unseen task of audio captioning using BLEU4, METEOR, and ROUGE. MMGPT's represent the upper-bound models.
Method
Train
Test
BLEU4
METEOR
ROUGE
GPT
A V.V. V.V +T
A + T
1.7
18.5
30.7
LoReTTa
A T
A + T
2.8
20.8
34.7
MMGPT
A + T V -
A + T - T
6.7
19.4
27.1
MMGPT
7.8
21.3
29.1
also avoid training a CLIP model, as this would require us to train a text diffusion model (or equivalent generator) on top of the pre-trained encoder, which neither end-to-end nor straightforward. For GPT, we train the model to generate video from audio and text from video. In this way, we force the model to implicitly learn how to caption audio tracks. In the case of LoReTTa, we model the transition between audio and text directly through transitive modeling. Remarkably, GPT trained on two bimodal datasets outperforms the audio-to-text baseline terms of ROUGE, showing strong positive transfer (Table 4). LoReTTa further improves GPT and even rivals the video-to-text baseline w.r.t METEOR and ROUGE. The BLEU4 values are low for both models. Since BLEU involves exact n-gram matching, these results are to be expected given the strict data constraints.
7 Discussion & Conclusion
With LoReTTa, we introduced an innovative self-supervised learning framework for multimodal integration that learns any missing joint conditional distribution given a linking modality. We demonstrated this theoretically and practically. In particular, our method combines the rules of commutativity and transitivity to model the multimodal datasets (A, and (A, B, C. given the two non-overlapping datasets (A, B) and (B, C.). In fact, we showed that the extracted features of transformer pre-trained with LoReTTa are very expressive for all modality combinations, including the unseen ones. While we only evaluated our algorithm on datasets with three modalities, it much more general because it can be used with any number of modalities and any mixture of modalities. As long as there a chain of aligned modalities in the dataset, LTReTTa is able to learn the transition (X; - - Xj).... ........ (X; + - X&) to model the missing distribution P(X,.... j.... Xx) (as shown in Figure 2d). To our knowledge, this scenario has never been considered in the literature without training ensemble of contrastive models based on the ideas of CLIP- most notably Wav2CLIP [68] and its extension ImageBind [26].
Broader impacts. We believe that LoReTTa will serve as a basis for training very powerful multimodal generative and discriminative models that will be useful in many areas of research. While our method itself cannot be misused, future models trained with LoReTTa could be exploited to generate harmful content of higher quality. However, LoReTTa was designed improve the quality of neural networks that contribute to society, for example by training more accurate multimodal systems that can help doctors and other practitioners in their daily work.
Environmental impacts. We trained all of our models on a single NVIDIA A A100-SXM4440GB GPU using PyTorch 2.0. Techniques such as mixed- precision training, FlashAttention and TF32 helped us reduce memory usage and runtime. Pre-training a model with causal masked modeling took us about 12 hours on average. On the other hand, fine-tuning with LoReTTa took longer about 3 days. This is because we had to autoregressively decode the predicted samples one token at a time.
Limitations. LoReTTa is best used in a setting where are missing some modality combinations. This only works if there is at least one connecting modality, otherwise, our method cannot be applied. But what about datasets where some samples have all modality combinations? Can our method still be used? Yes, this is possible. Although we have not analyzed this scenario, LoReTTa is likely to provide benefits by augmenting those data points that are still missing the remaining modalities.
Acknowledgements. M.T. is supported by the Helmholtz Association under the joint research school "Munich School for Data Science MUDS". There are other funding competing interests.
10

---

References
[1] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Cargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprini arXiv:2201.07520 2022. 3,4
[2] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Harnbardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed- modal language models. arXiv preprint arXiv:2301 033728, 2023. 8
[3] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel. Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv: 1325, 2023. 3,4
[4]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yama Hasson, Karel Lene, Arthur Mensch. Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Syxtems (NeurIPS), volume 35, pages
23716-23736, 2022.
[5] Mikel Artetxe, Jingfei Du, Naman Goyal, Luke Zettlemoyer, and Ves Stoyanov. On the role of bidirection- ality in language model pre-training. arXiv preprint arXiv:2205.11722 2022. 4,
[6] Akari Asai and Hannaneh Hajishirzi. Logic-guided data augmentation and regularization for consistent question answering In 58th Annual Meeting of the Association for Computational Linguistics, pages 5642-5650. Association for Computational Linguisties, 2020.
[7] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.006 00116007.0440... 2016. 4
[8] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in specch, vision and language. In 39th International Conference on Machine Learning (ICML), pages 1298-1312 PMLR, 2022. 3
Mohammad Bavarian, Hecwoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill the middle, 2022.
[10] Peng Bo. Blinkdl/rwkv-Imm 0.01, Aug 2021. URL https://doi 0rg/10..r 5281 5281/zenodo. 5196577. 4
[11] Marco Bolis, Daniela Bossi, Arianna Vallerga, Valentina Ceserani, Manuela Cavalli, Daniela Impellizzieri, Laura Di Rito, Eugenio Zoni. Simone Mosole, Angela Rita Elia, et al. Dynamic prostate cancer transcrip- tome analysis delineates the trajectory to disease progression. Nature Communications, 12(1):7033, 2021.
[12] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot Ieamerrs. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877-1901, 2020. 4
[13] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pages
IEEE, 2021. doi: 10.1109/1CC448922.0 2021.
00951. 3
[14] Lluis Castrejon, Yusuf Aytar, Carl Vondrick, Hamed Pirsiavash. and Antonio Torralba. Learning aligned cross-modal representations from weakly aligned data. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2940-2949, 2016.
[15] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning (ICML), pages 1691-1703. PMLR, 2020. 3,
[16] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021. 3
[17] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In 37th International Conference on Machine Learning (ICML), pages 1597-1607. PMLR, 2020. 3
[18] Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In 10th International Conference on Learning Representations (ICLR), 2022. 2
[19] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.00 Xiw32204.022111, 2022. 4

---

[20] Haotian Cui, Chloe Wang, Hassaan Maan. Kuan Pang. Fengning Luo, and Bo Wang. scept: Towards building a foundation model for single-cell multi-omics using generative ai. bioRxiv, 2023. 8
[21] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io- jo-awareness. In Advances in Neural Information Processing Systems (NeurlPS), volume 35, pages
2022. 4
[22] Grégoire Delétang, Anian Ruoss, Amuloiie Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, ct al. Language modeling is compression. arXiv preprint arXiv:2309. 10668, 2023. 2
[23] Jacob Devlin, Ming-Wet Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volune (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics, 2019. 3.
[24] Danny Driess, Fci Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Iehter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.033: 0333788 2023. 4
[25] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang. Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. In 11th International Conference on Learning Representations (ICLR), 2023. 4
[26] Rohit Girdhar, Alaacldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. arXiv preprint arXiv:2305. 05665,
2023. 3, 10
[27] Thomas Hayes, Songyang Zhang, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang. Songwei Ge, Qiyuan Hu, and Devi Parikh. Mugen: A playground for video-audio-ted multimodal understanding and generation. In 17th European Conference on Computer Vision (ECCV), pages 431-449 Springer, 2022. 8, 9
[28] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 30016-30030, 2022. 2,
[29] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning (ICML), pages 4651-4666 PMLR, 2021. 4
[30] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figumov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek. Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7833)5533-588)
2021. 20211 3
[31] Lyes Khacef, Laurent Rodriguez, and Benoit Miramond. Written and spoken digits database for multimodal learning. attps://zenonr
htt////eoorr//eee 2019. 7
[32] Pramnay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In 333th Advances in Neural Information Processing Systems (NeurIPS). pages 18661-18673, Curran Associates, Inc., 2020. 9
[33] Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing: System Demonstrations (EMNLP), pages 66 66-71, Association for Computational Linguistics, 2018, 4
[34] Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. Mnist handwritten digit database. http: //yann.lecun.
lecun. cwt/ wwtpc//c//m/aaceeeceeeeeee exdb//itt////// 2010. 7
[35] Kuang-Huei Lee. Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, lan Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. In 35th Advances in Neural Information Processing Systems (NeurlPS), pages 27921 -27936, 2022.
[36] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on a synthetic task. In 11th International Conference on Learning Representations (ICIR), 2023. 4,8 8
[37] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. 5
[38] Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language. arXiv preprint arXiv:2308.01399 2023, 4
12

---

[39] Ruixi Lin and Hwee Tou Ng. Does bert know that the is-a relation is transitive? In 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 94-99, Association for Computational Linguisties, 2022.
[40] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. Al Open, 3: 111-132, 2022. 2
[41] Zeming Lin, Halil Akin, Roshan Rao, Brian Hic, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel- -arrndi, Tom Scrcu, Sal Candido, et al. Language models of protein sequenees at the scale of evolution enable accurate structure prediction. BioRxiv, 2022. 3
[42] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint /907. / 1692, 2019. 3
[43] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computa- tion engines. arXiv preprint arXiv:2103.05247 1, 2021. 2
[44] Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence. pages 2302-22-2202 2310, 2021. 1
[45] Mengmeng Ma. Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 18177-18186, 2022.
[46] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303008774, 2023. 3
[47] Hai Pham, Paul Pu Liang. Thomas Manzini, Louis-Philii Phillipe Morency, and Barnabás Póezos. Found in translation: Learning robust joint representations by eyclic translations between modalities. In 33th AAAI
Conference on Artificial Intelligence, pages 6892-6899. AAAI, 2019, 3,5
[48] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866 2023. 4
[49] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, 2018.
[50] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfcrable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748-8764 PMLR, 2021.
[52] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslamides, Sarah Henderson, Roman Ring, Susannah Young, al. Sealing language models: Methods, analysis & insights from training gopher, 2021.
[53]
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to text transformer. The Journal of Machine Learning Research (JMLR), 21(1):5485 -55554 2020. 4
[54] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss. Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning
(ICML). pages 8821-8831. PMLR, 2021. 3, 3.4.8 8
[55] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019. 4
[56] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez. Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg. Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research (TMLR), 2022.
ISSN 2835-8856. 4,8
[57] Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5149-5152, IEEF, 2012. 4
[58] Rico Semnrich, Barry Haddow, and Alexandra" Birch. Neural machine translation of rare words with subword units. In 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1715-1725. Association for Computational Linguistics, 2016. 4
[59] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. U12: Unifying language learning paradigms. In 11th International Conference Learning Representations (ICLR), 2022. 3,4
13

---

[60] Zack Thoutt. Wine reviews: 130k wine reviews with variety, location, winery, price, and description. https://wwet wwwgle..ggge...m httrddaaaaatttttaiiiieeeeeeeeeee
2018. 7
[61] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023. 4
[62]
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NewrIPS), volume 30, 2017. 4
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In 31st Conference on Neural Information Processing Systems (NIPS), page 6000-6010. Curran Associates, Inc., 2017. 2,4
[64] Alex Wang and Kyunghyun Cho. Bert has mouth, and must speak: Bert a markov random field language model, arXiv preprint arviv: Aafiv:992.0000 1902.04094 2019. 9
[65] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Muchine Learning (ICML), pages 22964-22984.
PMLR, 2022.
[66] Wenhui Wang. Hangbo Bao, Li Dong. Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. Image as foreign language: Beit pretraining for all vision and vision-language tasks, arXiv preprint arXiv:2208. 10442, 2022. 3
[67] John N Weinstein, Eric A Collisson, Gordon B Mills, Kenna R Shaw, Brad A Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, and Joshua M Stuart. The cancer genome atlas pan-cancer analysis project.
Nature Genetics, 45(10):1113-- 1120, 09 2013. 7
[68] Ho- Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4563-4567. IEFF, 2022. 10
[69] Hitomi Yanaka, Koji Mineshima, and Kentaro Inui. Exploring transitivity in neural nli models through veridicality. In 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 920-934. Association for Computational Linguistics, 2021. 5
[70] Fan Yang, Wenchuan Wang. Fang Wang. Yuan Fang. Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert large-scal pretrained deep language model for cell type annotation of single-cell ma-seq data, Nature Machine Intelligence, 4(10)852 866, 2022. 8
[71] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov. and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems (NeurlPS), 32, 2019. 3
[72] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022. 3
[73] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang. Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan. Oron Ashual, Uriel Singer. Shang- Wen Li. Susan Zhang, Gargi Ghosh, Yaniv Taigman, Maryam -Zrrand,,,,, Asli Celikyilmaz, Luke Zettlemoyer, and Armen Aghajanyan. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. Technical report, Meta Al, 2023. 4
[74] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[75] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to- to-image translation using cycle-consistent adversarial networks. In IEFE International Conference on Computer Vision (ICCV), pages 2223-2232, 2017. 5
14


Reviews:
[{"review_id": "cmEsiRki68", "reviewer": "Reviewer_a22R", "summary": "This paper proposes a self-supervised learning framework, LoReTTa (Linking mOdalities with a tRansitive and commutativE pre-Training sTrAtegy), for multimodal learning with not-aligned multimodal datasets. To be specific, LoReTTa (i) first trains an autoregressive model for generating A->B and B->C, and then (ii) generates pseudo targets for A->C using the model. Then, it (iii) performs autoregressive modeling again. This paper demonstrates that this transitive modeling can improve generation performance between the unseen pair of modalities (i.e., A->C).\n", "questions": "1. About causal masked modeling (L140-L143)\n   - If randomly mask some tokens and move them to the end, then such a prediction task is somewhat different from the original masked language modeling (like BERT) since it loses positional information about the masked tokens. Is the approach ok? I also think it may interfere with the causal language modeling.\n2. In Equation (3), what is $L_X$?\n\n", "limitations": "This paper has addressed the limitations and the potential impacts.\n", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper is well-motivated.\n- The proposed framework is widely applicable.\n- The proposed framework outperforms baselines that do not consider transitive modeling.\n", "weaknesses": "- Limited methodological novelty\n  - The proposed framework is mainly based on the idea of CycleGAN, and it simply uses causal language/mask modeling with Transformers. I agree that the idea is more general, but I feel some lack of novelty. I'm also concerned about the scalability of the framework.\n  - In addition, due to the recent advance of foundation models (e.g., Llama2 [1], DINOv2 [2]), there are many research attempts to utilize them for multimodal learning. For example, Chameleon [3] uses expert models for multi-modalities (see [4] for a survey on multimodal LLMs). I think the authors should discuss what is the advantage of the proposed framework over the approach utilizing one foundation model as a general knowledge hub.\n- Weak theoretical analysis (Section 4)\n  - The analysis is too vague. Any theoretical statement is not stated concretely. Also, the authors should discuss the generalization error bound because, for example, the sample (C0...C3) in Figure 2c is not observed.\n  - Notations are somewhat confusing. Why do the authors state the definition (3)? $f(x_1|x_2)=x_1+e$ is a mathematically correct notation? Many notations are roughly written, so it may cause confusion to the readers.\n  - Section 4 should be reorganized.\n- Weak experiments\n  - In this topic, large-scale experiments and diverse applications are important. However, I think the provided experiments are very small-scale, e.g., MNIST images. I suggest using more challenging benchmarks, e.g., [5-6].\n\n[1] Llama 2: Open Foundation and Fine-Tuned Chat Models \\\n[2] DINOv2: Learning Robust Visual Features without Supervision \\\n[3] Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models \\\n[4] A Survey on Multimodal Large Language Models \\\n[5] Multi-modal Dense Video Captioning, CVPR Workshop 2020 \\\n[6] MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration, 2022\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer a22R,**\n\nWe appreciate your valuable comments on our paper. Your recognition of the motivation and broad applicability of our proposed framework is very encouraging. Below, we would like to address the open questions.\n\n**Limited methodological novelty.** In the global rebuttal, we pointed out that LoReTTa is much more general and applicable than CycleGAN [Zhu et al., 2017]. We used CycleGAN as an analogy because it has a similar flavor to our method. But beyond the high-level conceptual similarity, our transitive modeling approach is fundamentally different. For example, it can be used to transition across multiple modalities – not just two domains from the same modality. We use a single transformer as the central point for all modalities. By combining causal, masked, commutative, and transitive modeling, we have shown theoretically and experimentally that our model can meaningfully integrate information from different modalities. This is in contrast to recent approaches that use LLMs as a communication hub to explain and reason about one modality. For example, Chameleon [Lu et al., 2023] uses different modalities in a plug-and-play approach to describe images, process spreadsheets, or search the web. However, this approach does not effectively integrate different modalities to potentially gain new insights. In our TCGA-OMICS experiments, we have shown that combining genomic, transcriptomic, or proteomic data provides valuable prognostic results. Adding each modality individually to a foundation model is not enough, as they do not interact with each other. Thus, LoReTTa serves as a good starting point for fine-tuning these general models. One could also think about using LLMs that communicate with LoReTTa via API calls to solve more challenging problems that a user faces with their multimodal input.\n\n**Weak theoretical analysis.** Our theoretical analysis is supported by references and highlights the probabilistic principles behind our approach. It also discusses the error propagation caused by transitive modeling and how we mitigate this by modeling the omitted modality. We use standard notation from stochastic, analysis, and linear algebra to emphasize the intricate details that arise. Any confusion caused by this will be addressed in an updated version of the manuscript. Regarding the unobserved samples $(C0, ..., C3)$, they are indeed missing, but the goal of LoReTTa is precisely to impute this missing modality and align it with the existing data points. It corresponds to the predicted sample $\\hat{x}_3 = x_3 + e$ in Equation 6 for which we have provided the error bounds. \n\n**Weak experiments.** We agree that large-scale experiments and diverse applications are very important. That is why we have provided results on a real medical dataset for the crucial task of survival prediction. Our results show that the proposed approach works not only on synthetic data but also on complex biomedical samples. We are very grateful to the Reviewer for suggesting the MUGEN [Hayes et al., 2022] dataset. It is indeed a challenging and large-scale benchmark. After hearing about it, we immediately downloaded the dataset and started training. We summarized the results in the global rebuttal, which shows that LoReTTa works on even more problems and is easily scalable. These results will be included in the final version of the manuscript.\n\n**New results**. The table below shows the results of our new experiments. Both GPT and LoReTTa were trained on disjoint (video, audio) and (video, text) pairs to solve the problem of cross-modal translation. We then evaluated the models on the unseen task of audio captioning. As can be seen, LoReTTa is more than capable of overcoming the modality gap and rivals the upper-bound models (MMGPT). More details can be found in the global rebuttal.\n\n| Method $\\phantom{.}$ | $\\phantom{...}$ Train | $\\phantom{a}$ Test | BLEU4 | METEOR | ROUGE |\n|----------|----------|----------|----------|----------|----------|\n| GPT | A $\\rightarrow$ V, V $\\rightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T |  $\\phantom{...}$ 1.7 |  $\\phantom{...}$ 18.5 | $\\phantom{...}$  30.7 |\n| LoReTTa | $\\phantom{.}$ A $\\leftrightarrow$ V $\\leftrightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T | $\\phantom{...}$ 2.8 |  $\\phantom{...}$ 20.8 |  $\\phantom{...}$ 34.7 |\n|     |     |     |     |\n| MMGPT | $\\phantom{...}$ A $\\rightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T | $\\phantom{...}$ 6.7 | $\\phantom{...}$ 19.4 | $\\phantom{...}$ 27.1 |\n| MMGPT| $\\phantom{...}$ V $\\rightarrow$ T |$\\phantom{.}$  V $\\rightarrow$ T | $\\phantom{...}$ 7.8 | $\\phantom{...}$  21.3 | $\\phantom{...}$ 29.1 |\n\n\n**References**\n\nHayes et al., MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration, 2022\n\nLu et al., Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models, 2023\n\nZhu et al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, 2022"}, {"review_id": "QY8YRaZpc9", "reviewer": "Reviewer_u4o4", "summary": "The paper proposes a new self-supervised learning method to pretrain from multiple modalities. More specifically, the paper argues that it is hard to obtain datasets with 3 aligned modalities (e.g., text, images, and speech) and tries to address this issue. This can be especially cumbersome for medical datasets where some modalities can be missing for certain patients. As a result, this work proposes a self-supervised framework based on (causal) masked modeling, commutativity, and transitivity to go from one modality to another. These can be regarded as additional consistencies that are imposed during the pretraining stage. While the pretraining only has seen disjoint modality pairs, the model is able to handle any modality combination at test time. The evaluation considers the perplexity metric and also the classification score for unseen modality pairs during the pretraining stage.", "questions": "Please address the questions and issues raised in the Weaknesses section.", "limitations": "The paper briefly discusses the limitations, environmental impact and broader impact at the end of the paper.", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- Focus: The premise and focus of the paper is valuable as it addresses a real-world problem in order to pretrain on multiple modalities. The paper starts with the argument that it is  hard to collect perfectly aligned datasets when aiming to pretrain on multiple modalities. This alignment step is indeed a very difficult procedure that can cost a significant amount of time and money. Furthemore, in some cases, aligning the data might not even be possible due to missing information/modalities, such as in medical datasets. For instance, the paper considers mRNA, miRNA, and RPPA samples from the TCGA dataset (a medical dataset) when a certain modality is missing. This could be useful in practice.\n\n\n- Method: To address the aforementioned issue, the approach relies on causal masked modeling and cycle consistency, which are powerful tools that I have not seen being combined.\n- Style: The paper is well-written and easy to follow. The overview figure (see Figure 2)  is especially valuable to quickly grasp the aim and mechanics of the presented approach. \n\n\n- Experiments: The experiments show that the approach can improve over strong baselines, such as BERT and GPT (see Table 3) and can leverage multiple modalities when some might be missing. Advantageously the presented method can leverage multiple modalities as these simply need to be tokenized, such as audio, images, and speech.", "weaknesses": "- The scale and scope of the experiments is limited: As the paper considers pretraining, the paper only considers two datasets with a fairly small amount of samples. For instance the SVL-MNIST and TCGA-OMICS datasets are small-scale datasets with respectively 12k and 3.5k sample pairs. Furthermore, only limited finetuning experiments  are considered. What about other tasks, such as retrieval or (visual) question answering? Competing works such as CLIP and BERT have shown to work well for these tasks as well. Overall, it is difficult to judge the generality of the approach as the scope of the experiments is limited. \n\n- The originality of the approach is fairly limited as the approach can be seen as a combination of ideas in prior works. For example, the causal masking strategy is similar to [1, 28, 11], and the transitive loss is similar to cycle consistency [90]. While the paper mentions modification to these works in the paper (L141 and L153), it is not clear how significant these contributions are to the final performance. More information would be useful.\n\n- It can be observed from Table 3 that CLIP already obtains relatively good results compared to the proposed method. More information on why this is the case would be useful. While CLIP requires the pretraining of 3 different encoders, as the paper mentions, I don’t think this is a strong argument. Especially, as the paper under review also relies on expensive ways to compress high-dimensional data with a VQVAE (see L116). More information on the computational cost  when comparing the approach to CLIP would be valuable during the rebuttal.\n\n- There is currently no experiment that includes all modalities during pretraining. This could be seen as an upperbound but is currently missing as mentioned in the limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer u4o4,**\n\nThank you for pointing out that our proposed method improves on strong baselines such as BERT and GPT by combining powerful ideas and applying them to real-world problems. In the comments below, we address each of the remaining concerns that were raised.\n\n**Scale and scope of the experiments.** We chose SVL-MNIST to perform extensive experiments with a limited computational budget. In total, we have trained 12 models and refined them 162 times on this dataset alone. This does not even include the search for hyperparameters. We complemented this with experiments on a real medical dataset for the important task of survival prediction. In both cases, LoReTTa works as theorized in Section 4. To assure the Reviewer that our method also extends to larger datasets, we performed an additional experiment (cross-modal translation) on the MUGEN [Hayes et al., 2022] dataset with 375,000 aligned video, audio, and text samples. We refer to the global rebuttal for the results, which show that LoReTTa is generalizable and scales very well to larger datasets and other tasks.\n\n**Originality of the approach.** We noted in the paper that causal modeling and masked modeling are two of the current state-of-the-art methods for self-supervised learning. Both methods had been combined into a strategy called causal masked modeling [Aghajanyan et al., 2022; Bavarian et al., 2022; Fried et al., 2023], which we used as a starting point and never claimed as our invention. However, we greatly improved causal masked modeling by extending it with commutative modeling (C2M3) and transitive modeling (LoReTTa), which had never been done before. By comparing LoReTTa with C2M3, GPT [Radford et al., 2018], and BERT [Devlin et al., 2018], we indirectly included an ablation study. In fact, transitive modeling is a very unique contribution. On the surface, it shares similarities with cycle consistency [Zhu et al., 2017]. But despite this high-level conceptual relationship, the two approaches work quite differently. For example, CycleGAN is only able to transition between two domains (image styles) of the same modality (images), while LoReTTa works on three or more modalities, e.g. image, video, audio, and text, with complex multimodal relations between them. This also helps to avoid representation collapse, which is common in adversarial training. We have given a more detailed explanation in the global rebuttal.\n\n**About CLIP results.** LoReTTa consistently outperforms CLIP [Radford et al., 2022]  in 5 out of 7 modality combinations (see paper) by a large c-index. Only in one case (R), CLIP outperforms LoReTTa and in another (I, R) it is equal. In both cases, the reverse-phase protein array (R) is included as a modality. For GPT, BERT, C2M3, and LoReTTa, the combination (I, R) leads to a worse prediction than I or R alone. Only for CLIP the c-index increases. We assume that I and R lead to negative transfer, but since CLIP learns to create similar embeddings for both, the embedding vectors may be devoid of features causing the problem. This could be considered a kind of smoothing or regularization. However, this could also cause the model to ignore important high-frequency features inherent in each modality, leading to worse performance as seen for the other modality combinations. Another issue with self-supervised contrastive learning is that two samples with the same label might be pushed away in the embedding space. For example, a batch might contain an image of a dog with a corresponding caption and another image of a similar dog with another similar caption. The CLIP loss only learns to pull the feature vector of each pair closer but pushes each sample away. Thus, contrastive learning requires very large batch sizes and huge amounts of training data [Chet et al., 2020]. In addition, each modality requires its own encoder. LoReTTa, however, already works on smaller datasets which is a huge advantage for data-deprived domains like medicine. In addition, it can reuse existing and pre-trained tokenizers.\n\n**Upper bound.** We did not include experiments for a model trained with all three modalities. This was done to emphasize the real-world problem that often there is simply no dataset with all three modalities at once. The upper bound is the current state-of-the-art for existing datasets, which LoReTTa manages to outperform. We have focused our resources on providing extensive experiments for this setting. In the global rebuttal, we have also included an even larger experiment on a new dataset [Hayes et al., 2022] to show the usefulness of LoReTTa on even more problems.\n\n**New results.** We have added the additional large-scale experiment in the global rebuttal. Both GPT and LoReTTa were trained on disjoint (video, audio) and (video, text) pairs to solve the problem of cross-modal translation. We then evaluated the models on the unseen task of audio captioning. As can be seen above, LoReTTa is more than capable of overcoming the modality gap and rivals the upper-bound models (MMGPT).\n\n\n**References**\n\nAghajanyan et al., CM3: A Causal Masked Multimodal Model of the Internet, 2022\n\nBavarian et al., Efficient Training of Language Models to Fill in the Middle, 2022\n\nChen et al., A Simple Framework for Contrastive Learning of Visual Representations, 2020\n\nDevlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018\n\nFried et al, InCoder: A Generative Model for Code Infilling and Synthesis, 2023\n\nHayes et al., MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration, 2022\n\nRadford et al., Improving Language Understanding by Generative Pre-Training, 2018\n\nRadford et al., Learning Transferable Visual Models From Natural Language Supervision, 2022\n\nZhu et al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, 2022"}, {"review_id": "w79suT5J0a", "reviewer": "Reviewer_QEx4", "summary": "In this paper, the authors introduce the problem of learning from multi-modal data (e.g. modalities A, B, and C) if not all modality combinations are available at training time (e.g. only paired data (A, B) and (B, C) are given). This constitutes a highly relevant research problem, as learning from multiple modalities has shown a lot of potential, but collecting fully annotated multi-modal datasets is a costly endeavour.\n\nIn order to take advantage of _existing_ data and modality combinations, the authors propose a self-supervised learning paradigm (LoReTTa), which combines cycle consistency with masked modelling: i.e., the models are trained to predict masked tokens in A from B (masked modelling) or C from A via B (masked modelling with cycle consistency). As a result, the underlying model is able to perform well not only on the seen modality combinations (A, B) and (B, C), but is also able to handle unseen combinations such as (A, C) or (A, B, C).\n\nIn particular, the authors show on a synthetically created data pairing (A=MNISt + B=AudioMNIST + C=WineReviews) that a model trained with cycle consistency (LoReTTa) achieves significantly lower perplexity and higher linear probing accuracy on the test data than models trained without such cycle consistency. Additionally, the authors evaluate their approach on the real-world TCGA-OMICS dataset and report consistent performance improvements, especially on unseen modality combinations.", "questions": "Please see the weakness section. If the authors are able to address my concerns, I will gladly update my score.", "limitations": "The authors have discussed societal impacts and limitations of their work. Nonetheless, as discussed in the weakness section, I believe the paper could further benefit from an increased discussion and comparison to related works. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "This paper is a good submission for the following reasons.\n\n- S1: The authors introduce a novel and highly relevant setting for learning from multiple modalities.\n- S2: The proposed approach is well-motivated and simple yet effective.\n- S3: The presentation and writing make this paper a pleasure to read.\n- S4: The experimental results show convincing and significant improvements over the chosen baseline models (however, see W1).", "weaknesses": "While I find this to be a good submission in general, there are several points of concern that make me hesitant. Specifically, I would highly appreciate additional feedback from the authors on the following aspects.\n\n- W1 (baselines and evaluation): The two main novel contributions of this paper seem to be (1) introducing the problem of learning from multi-modal data with disjoint modality pairings, and (2) showing that cycle consistency can be highly beneficial in this context. In order to validate (2), the authors compare to backbones without this cycle consistency in Tables 1+2, and to other training paradigms (contrastive loss, CLIP) or that use different model architectures without cycle consistency (different attention masks in GPT and BERT) in Table 3. While C2M3 seems to be a strong contender among the models without cycle consistency (Table 3), the other baselines yield competitive results, too. __My question in this context is thus the following__: why do the authors only include the cycle consistency in the C2M3 model? In principle, cycle consistency could also be integrated into BERT and GPT. It seems to me like the submission could be strengthened by showing that all of the baselines benefit from the addition of cycle consistency. \n- W2 (causal modelling): Following from W1: what is the motivation to focus on causal masked modelling in the first place? In the context of multi-modal learning, I fail to see why a causal attention structure is desired. Even more, only due to the causal modelling approach does the need for commutative switching arise, which the authors introduce as an important contribution. If the authors were to apply the cycle consistency approach to BERT e.g., this would not be necessary. \n- W3 (structure of the paper): Especially in light of W1 and W2, the space usage in the paper seems to be suboptimal to me. E.g., half a page in the method section is spent on the model architecture and tokenization options, which seem to be experimental details rather than essential aspects of the proposed method. Similarly, the definition and discussion of what constitutes a modality seems to be tangential, yet is given another 1/2 page in section 4. More generally, while interesting to read, the relevance of section 4 and its integration with the rest of the paper are not fully convincing to me. Instead, I think the paper would benefit from an extended experimental evaluation, including additional comparisons to other methods under the addition of cycle consistency (see W1) and potentially additional datasets and methods (e.g., as in [4]), which would allow for a more fine-grained understanding of the benefits and limitations of the proposed method (e.g., how much does it help if full modality combinations are in fact, at least sometimes, available?)\n- W4 (reproducibility), following from W3: While the authors spend a significant amount of space on describing tokenization strategies, it remains unclear to me when and how e.g. VQ-VAEs or CNNs are used, as the authors only mention the usage of PCA in the experimental details. With the current amount of information given, I am concerned about the reproducibility, as the final model architectures and the tokenization (where do CNNs and VQ-VAEs enter the picture) remain unclear to me.\n\n\n\nAdditional minor remarks:\n- In the light of the strong results reported in ImageBind [29], it would be interesting to include additional discussion on how LoReTTa conceptually compares to CLIP (bringing 'transitivity' to masked modelling, which is somewhat inherent in CLIP?), how LoReTTa might perform in such a large scale setting, and why CLIP might perform suboptimally in the reported results (Table 3). Does CLIP simply require more data? \n- The authors argue that their transitivity approach is 'much more general' than cycle consistency (see caption Fig. 2, or line 153). I currently fail to see the difference, however, and would appreciate if the authors could elaborate.\n- Line 313: 'communicative switching' -> do the authors mean commutative?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer QEx4,**\n\nWe are very grateful for the thorough review and appreciate that our proposed method is considered novel and highly relevant, showing significant and consistent improvements across different baselines. While we have addressed many of the Reviewer's concerns in the global rebuttal, we provide a customized response below.\n\n**W1 (baselines and evaluation).** We refer to our approach as transitive modeling instead of cycle consistency [Zhu et al., 2017] because, upon closer inspection, the two are quite different. CycleGAN was proposed to transition between two domains (image styles) of the same modality (image), while LoReTTa can transition between three or more modalities (i.e., image, text, and audio). It also avoids using the input as a target, which could cause the representation to collapse. Furthermore, our approach works with more complex modality relations, as described in the global rebuttal. All this makes transitive modeling much more general and applicable than cycle consistency – with which it again shares only some high-level conceptual similarities.  By design of transitive modeling, the missing modality must be generated. This is not possible with BERT [Devlin et al., 2018]. It is trained with masked modeling, which simply predicts the missing tokens, not a completely missing sequence on the right. On the other hand, GPT-style [Radford et al., 2018] models are generative and can be extended with our proposed transitive modeling approach.  In particular, we start with CM3, which improves GPT by introducing causal masked modeling [Aghajanyan et al., 2022; Bavarian et al., 2022; Fried et al., 2023]. This effectively combines BERT and GPT into a single model. We extend CM3 with commutative (C2M3) and transitive (LoReTTa) modeling.  Thus, with LoReTTa we have a unified architecture that includes transitive modeling in both GPT and also BERT. The latter would not be possible otherwise, as explained above.\n\n**W2 (causal modeling).** To use transitive modeling, we must predict the missing modality by generating it in its entirety. This requires a generative model, which is only possible with causal training in a sequential model. However, we are aware of the advantages of bidirectional attention in BERT over causal masks in GPT. Therefore, we use a unified version called causal masked modeling to take advantage of both. This is our way to use transitive modeling in BERT since it is not intended to be used for generative tasks.\n\n**W3 (structure of the paper).** We provide an additional large-scale experiment on the MUGEN [Hayes et al., 2022] dataset with 375,000 aligned samples from video, audio, and text samples to make our empirical findings even more comprehensive. In particular, we apply LoReTTa to the task of cross-modal translation (the task is analogous to visual question answering). This generative task is not possible with the masked (BERT) and contrastive (CLIP) approaches. Therefore, we compare it only with GPT and its commutative (C2M3) and transitive version (LoReTTa). Thanks to the Reviewer's suggestions, we will restructure our paper to make room for this additional experiment.\n\n**W4 (reproducibility).** We briefly mentioned our tokenization scheme in the methods and results sections. Since the MNIST dataset contains low-dimensional samples, we encoded each byte stream as a token. For the high dimensional TCGA dataset, we used PCA for dimensionality reduction, and bin each dimension into tokens. For the new dataset, we used pre-trained video and audio VQ-VAE encoders provided along the dataset. We will emphasize this more in the final version. \n\n**Comparison to CLIP and ImageBind.** The advantage of LoReTTa is that it can be used as both an encoder and a decoder to solve both discriminative and generative problems, as demonstrated in our old and new experiments. CLIP [Radford et al., 2022] and its extension ImageBind [Girdhar et al., 2023] (published shortly before the NeurIPS 2023 submission deadline) train their models primarily as encoders – one for each modality. For generative tasks, a diffusion model or other variants must be fitted to these embeddings. Thus, contrastive methods must train multiple encoders and decoders. Worse, contrastive training is inefficient. It requires large batch sizes and large datasets [Chet et al., 2020]. This is because self-supervised contrastive loss only clusters embeddings from the same sample and pushes away those from other samples. Consider a batch with an image of a cat and an accompanying text. In the same batch, there is another image of a similar cat with a similar text description. Now, the CLIP loss will individually push the feature vectors of the first and second image-text pairs closer together. But at the same time, it will push the representations of each sample away from each other, regardless of their similarity.\n\n**New results.** Please have a look at the global rebuttal, where we have added the additional large-scale experiments.\n\n\n**References**\n\nAghajanyan et al., CM3: A Causal Masked Multimodal Model of the Internet, 2022\n\nBavarian et al., Efficient Training of Language Models to Fill in the Middle, 2022\n\nChen et al., A Simple Framework for Contrastive Learning of Visual Representations, 2020\n\nDevlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018\n\nFried et al, InCoder: A Generative Model for Code Infilling and Synthesis, 2023\n\nGirdhar et al., ImageBind: One Embedding Space To Bind Them All, 2023\n\nHayes et al., MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration, 2022\n\nRadford et al., Improving Language Understanding by Generative Pre-Training, 2018\n\nRadford et al., Learning Transferable Visual Models From Natural Language Supervision, 2022\n\nZhu et al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, 2022\n"}, {"review_id": "acBNdpR3gB", "reviewer": "Reviewer_7Goz", "summary": "This paper aims to remedy the problem of training a mobility to perform well on any combination of modalities in the training data, regardless of if these combinations appear in the training data. This is done through commutative and transitive pretraining: the former allows for the model to learn modality A from B and B from A, and the latter learns the missing joint distributions. This results in a model which outperforms models trained on fewer modalities in the pre-training stage or models trained without transitive modeling.", "questions": "see above", "limitations": "The main limitation I think is missing is the amount of compute required.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "This paper addresses an important and under-addressed problem of training with mismatching modality pairings, and their commutative and transitive modeling is proven to be very affective on training in both toy and real datasets. While I do not keep up closely with the literature in many-modality pretraining, if this is the first paper to address this problem of training on mismatching modalities, it is incredibly valuable to the community and would be a good baseline for any future methods due to its simplicity.", "weaknesses": "My main concerns are surrounding the writing, as I found the methods section to be quite wordy and I think you could remove a lot of pretext on transformers and existing pretraining strategies. Furthermore, possibly due to the density of the method section, I found the explanation of the perplexity to be a bit confusing. \n\nAside from the writing, I wonder what the difference in compute requirements is between the authors method and the baselines. It was mentioned that LoReTTa was trained on on A100 for 3 days, which seems like a large amount of compute for the datasets presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer 7Goz,**\n\nWe sincerely appreciate your critical insights and expertise during the review of our manuscript. We are greatly encouraged by your recognition that our work addresses an important and understudied problem. Such recognition underscores the relevance of our research and further motivates us. Below, we would like to address your two concerns in detail.\n\n**Writing.** Our proposed learning algorithm covers a wide area of research including transformers, computer vision, natural language processing, audio signal processing, computational pathology, self-supervised learning, and multimodal learning. We sought to appropriately contextualize our method within the existing literature and bring readers from a wide variety of backgrounds on the same page, as we believe that LoReTTa will have a profound impact in a wide variety of fields. Still, we appreciate your concern and will make the related work and methods section even more precise.\n\n**Compute requirements.** LoReTTa relies on the auto-regressive generation of the missing modality for transitive training. This is a slow process for transformer decoders. However, we are happy to report that our community has made great progress in reducing computational complexity. Recently, FlashAttention-2 [Dao, 2023] has been announced, which doubles the speed of FlashAttention [Dao et al., 2022]. Multi-query attention [Shazeer, 2019] and grouped-query attention [Ainslie et al., 2023] have been shown to reduce inference time by a factor of up to 6. In addition, speculative sampling [Chen et al., 2023] achieves a 2-2.5x decoding speedup. The combination of all these advances promises to significantly reduce the training time of LoReTTa. We have omitted them to avoid too many moving parts in our experiments that would distract from the main results. In the global rebuttal, we presented the results of an additional experiment on a large-scale dataset [Hayes et al., 2022] with 375,000 samples. We were able to achieve excellent results with LoReTTa with a limited time budget of just a few days.\n\n**New results**. The table below shows the results of our new experiments. Both GPT and LoReTTa were trained on disjoint (video, audio) and (video, text) pairs to solve the problem of cross-modal translation. We then evaluated the models on the unseen task of audio captioning. As can be seen, LoReTTa is more than capable of overcoming the modality gap and rivals the upper-bound models (MMGPT). More details can be found in the global rebuttal.\n\n| Method $\\phantom{.}$ | $\\phantom{...}$ Train | $\\phantom{a}$ Test | BLEU4 | METEOR | ROUGE |\n|----------|----------|----------|----------|----------|----------|\n| GPT | A $\\rightarrow$ V, V $\\rightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T |  $\\phantom{...}$ 1.7 |  $\\phantom{...}$ 18.5 | $\\phantom{...}$  30.7 |\n| LoReTTa | $\\phantom{.}$ A $\\leftrightarrow$ V $\\leftrightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T | $\\phantom{...}$ 2.8 |  $\\phantom{...}$ 20.8 |  $\\phantom{...}$ 34.7 |\n|     |     |     |     |\n| MMGPT | $\\phantom{...}$ A $\\rightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T | $\\phantom{...}$ 6.7 | $\\phantom{...}$ 19.4 | $\\phantom{...}$ 27.1 |\n| MMGPT| $\\phantom{...}$ V $\\rightarrow$ T |$\\phantom{.}$  V $\\rightarrow$ T | $\\phantom{...}$ 7.8 | $\\phantom{...}$  21.3 | $\\phantom{...}$ 29.1 |\n\n\n**References**\n\nAinslie et at., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023\n\nChen et al., 2023 Accelerating Large Language Model Decoding with Speculative Sampling, 2023\n\nDao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022\n\nDao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, 2023\n\nHayes et al., MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration, 2022\n\nShazeer, Fast Transformer Decoding: One Write-Head is All You Need, 2019\n"}, {"review_id": "DEzzeB9SMj", "reviewer": "Reviewer_FSDR", "summary": "This paper proposes a new method LoReTTa (Linking mOdalities with a tRansitive and commutativE pre-Training sTrAtegy) to address the setting where not all modality pairs are available during training and inference. Concretely, it leverages casual masked modeling and transitive modeling to accomplish the objective. Theoretical analysis has been provided to help the understanding of the proposed method, with empirical results presented as well.  ", "questions": "Please refer to the weaknesses. ", "limitations": "None. ", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The target problem is practical and meaningful, which may indeed occur in our real implementation of multimodal algorithms. \n\n+ The proposed method is simple and easy-to-follow. \n\n+ Theoretical analysis contributes to the understanding. ", "weaknesses": "I am not an expert in this field, but I find some issues that might affect the quality of this manuscript. \n\n-  The proposed method seems a bit tricky; I am concerned whether this is up to the standard of NeurIPS. And the authors should implement ablation studies to verify the effectiveness of each proposed technique. \n\n- In transitive modeling, we need to use a \"predicted\" modality to bridge the missing modality pair. However, since the prediction can not be that accurate, how can we ensure robustness against such noise?  \n\n- The experimental datasets are sort of small (e.g. SVL-MNIST dataset). It would be better if experiments are conducted on more datasets or even larger ones. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer FSDR,**\n\nIt is a great pleasure to hear that our work addresses a practical and meaningful problem. Our proposed algorithm is indeed simple but has a big impact. We demonstrated this by applying our novel method to a real medical problem. The results we saw were fully consistent with our theoretical analysis of the mechanisms behind LoReTTa. In the following sections, we would like to address the remaining concerns.\n\n**Ablation studies.** We have not forgotten about the ablation studies. They are critical to understanding which component contributes to which outcome. The two new concepts we have introduced are commutative modeling and transitive modeling. We applied commutative modeling to CM3 [Aghajanyan et al., 2022] to get C2M3. We then extended C2M3 with transitive modeling – resulting in LoReTTa. CM3 and its variants [Aghajanyan et al., 2022; Bavarian et al., 2022; Fried et al., 2023] are themselves extensions of GPT [Radford et al., 2018] and BERT [Devlin et al., 2018]. Thus, an ablation study for causal masked modeling has already been done in the literature (they show noticeable improvements).  Nevertheless, we used all of these models as baselines in our real-world experiment on TCGA-OMICS, comparing GPT, BERT, C2M3 (GPT + BERT + Commutative Modeling), LoReTTa (C2M3 + Transitive Modeling), as well as CLIP  [Radford et al., 2022]. Hence, our benchmark automatically included an ablation study. Since C2M3 is already a strong improvement over GPT, BERT, and CLIP, we only compared LoReTTa with C2M3 in our SVL-MNIST experiments, where we extensively studied the effect of transitive modeling under different scenarios.\n\n**Robustness against noise.** LoReTTa predicts the missing modality to bridge the modality gap. This only works if the prediction is accurate. Otherwise, we would be training with noisy samples. We ensure this by pre-training our transformer with C2M3 until convergence. It learns to effectively transition between all seen modality pairs through causal modeling, within modalities through masked modeling, and vice versa through commutative modeling. With this strong generative model, we are able to faithfully infer the missing modality. We further reduce the noise by adding another level of alignment: using the predicted modality to infer the omitted sample (see Figure 2 for more details).\n\n**Large-scale experiments.** We chose SVL-MNIST to pre-train 12 models and fine-tune them a total of 162 times. This allowed us to gain valuable insights into the inner workings of transitive modeling with a limited computational budget. As a real-world application, we chose the medical domain, which is known to lack large datasets for training foundation models. It is still worth noting that LoReTTa already gives impressive results on a smaller medical dataset with highly complex modalities such as genomics, transcriptomics, or proteomics. Nevertheless, it is interesting to see how well LoReTTa scales to larger experiments. Thanks to the valuable feedback we received during the rebuttal, we were directed to a new multimodal dataset with 375,000 samples from a real-world vision-language-audio problem [Hayes et al., 2022]. We immediately downloaded the data and started training. The results are presented in the global rebuttal and show that LoReTTa is scalable and effective in more complex (non-medical) environments.\n\n**New results**. The table below shows the results of our new experiments. Both GPT and LoReTTa were trained on disjoint (video, audio) and (video, text) pairs to solve the problem of cross-modal translation. We then evaluated the models on the unseen task of audio captioning. As can be seen, LoReTTa is more than capable of overcoming the modality gap and rivals the upper-bound models (MMGPT). More details can be found in the global rebuttal.\n\n| Method $\\phantom{.}$ | $\\phantom{...}$ Train | $\\phantom{a}$ Test | BLEU4 | METEOR | ROUGE |\n|----------|----------|----------|----------|----------|----------|\n| GPT | A $\\rightarrow$ V, V $\\rightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T |  $\\phantom{...}$ 1.7 |  $\\phantom{...}$ 18.5 | $\\phantom{...}$  30.7 |\n| LoReTTa | $\\phantom{.}$ A $\\leftrightarrow$ V $\\leftrightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T | $\\phantom{...}$ 2.8 |  $\\phantom{...}$ 20.8 |  $\\phantom{...}$ 34.7 |\n|     |     |     |     |\n| MMGPT | $\\phantom{...}$ A $\\rightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T | $\\phantom{...}$ 6.7 | $\\phantom{...}$ 19.4 | $\\phantom{...}$ 27.1 |\n| MMGPT| $\\phantom{...}$ V $\\rightarrow$ T |$\\phantom{.}$  V $\\rightarrow$ T | $\\phantom{...}$ 7.8 | $\\phantom{...}$  21.3 | $\\phantom{...}$ 29.1 |\n\n\n**References**\n\nAghajanyan et al., CM3: A Causal Masked Multimodal Model of the Internet, 2022\n\nBavarian et al., Efficient Training of Language Models to Fill in the Middle, 2022\n\nDevlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018\n\nFried et al, InCoder: A Generative Model for Code Infilling and Synthesis, 2023\n\nHayes et al., MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration, 2022\n\nRadford et al., Improving Language Understanding by Generative Pre-Training, 2018\n\nRadford et al., Learning Transferable Visual Models From Natural Language Supervision, 2022\n"}, {"review_id": "rQHbWNGndq", "reviewer": "Reviewer_SR2A", "summary": "This paper presents a learning paradigm to account for missing paired modalities during training. For example, for three modalities A, B, and C, the proposed system can be trained on (A, B) and (B, C) paired data but transfer to (A, C) or (A, B, C) paired data. To tokenize the modalities, the authors uses standard tokenizers for language, and raw bytes or spatially-reduced methods (CNN/VQ-VAE) to tokenize images and audios. The model is based on a standard transformer decoder, and the key novelties are: (a) commutative modeling by asking the model to next-token predict modality A from B, or B from A. (b) transitive modeling by producing pseudo data using a linking modality. To incorporate bidirectional context while using causal attention, casual masked modeling loss is employed. The proposed system is evaluated on both customized and real-world datasets and its performance is compared against popular multimodal learning paradigms including masked, casual, and contrastive objectives.", "questions": "Some technical concerns that I have:\n- The baseline linear probing accuracy on SVL-MNIST is rather poor. For instance, the vision-MNIST should at least have an accuracy above 95%, but the vision-only transformer in this paper only achieves ~80%. \n- Also, the text modality of SVL-MNIST seems to be noisy as its linear probing accuracy is only 60%. Is the poor accuracy resulting from low-shot training with 100-shot per class, or due to some label noise?", "limitations": "Yes, the authors discussed limitations of their approach from both technical and societal perspectives.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper conducts extensive experiments (section 5 and 6) on both constructed and real-world medical datasets with three modalities, and show that the proposed system is very effective when there is missing paired modality data. Especially, the authors perform careful ablations against mainstream popular pre-training objectives, including contrastive, casual, and masked modeling losses. \n\nTheoretical analysis (section 4) based on perplexity is interesting and sound. ", "weaknesses": "The proposed system should ideally be tested on larger-scale multimodal datasets, such as AudioSet [1] with image-text-audio modalities. Importantly, this would be crucial to see if the proposed system generalize to real-world vision-language-audio domains where the three modalities are naturally aligned. This setup would be different from the constructed SVL-MNIST dataset where the language modality (from WineReview) has weak or no correlation to the vision (MNIST) and audio (AudioMNIST) modalities.\n\nFor complete ablations, the authors could provide results on the “upper bound” training setups where:\n- All samples have (A, B, C) modalities.\n- There are (A, B), (B, C), (A, C) paired modality data.\n\nNot a weakness but a suggestion:\nL38: “To the best of our knowledge, this scenario has never been considered before in the literature.” —> There is a recent related work [2] in vision community that use images as the linking modality to bind all other modalities such as depth/audio/text.\n\n[1] Audio set: An ontology and human-labeled dataset for audio events. Gemmeke et al. 2017.\n[2] ImageBind: One Embedding Space To Bind Them All. Girdhar et al. 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer SR2A,**\n\nWe appreciate your time and effort in reading our manuscript and providing us with valuable feedback. It is wonderful to hear that the soundness, presentation, and contributions of our paper are considered excellent. We would like to address the remaining questions below.\n\n**Large-scale experiments.** We were faced with a limited computational budget. However, we were still able to perform extensive experiments on a synthetic and real medical dataset with meaningful applications. It is impressive to see that LoReTTa can effectively integrate highly complex omics data across different domains to improve survival predictions. The question, of course, is how to scale the model. According to the AudioSet website and paper, it is primarily a dataset for audio classification. The corresponding video clips must first be found and transcribed to obtain the video and text modalities. However, a recent video-language-audio multimodal dataset [Hayes et al., 2022] was recommended during the rebuttal. It contains 375,000 aligned data pairs sampled from a real-world reinforcement learning problem. Even with limited time and resources, we were able to finish the experiment and achieve performances that are competitive with the upper bound model (see global rebuttal). Because of this additional experiment, we could not provide the upper bound results for the other experiments. However, it was a reasonable decision to omit them. We considered a scenario where there is no dataset with all modalities present and aligned. Thus, in practice, we would not know the upper bound. The only relevant upper bound is the current state-of-the-art, which we have shown to outperform.\n\n**Linear probing accuracy.** Training on raw pixel values instead of image patches is more challenging since we lose potentially important spatial information [Chen et al., 2020; Jaegle et al., 2021; Yu et al., 2022]. We committed ourselves to this setting to make the classification problem of SVL-MNIST more challenging. Otherwise, all models would obtain high accuracy. This would have made it hard to see the effect of each individual design decision.  We also chose the WineReview dataset because classifying wines based on written text alone is particularly challenging, as evidenced by public leaderboards.\n\n**New results**. The table below shows the results of our new experiments. Both GPT and LoReTTa were trained on disjoint (video, audio) and (video, text) pairs to solve the problem of cross-modal translation. We then evaluated the models on the unseen task of audio captioning. As can be seen, LoReTTa is more than capable of overcoming the modality gap and rivals the upper-bound models (MMGPT). More details can be found in the global rebuttal.\n\n| Method $\\phantom{.}$ | $\\phantom{...}$ Train | $\\phantom{a}$ Test | BLEU4 | METEOR | ROUGE |\n|----------|----------|----------|----------|----------|----------|\n| GPT | A $\\rightarrow$ V, V $\\rightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T |  $\\phantom{...}$ 1.7 |  $\\phantom{...}$ 18.5 | $\\phantom{...}$  30.7 |\n| LoReTTa | $\\phantom{.}$ A $\\leftrightarrow$ V $\\leftrightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T | $\\phantom{...}$ 2.8 |  $\\phantom{...}$ 20.8 |  $\\phantom{...}$ 34.7 |\n|     |     |     |     |\n| MMGPT | $\\phantom{...}$ A $\\rightarrow$ T | $\\phantom{.}$ A $\\rightarrow$ T | $\\phantom{...}$ 6.7 | $\\phantom{...}$ 19.4 | $\\phantom{...}$ 27.1 |\n| MMGPT| $\\phantom{...}$ V $\\rightarrow$ T |$\\phantom{.}$  V $\\rightarrow$ T | $\\phantom{...}$ 7.8 | $\\phantom{...}$  21.3 | $\\phantom{...}$ 29.1 |\n\n\n**References**\n\nChen et al., Generative Pretraining from Pixels, 2020\n\nHayes et al., MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration, 2022\n\nJaegle et al., Perceiver: General Perception with Iterative Attention, 2021\n\nYu et al., Scaling Autoregressive Models for Content-Rich Text-to-Image Generation, 2022\n"}]

Author rebuttal:
**Dear Area Chair and Reviewers,**

We are very grateful for your insightful comments and sincerely appreciate the time and effort you have taken to provide constructive feedback. It is gratifying to hear that all reviewers unanimously agree that the paper is well-written and well-motivated. They acknowledge that our proposed method is "widely applicable" [a22R] and "valuable as it addresses a real-world problem" [u4o4]. In particular, our "results show convincing and significant improvements over the chosen baseline models" [QEx4] and “would be a good baseline for any future methods" [7Goz]. We are also thrilled to highlight that the "theoretical analysis contributes to the understanding" [FSDR] of our novel multi-modal and self-supervised framework LoReTTa. Moreover, the extensive experiments "show that the proposed system is very effective" and supported by "careful ablations against mainstream popular pre-training objectives” [SR2A]. Below, we would like to respond to the most common concerns raised by reviewers. We have also addressed individual questions in the comments section.

**Extending Transitive Modeling to BERT and GPT.**
We would like to clarify the differences between LoReTTa, C2M3, GPT, and BERT, as this has caused some confusion. BERT uses masked modeling, which has proven to be a powerful approach for training encoders and bidirectional representations. GPT, on the other hand, uses causal modeling and unidirectional representations to train autoregressive decoders. GPT excels at generative tasks but has more difficulty with discriminative tasks as compared to BERT. Conversely, BERT is not designed to solve generative problems which is also a drawback of the contrastive method CLIP. FIM and InCoder combine the strengths of both BERT and GPT by introducing causal masked modeling, which reinterprets masked modeling as a causal problem. Thus, we get a GPT-style architecture augmented by BERT-style training. This method was extended to the multimodal case by CM3. We go further and apply commutative modeling to CM3, yielding C2M3. The next step is to incorporate transitive modeling and we arrive at LoReTTa. Thus, LoReTTa unifies BERT and GPT training by combining and extending them. In essence, C2M3 is an extended version of GPT, while BERT and CLIP cannot be directly modified with transitive modeling because they are unable to generate the missing modality. For example, one would have to train a diffusion model on top of the CLIP embeddings to generate an image as in DALLE-2. 

**Comparison to CycleGAN.**
We agree that LoReTTa has a similar flavor to the cycle consistency concept introduced in CycleGAN. However, our proposed approach is much more general and applicable. The main idea of CycleGAN is to translate an image from domain A to domain B, compute the adversarial loss on the generated image, and reconstruct the original image from the generated image. This involves two domains from the same modality. On the other hand, LoReTTa works on three or more domains from three or more modalities. In addition, we avoid modeling a → b'→ a as in CycleGAN. Instead, we use the idea of transitivity to model a → b' → c, given the pair (a, c). This is a stronger learning signal since we have to reconstruct an entirely new data point c that was not part of the model input (a). In fact, this idea can be generalized to more complex multimodal relationships such as (A, B), (A, D), (A, C), and (B, C) (as shown in Figure 2d). As long as there is a path from one modality to the other, we can use the principle of transitivity to approximate the full data distribution. How would we learn the missing distribution (C, D)? By taking a real data point (a, d) and learning the consistent transition a → b' → c' → d or a → c' → d, respectively. This would not be possible at all using only the CycleGAN approach.

**Large-scale Experiments.**
We appreciate the reviewer's insights and understand the potential value of large-scale experiments. However, due to resource limitations, we were faced with constraints on the amount of computations. Nevertheless, not counting hyperparameter search, we pre-trained 17 models and fine-tuned them a total of 169 times to obtain a comprehensive experimental insight into the operation of LoReTTa that strongly complements our theoretical analysis. Within these constraints, we were able to derive meaningful and impactful results that contribute significantly to the ongoing scientific conversation in our rapidly changing field. We are fully committed to our research and recognize the importance of scalability and generalizability. Reviewer a22R recommended the new large-scale and multimodal MUGEN dataset with 375K naturally aligned video (V), audio (A), and text (T) samples from a real reinforcement learning problem.

We immediately downloaded all the files and started to run an additional set of experiments. In particular, we used video as the linking modality and considered the disjoint datasets (A, V) and (V, T). We pre-trained the same transformer as the state-of-the-art upper baseline MMGPT. As can be seen below, LoReTTa clearly manages to rival the upper bound.

| Method $\phantom{.}$ | $\phantom{...}$ Train | $\phantom{a}$ Test | BLEU4 | METEOR | ROUGE |
|----------|----------|----------|----------|----------|----------|
| GPT | A $\rightarrow$ V, V $\rightarrow$ T | $\phantom{.}$ A $\rightarrow$ T |  $\phantom{...}$ 1.7 |  $\phantom{...}$ 18.5 | $\phantom{...}$  30.7 |
| LoReTTa | $\phantom{.}$ A $\leftrightarrow$ V $\leftrightarrow$ T | $\phantom{.}$ A $\rightarrow$ T | $\phantom{...}$ 2.8 |  $\phantom{...}$ 20.8 |  $\phantom{...}$ 34.7 |
|     |     |     |     |
| MMGPT | $\phantom{...}$ A $\rightarrow$ T | $\phantom{.}$ A $\rightarrow$ T | $\phantom{...}$ 6.7 | $\phantom{...}$ 19.4 | $\phantom{...}$ 27.1 |
| MMGPT| $\phantom{...}$ V $\rightarrow$ T |$\phantom{.}$  V $\rightarrow$ T | $\phantom{...}$ 7.8 | $\phantom{...}$  21.3 | $\phantom{...}$ 29.1 |

Meta-review: This work proposes a multi-modal learning framework for non-aligned modalities, linking modalities A and C via A-B and B-C. The reviewers enjoyed the paper's performance, simplicity and importance and clear presentation. While some questions about additional datasets, upper-bounds, messaging (about BERT's ability to generate text), these were sufficiently cleared in the extensive rebuttal and discussions.
Decision: Accept (poster)