SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $J_{f}$  is the Jacobian of  $f$ . By iteratively applying the Taylor rule, a more general error propagation formula is obtained for longer chains. Intuitively, errors from the first transitions are amplified more and more. However, because we use the predicted  $x_{3}$  to reconstruct  $x_{1}$ , we can actually bound the error term  $e_{1,3} \coloneqq J_{f}(x_{3}|f(x_{2}|x_{1}))J_{f}(x_{2}|x_{1})e_{1,2} + e_{2,3}$ . Recall that we have a ground truth  $x_{1}$  for which the following is true
$$
\begin{array}{l} x _ {1} = f \left(x _ {1} \mid x _ {3}\right) + e _ {3, 1} = f \left(x _ {1} \mid f \left(x _ {3} \mid x _ {2}\right) + e _ {2, 3}\right) + e _ {3, 1} \\ = f \left(x _ {1} \mid f \left(x _ {3} \mid f \left(x _ {2} \mid x _ {1}\right) + e _ {1, 2}\right) + e _ {2, 3}\right) + e _ {3, 1} \tag {5} \\ \end{array}
$$
Table 2: After pre-training, a classifier is trained on the frozen models. We analyze the classification accuracy in a few-shot scenario with 100 samples per class over three trials on SVL-MNIST. The subscript denotes the pre-training datasets with image (I), text (T), or speech (A) modalities, while the blue colors highlight the results on the unseen modality tuple during pre-training.
Training	Testing
I	T	A	(I,T)	(I,A)	(T,A)	(I,T,A)
CM3(I)	79.7±0.8	-	-	-	-	-	-
CM3(T)	-	57.0±0.5	-	-	-	-	-
CM3(A)	-	-	80.3±0.6	-	-	-	-
C2M3(I,T)	82.9±0.6	61.9±0.7	-	89.7±0.2	-	-	-
C2M3(I,A)	83.8±0.5	-	79.9±0.2	-	89.9±0.1	-	-
C2M3(T,A)	-	61.1±0.4	79.5±0.2	-	-	85.2±0.3	-
C2M3(I,T), (I,A)	83.1±0.8	59.8±0.4	80.7±0.4	87.9±0.6	89.3±0.2	71.2±0.4	87.3±0.4
C2M3(T,I), (T,A)	81.7±0.1	63.2±0.6	78.5±0.6	89.1±0.3	72.9±0.2	84.8±0.5	83.2±0.4
C2M3(A,I), (A,T)	80.8±0.4	63.9±0.6	82.8±0.7	77.8±0.5	89.8±0.1	88.1±0.5	89.9±0.5
LoReTTa(I,T), (I,A)	82.7±0.9	62.8±0.4	82.5±0.6	88.5±0.7	89.7±0.1	84.0±0.2	90.7±0.6
LoReTTa(T,I), (T,A)	81.2±0.9	63.3±0.4	81.0±0.5	89.0±0.3	80.9±0.6	85.5±0.5	87.8±0.3
LoReTTa(A,I), (A,T)	80.1±0.3	62.1±0.6	84.2±0.5	83.0±0.7	90.4±0.3	89.0±0.2	91.6±0.7
C2M3(I,T,A)	80.3±0.4	58.3±0.4	79.3±0.7	85.2±0.2	85.3±0.4	82.5±0.3	88.5±0.2
Since we are minimizing the reconstruction error of  $x_{1}$  given  $\hat{x}_3 = x_3 + e_{1,3}$  during backpropagation, the error terms  $e_{1,2}$  and  $e_{2,3}$  must be bounded. And since they make up the error term  $e_{1,3}$ , it too must be bounded, as long as  $f$  and  $J_{f}$  are.
Evaluation protocol: We empirically analyze LoReTTa on a constructed dataset, a real-world medical dataset, and an offline reinforcement learning dataset with three modalities. For the downstream tasks, we choose object classification, survival prediction, and cross-modal translation, respectively. With these three tasks, we cover both discriminative and generative problems. We assess the former through linear probing and the latter in a zero-shot scenario. This allows us to directly evaluate the quality of the learned features. The order of the experiments is as follows: First, we perform an ablation study on the synthetic dataset to analyze the effect of transitive modeling on different modality combinations. Second, we compare LoReTTa with other methods such as masked (BERT), causal (GPT), and contrastive (CLIP) modeling on the real-world medical dataset. Third, we compare our model with a state-of-the-art autoregressive cross-modal transformer on the offline gaming dataset. In the Appendix, we list all the optimization hyperparameters and model configurations. We also publish the pseudocode and data processing pipeline.
SVL-MNIST: We test our method on a custom dataset derived from real-world data that includes speech (A), vision (I), and language (T) modalities representing 10 different categories. The speech dataset features about 40,000 spectrograms from AudioMNIST [31], the vision dataset comprises 70,000 images from MNIST [34], and the language dataset consists of 130,000 documents from WineReviews [60]. We link the datasets via their labels 0-9. This approach results in weakly aligned modalities. For the sake of clarity, we will refer to this dataset as Speech-Vision-Language-MNIST (SVL-MNIST). We randomly split the dataset such that they have exactly the same relationship, as shown in Figure 1c. In particular, our bimodal datasets (A, I), (A, T), and (I, T) each have 12,000 non-overlapping samples. All remaining samples are part of the unimodal datasets A, I, and T.
TCGA-OMICS: The medical dataset contains omics (e.g., genomics, proteomics, and transcriptomics) sequencing values from The Cancer Genome Atlas Program (TCGA) [67]. We select the three largest subsets, which include messenger ribonucleic acid (mRNA), micro-ribonucleic acid (miRNA), and reverse-phase protein array (RPPA) from up to  $\sim 11,000$  patients. We align the dataset at the patient level and obtain approximately 7,000 patients with all three modalities. To simulate a setting with missing modality combinations, we split the dataset into two subsets (mRNA, miRNA) and (mRNA, RPPA) with non-overlapping 3,500 samples each. We choose mRNA as the linking modality because it is one of the most widely available genetic modalities. TCGA is quite unique in that it is the only publicly available medical dataset with multiple aligned modalities.
MUGEN-GAME: As a large-scale and multimodal dataset, we choose MUGEN-GAME [27] with 375,000 naturally aligned video (V), audio (A), and text (T) samples collected by a reinforcement learning agent in the closed-world platform game CoinRun. Mugen, the main character, walks, jumps, collects coins, kills monsters, climbs ladders, and dies – triggering various sound effects that are overlaid with background music. All of their adventures are recorded in three-second video clips and audio tracks. A group of human narrators provide rich text descriptions for each scene. We use the most important modality in video games, video, as the linking modality and consider the disjoint datasets (V, A) and (V, T) in our final experiments.
Implementation details: For the first two experiments, we use a transformer decoder with 8 layers and 8 attention heads, with an embedding dimension of 512 and query, key, and value dimensions of 64 each. This compact architecture has proven effective in various tasks and has been widely used in the literature [23, 28, 36]. We apply a single parameterized embedder to encode tokens and add modality-specific learned position embeddings [54, 56, 2]. For optimization, we choose the AdamW algorithm with a learning rate of 6e-4, a weight decay factor of 0.1, and a gradient clipping of 1. The learning rate undergoes a 10-fold decay using cosine annealing and a linear warm-up during the first couple hundred steps. For the third experiment, we use exactly the same training hyperparameters and model configurations as in the baseline [27]. In particular, the transformer has 12 layers, 8 attention heads, and an embedding dimension of 768. The query, key, and value dimensions are each 96.
Tokenization scheme: Due to the low dimensionality of the SVL-MNIST dataset, we directly encode the raw byte streams by binning the continuous values to a dictionary of 256 each. TCGA-OMICS data can be very high-dimensional and must be handled differently. For example, an mRNA expression array typically contains more than 20,500 entries. Therefore, we first reduce the dimensions by a factor of 10 using PCA and then bin the values to a dictionary of 1,000 (similar to recent work [11, 70, 20]). The final input sizes are 2,048, 64, and 16, respectively – retaining their original relative lengths. The high-dimensional video and audio samples in MUGEN-GAME are tokenized identically to Hayes et al. [27] using their pre-trained VQ-VAE encoders.
6.1 SVL-MNIST
In the first experiment, we train one model for each modality (A, I, and T) using causal masked modeling (CM3). We then train four additional models using commutative causal masked modeling (C2M3) on the bimodal datasets (A, I), (A, T), and (I, T), as well as the trimodal dataset (A, I, T). These seven models serve as unimodal, multimodal, and "upper" baselines, respectively. Then, using C2M3 with T as the linking modality, we train simultaneously on both (I, T) and (A, T), and repeat this process for all modality combinations. Next, we initialize LoReTTa with the C2M3 model weights to explicitly merge the bimodal datasets via the linking modality. After pre-training, we freeze all models and evaluate them in a low-data regime with only 1,000 labeled samples (100 per class). We perform three runs, each with a newly randomized subset, and report the classification accuracy along with the perplexity on the test set. Note that we only train the models on the modalities that the model saw during pre-training. This also means that during probing, the linear classifier will see some samples of the unseen modality pair. However, the pre-trained backbone itself did not see them. In this way, we can analyze how the model combines previously unseen mixture of input modalities (modality combinations).
When pre-trained only on the pairs (T, I) and (T, A), LoReTTa achieves a remarkably low perplexity of 5.04 on the unseen pair (I, A), as shown in Table 1, significantly outperforming the baseline non-transitive model trained with C2M3, which has a perplexity of 104.27. This trend extends to other combinations as well. For example, the model trained on (A, I) and (A, T) and evaluated on (I, T) has a perplexity of 3.13 (LoReTTa) compared to 9.70 (C2M3). Similarly, training on (I, T) and (I, A) and evaluating on (T, A) yields a perplexity of 11.21 (LoReTTa) versus 30.88 (C2M3). Importantly, LoReTTa consistently achieves similar perplexities for unseen and seen modality combinations, demonstrating its ability to effectively learn the missing distribution. A similar trend appears in the downstream task (Table 2). In almost all cases, LoReTTa consistently outperforms C2M3 on the unimodal datasets A, I, and T, as well as the bimodal datasets (A, I), (A, T), or (I, T). In a nutshell, if any of the modality pairs are missing, training a model with LoReTTa is recommended because it uniquely integrates these modalities and improves accuracy compared to any single-modality model.
Table 3: After pre-training, a Cox proportional hazards model is trained on the extracted features from TCGA-OMICS. We use mixtures of mRNA (M), miRNA (I), and RPPA (R) as inputs. Blue highlights the c-index on the unseen tuples, and the underline highlights the second-highest value.
Training	Testing
M	I	R	(M, I)	(M, R)	(I, R)	(M, I, R)
BERT	0.592	0.621	0.594	0.595	0.613	0.594	0.610
T-BERT	0.573	0.618	0.626	0.580	0.591	0.623	0.608
GPT	0.575	0.590	0.611	0.585	0.576	0.606	0.588
T-GPT	0.616	0.607	0.599	0.620	0.619	0.611	0.622
CLIP	0.561	0.603	0.587	0.610	0.600	0.623	0.612
L-CLIP	0.588	0.615	0.587	0.620	0.599	0.614	0.620
C2M3	0.621	0.599	0.599	0.624	0.620	0.571	0.624
LoReTTa	0.652	0.623	0.563	0.660	0.652	0.623	0.657
C2M3(M, I, R)	0.665	0.635	0.645	0.643	0.671	0.650	0.620
In fact, providing a few samples with three modalities during linear probing further increases the classification score of models pre-trained with LoReTTa. Interestingly, the "upper" baseline trained on all aligned modalities often lags behind LoReTTa despite favorable perplexity scores. This strongly suggests some kind of negative transfer or modality competition that LoReTTa is able to overcome.
6.2 TCGA-OMICS
We pre-train our transformer with C2M3 and initialize LoReTTa with the weights of the C2M3 model, as we did in our SVL-MNIST experiments. For GPT, we simply use the algorithm of C2M3, but disable causal masking and commutative switching. BERT uses the same transformer architecture; we simply change the unidirectional attention mask to a bidirectional attention mask. To directly demonstrate the benefit of transitive modeling, we extend both GPT and BERT using this technique. The resulting models are named T-GPT and T-BERT. The CLIP-based model consists of 3 encoders (each using our same transformer architecture) and a contrastive loss [32] to align the features. After pre-training, we extract the embeddings and fit a Cox proportional hazards model with an elastic net penalty for each modality combination using all available labels. In 6 out of 7 test sets, the LoReTTa pre-trained transformer achieves the highest c-index (Table 3). Most importantly, this includes the unseen modality pair (miRNA, RPPA) and the triplet (mRNA, miRNA, RPPA). The only frameworks on par with LoReTTa are T-BERT and CLIP on the unseen dataset (miRNA, RPPA). On the unseen trimodal dataset, T-GPT and CLIP give good results but are far behind LoReTTa. Overall, transitive modeling mostly improves the performance of all models. Notably, LoReTTa and CLIP are the only strategies that consistently allow for positive or neutral transfer, which is not always the case with other frameworks. A major drawback of contrastive learning is that it requires a separate encoder for each additional modality. In addition, CLIP-based approaches only excel at large batch sizes. Here, we compare models with the same computational constraints. We spent our limited resources on additional computing capacity. This allows us to scale up the CLIP experiment by increasing the batch size by a factor of 5 and the number of training steps by a factor of 3. L-CLIP consistently improves the c-index of CLIP but requires 3x more VRAM than LoReTTa. In contrast to the SVL-MNIST experiments, the 3-modal "upper-bound" model suffers less from negative transfer or modality competition and achieves the best result for the majority of modality combinations.
6.3 MUGEN-GAME
Recently, there has been a growing interest in cross-modal generation. While much research has focused on text-to-video or video-to-text translation, other cross-modal tasks such as video-to-audio, audio-to-video, text-to-audio, or audio-to-text have not been investigated as thoroughly. In our large-scale experiment, we address the latter task. We use video as the linking modality and consider the disjoint datasets (video, audio) and (video, text). Splitting the training set equally gives us about 187,500 pairs in each set. We use the same optimization hyperparameters and model architecture as the state-of-the-art upper baseline MMGPT [27] to train our version of (MM)GPT and also LoReTTa. As seen in our TCGA-OMICS experiment, BERT, unsurprisingly [5, 64], fails to generate long-range coherent samples (see results with mRNA). Therefore, we do not use BERT in this experiment. We
Table 4: Both GPT and LoReTTa are trained on disjoint (video, audio) and (video, text) pairs on MUGEN-GAME. We then evaluate the models on the unseen task of audio captioning using BLEU4, METEOR, and ROUGE. MMGPTs represent the upper-bound models.
Method	Train	Test	BLEU4	METEOR	ROUGE
GPT	A → V, V → T	A → T	1.7	18.5	30.7
LoReTTa	A ↔ V ↔ T	A → T	2.8	20.8	34.7
MMGPT	A → T	A → T	6.7	19.4	27.1
MMGPT	V → T	V → T	7.8	21.3	29.1
also avoid training a CLIP model, as this would require us to train a text diffusion model (or equivalent generator) on top of the pre-trained encoder, which is neither end-to-end nor straightforward. For GPT, we train the model to generate video from audio and text from video. In this way, we force the model to implicitly learn how to caption audio tracks. In the case of LoReTTa, we model the transition between audio and text directly through transitive modeling. Remarkably, GPT trained on two bimodal datasets outperforms the audio-to-text baseline in terms of ROUGE, showing strong positive transfer (Table 4). LoReTTa further improves on GPT and even rivals the video-to-text baseline w.r.t. METEOR and ROUGE. The BLEU4 values are low for both models. Since BLEU involves exact n-gram matching, these results are to be expected given the strict data constraints.
With LoReTTa, we introduced an innovative self-supervised learning framework for multimodal integration that learns any missing joint conditional distribution given a linking modality. We demonstrated this theoretically and practically. In particular, our method combines the rules of commutativity and transitivity to model the multimodal datasets  $(A,C)$  and  $(A,B,C)$  given the two non-overlapping datasets  $(A,B)$  and  $(B,C)$ . In fact, we showed that the extracted features of a transformer pre-trained with LoReTTa are very expressive for all modality combinations, including the unseen ones. While we only evaluated our algorithm on datasets with three modalities, it is much more general because it can be used with any number of modalities and any mixture of modalities. As long as there is a chain of aligned modalities in the dataset, LoReTTa is able to learn the transition  $(X_{i}\rightarrow \ldots \rightarrow X_{j}),\ldots ,(X_{i}\rightarrow \ldots \rightarrow X_{k})$  to model the missing distribution  $P(X_{j},\dots,X_{k})$  (as shown in Figure 2d). To our knowledge, this scenario has never been considered in the literature without training an ensemble of contrastive models based on the ideas of CLIP - most notably Wav2CLIP [68] and its extension ImageBind [26].
Broader impacts. We believe that LoReTTa will serve as a basis for training very powerful multimodal generative and discriminative models that will be useful in many areas of research. While our method itself cannot be misused, future models trained with LoReTTa could be exploited to generate harmful content of higher quality. However, LoReTTa was designed to improve the quality of neural networks that contribute to society, for example by training more accurate multimodal systems that can help doctors and other practitioners in their daily work.
Environmental impacts. We trained all of our models on a single NVIDIA A100-SXM4-40GB GPU using PyTorch 2.0. Techniques such as mixed-precision training, FlashAttention, and TF32 helped us reduce memory usage and runtime. Pre-training a model with causal masked modeling took us about 12 hours on average. On the other hand, fine-tuning with LoReTTa took longer – about 3 days. This is because we had to autoregressively decode the predicted samples one token at a time.
Limitations. LoReTTa is best used in a setting where we are missing some modality combinations. This only works if there is at least one connecting modality, otherwise, our method cannot be applied. But what about datasets where some samples have all modality combinations? Can our method still be used? Yes, this is possible. Although we have not analyzed this scenario, LoReTTa is likely to provide benefits by augmenting those data points that are still missing the remaining modalities.
Acknowledgements. M.T. is supported by the Helmholtz Association under the joint research school "Munich School for Data Science - MUDS". There are no other funding or competing interests.