Sparsely multimodal data fusion

## 1 Abstract

The ability of masked multimodal transformer architectures to learn a robust embedding space when modality samples are sparsely aligned is studied by measuring the quality of generated embedding spaces as a function of modal sparsity. An extension to the masked multimodal transformer model is proposed which incorporates modal-incomplete channels in the multihead attention mechanism called modal channel attention (MCA). Two datasets with 4 modalities are used, CMU-MOSEI for multimodal sentiment recognition and TCGA for multiomics. Models are shown to learn uniform and aligned embedding spaces with only two out of four modalities in most samples. It was found that, even with no modal sparsity, the proposed MCA mechanism improves the quality of generated embedding spaces, recall metrics, and subsequent performance on downstream tasks.

## 2 Introduction

Multimodal models are increasingly becoming the norm for deep learning applications.[, , ] Many studies use model trained with two aligned modalities.[, , , , , , , , , , ], including incorporating images into large language models.[, ] Training models with more than two aligned modalities has also been studied,[, , , , , , ], and recent examples explore learning from multiple unaligned or partially aligned modalities[, , , , , ].

Most of these examples use a combination of text, audio, image and video. However, applications can use data other than these traditional media formats. For example, multisensor fusion in home monitoring systems and robotics includes tabular sensor data and time series data from different types of sensors.[]. Bioinformatics and biomedical applications use data that consists of tabular, image, and sequence data. In these fields, each media format in the data can also be comprised of different modalities having the same data type but different data source, for example as in two tables of data from different types of experiments.[]

Constrastive learning of multimodal data is an important class of multimodal model.[, , , , , ] In the well-known CLIP model, it generates joint embeddings for modalities, which can be used for multi-modal classification and regression or for conditioning of diffusion probablistic models. These embeddings can also be used for multi-modal retrieval, such as in Everthing At Once where video retrieval can be performed via text or audio via a trimodal embedding. As the number of modalities used for training a model increases, modal-incomplete samples are more likely to occur. Modal fusion models which can not include modal-incomplete samples are thus not well suited to be used for applications with datasets that have many modalities.

There are a few approaches which can achieve self-supervised learning with modal-incomplete samples outside of contrastive representation learning. Interleaved data can use incomplete modalities due to their treatment of samples as a sequence.[] Similarly, masking of the model in a late stage fusion block can be used[, ]. These model architectures require an autoregressive or a masked language objective in order to train on the interleaved data or predict missing data. These types of models may be incompatible with objectives where an embedding at inference time should be equivalent whether or not a modality is missing from the sample being embedded. For example, given a sample with 3 modalities, it would not be possible to generate embeddings from any combination of only 2 of the modalities which are all numerically similar.

Recent studies have shown that using contrastive learning between unimodal representations and a learnable fusion representation can generate useful joint embeddings.[, ] This type of model uses block attention masking to effectively perform self-attention on unimodal representations and cross-attention with a fused representation in the attention block. However, the application of this masked multimodal attention (MMA) model architecture to datasets which are modality incomplete is yet underexplored. This study aims to show that masked multimodal attention functions well even when modalities are sparsely aligned. We propose an extension of the masked multimodal transformer model that incorporates modal-incomplete channels in the multihead attention block called modal channel attention (MCA). Using two well-known datasets, each with four modalities, both models are shown to learn quality embedding spaces up to a sparsity level of at least 40%, corresponding to an average number of modalities per sample of 2.4. We find that MCA generally improves the quality of generated embedding spaces, recall metrics, and subsequent performance on downstream tasks by linear probing. We conclude with a discussion on potential applications and directions for future work.

## 3 Related Work

The Everything At Once [] model uses a transformer encoding of multiple modalities (i.e. a fusion transformer) with contrastive loss. The fusion transformer is applied multiple times per minibatch in order to formulate embeddings for each modality and each pair of modalities. These embeddings are applied in a combinatorial contrastive loss function. The final unimodal and bimodal embeddings are pooled and projected to a multimodal embedding space. This method requires a number of forward passes that scales exponentially with the number of modalities, which is undesirable for computational efficiency and scalability.

FLAVA[] uses multiple loss models for data which is missing a modalities. For example, with language they use a masked language objective, while for image-text pairs, they use a contrastive loss. Their approach allows for unimodal and multimodal inference, but does not generate a multimodal fusion embedding space. Similarly, Zhang et al. developed a model which can be trained with missing modality combinations.[] The model projects unimodal encodings into a modality-aligned feature sapce, and then performs weight-shared dual attention prediction of two sets of outputs. The first output is trained with supervision to class labels, while the second prediction is trained with supervision to unimodal predictions across epochs. This is shown to improve predictions for unseen modalities.

Another approach by Wei et al. uses a multimodal fusion model that is trained to generate missing modalities.[] A similar approach was taken with LORRETA using an objective of predicting a third modality given two other modalities. This approach requires bimodal pairs for each forward pass and can not directly embed higher order modality combinations.[] Neither approach generates a fused embedding space that is directly applicable to tasks based on the fused data embeddings such as retrieval or linear probing.

Zorro is an MMA model which produces both unimodal and multimodal outputs trained with a contrastive loss.[] A similar MMA architecture was recently applied by Shi et al. in order to fuse 3 modalities for image segmentation in a biomedical application.[] While the latter study notes that MMA seemed to function well with missing modalities, they did not directly explore the performance of this type of modal fusion with sparsely aligned multimodal data.

Figure: Figure 1: Overall model architecture demonstrating a single forward pass. $N_{i}$ represents the number of tokens. The model architecture is related to models present in Refs. [] and [].
Refer to caption: extracted/2403.20280v1/figure0.png

Figure: (a) Multimodal self-attention mask without modal-incomplete channels (MMA).
Refer to caption: extracted/2403.20280v1/figure1b-x.png

## 4 Model

The studied model architecture is presented in Figure 1. It consists of a transformer encoder with learnable pooling layer and noise contrastive estimation (NCE) loss between pooled tokens. Also used in this study are trainable transformations on the input data - linear transformations or MLPs which are applied on a token-by-token basis in order to encode tabular data or compress pretrained frozen embeddings as inputs. The attention model uses MMA. MMA uses fusion token blocks which are attended by unimodal token blocks. Unimodal token blocks are only able to self-attend, as shown in Figure 2(a).

This study introduces multi-channel attention into MMA, where blocks of fusion attention are only attended to by a subset of the modalities. In contrast to the method of Ref [], this maintains the efficiency of using a single forward pass per sample, regardless of the number of modalities present in the sample. Each fusion channel corresponds to the fusion of a different set of modalities as shown in Figure 2(b). In the learnable pooling layer, each fusion channel corresponds to a unique pooling token. NCE Loss is applied between each pair of unimodal and fusion channel tokens. The fusion channel which is attended to by all modalities is taken as the fusion embedding for embedding space analyses and downstream tasks. The learnable pooling layer has a similar multimodal mask, preventing fusion tokens from attending to unimodal tokens. The pooled layer includes a token for each modality and each type of fusion.

For this study, 5 transformer encoder layers are used, each with a hidden size of 512 and 8 attention heads. The feed-forward layers use a feed-forward multiplier of 4 and the GeGLU activation function. There are a total of 88 fusion tokens in both studied models. In the masked multimodal transformer, all 88 fusion tokens are pooled in the pooling layer, while in the MCA transformer, each channel uses 8 tokens and 11 channels are present as shown in Figure 2(b). As described above, each set of 8 tokens is pooled separately in the pooling layer. Importantly, there are an identical number of trainable input tokens in each model and a nearly equal number of parameters, increased in MCA only by 10 additional learnable pooling tokens in the pooling layer.

Figure: (a) CMU-MOSEI dataset with MCA

## 5 Methods

### 5.1 Datasets

#### 5.1.1 CMU-MOSEI

The CMU-MOSEI was obtained and processed using the mmdatasdk Version 1.1 using the included word level alignment example. This results in 23248 samples of aligned unimodally embedded data corresponding to glove vector, OpenFace, COVAREP, and FACET encoders.[] A test split is randomly chosen with 2324 samples.

For each modality, a sample is a series of embeddings for multiple time steps. The number of vectors vary per sample, due to the embedded video clips having varying duration. To prepare the modalities for input into the transformer block, each vector in a sample is transformed by using a linear layer and layer normalization, resulting in a token embedding size of $N_{emb}$. This normalized, transformed vector is then added to a standard sinusoidal positional embedding vector to encode it’s position in the sequence of vectors for a given modality. No other learnable token embedding is used for these samples. The result of this process is that each sample token is transformed into a token for input in to model studied in this paper

#### 5.1.2 TCGA

The Cancer Genome Atlas (TCGA)[] provides a multi-omics dataset that consisting of tabular data for gene expression, reverse phase protein arrays (RPPA), DNA methylation, and miRNA measurements. This data was downloaded from the supporting information of Ref []. To reduce the number of gene expression and DNA methylation columns in the dataset, the top 800 genes and methylation sites with the highest variance were used to create a signature of the gene expression and methylation data. For RPPA data and miRNA tables, there are 198 protein columns and 662 MiRNA columns. To align unimodal samples into a multimodal dataset, we use provided identification numbers for patient and sample, resulting in an intersection of 7017 samples that have all modalities. A test split is randomly chosen with 707 samples.

To prepare TCGA tables for the MCA and MMA encoders, tabular values are passed tokenwise through a trainable 2 layer MLP(1,$N_{emb}$,$N_{emb}$) and ReLU activation function. This allows a continuous representation of the tabular value into a vector with the same size as the transformer encoder embedding space $N_{emb}$. The tabular column index is encoded with the standard learnable embedding vector of size $N_{emb}$ for each index. The value and column index encodings are added together to form the initial embedding vectors for the transformer encoders.

### 5.2 Modal Sparsity

In order to evaluate the performance of masked multimodal attention with missing modalities, unimodal samples are dropped from the datasets. Randomly selected samples are dropped uniquely for each modality with a probality defined herein as the modal sparsity. The modal sparsity reported in the following figures represents the fraction of dropped samples in each modality. Due to the training cost of many models, experiments were performed with datasets corresponding to 0, 0.2, 0.4, 0.6, and 0.8 modal sparsity.

Since both datasets used in this study contain 4 modalities, a modal sparsity of 0.2 indicates that most samples have 3 modalities present, while at a modal sparsity of 0.6 indicates that most samples have only 1 or 2 modalities present. When a modality is dropped from a sample, a padding token is used for all tokens corresponding to that modality such that they are masked from subsequent attention blocks and subsequent loss function terms are dropped. When all modalities attending to a specific fusion channel are masked, InfoNCE loss function terms using the corresponding token are also eliminated from the loss function.

### 5.3 Training

Training was performed for each model on 4xA10G Nvidia GPUs. Hyperparameters were chosen identically for MMA and MCA experiments. An effective batch size of 32 and cosine scheduled learning rate with a maximum of $10^{-4}$ and warm up of 2000 steps is used for all experiments. Test splits of datasets were selected randomly as a fraction of 0.15 of a dataset. For CMU-MOSEI experiments, 32 epochs are trained. For TCGA fusion, 128 epochs are trained. The epoch number selected for evaluating embeddings was hand selected by identifying the best set of eval loss scores for a model/dataset pair at all modal sparsities as shown in Fig.  3 . The model and training code were developed using the Pytorch 2.1.[]

Figure: (a) Uniformity of fusion token embedding spaces.($\downarrow$)

Figure: (a) Modal average of Median Rank for CMU-MOSEI ($\downarrow$)

## 6 Evaluation of Embeddings

This section aims to analyze the characteristics of the embeddings produced by the trained models. Embedding spaces trained with contrastive learning are characterized by alignment and uniformity.[] Alignment and uniformity were calculated according to Ref.  [] with the test split of the datasets. The results of averaging these quantities over the unimodal embeddings produced by a given model are displayed in Figure 4, while additional plots for each modality are included in Section  9.

If a model is trained to generate an embedding space with lower uniformity (indicating a more uniform embedding space), the alignment of positive examples will tend to be decreased.[] The uniformity of MCA is significantly lower (improved), producing a more uniform embedding space than MMA, while the alignment is increased (worsened). The alignment is calculated here for the complete (all modality) fusion channel in MCA, while the loss in MCA calculated with multiple fusion channels. As the modal sparsity is increased beyond 40%, both model’s fusion uniformities increased (worsened). The unimodal uniformity also increased, indicating less uniform unimodal embedding spaces. The alignment, however, does not change significantly as the modal sparsity is increased.

Recall metrics were examined for the generated validation datasets (median rank, $R_{1}$, $R_{5}$, and $R_{10}$). These recall metrics demonstrate the ability of the model to use a unimodal embedding to recall a fused embedding. Both MMA and MCA architectures always produce the same unimodal embeddings for a given set of unimodal tokens, regardless of whether other modalities are absent, or different from multimodal samples on which the model is trained. Here, the rank is defined as the median of the ranks of a matched fusion embedding vector to a unimodal embeddings in the ranked set of cosine similarities between a single unimodal embeddings and each fusion embedding.

The median rank is the median of these ranks for all unimodal embeddings. The $R_{x}$ metric is the probability that the correct embedding vector is in the top $x$ most similar fusion embeddings to a unimodal embedding. The average of these metrics across all modalities is shown in Figure 5. Importantly, as modal sparsity is increased, the number of unimodal samples decreases proportionally, while some fusion samples are dropped from the set of embeddings when all unimodal samples have been dropped by the procedure of introducing modal sparsity. While this prevents a direct comparison of metrics between datasets preprocessed with different modal sparsity, it is still useful as a comparison between MCA and MMA models.

The average median rank of unimodal test split embeddings is improved in MCA over MMA. In the larger CMU-MOSEI dataset, the difference is greatest at high modal sparsity. In the smaller TCGA dataset, it is greatest at low modal sparsity. This may be due to better uniformity and worse alignment in MCA. At the highest sparsity of 0.8, there is an average number of modalities per sample of ¡ 1 and the average median rank drops significantly. This may be attributed to a fewer number of unimodal and fusion embeddings.

The recall metrics ($R_{1}$, $R_{5}$, and $R_{10}$) show the capability of the unimodal embedding spaces to be used to identify matching fusion embeddings. MCA has improved recall metrics over MMA for most modal sparsities examined in both datasets. Similar to the average median rank metrics, the smaller TCGA dataset shows the best recall metric improvement of MCA at low modal sparsity. For the larger CMU-MOSEI dataset, the improvement is smaller and may be within the range of model variation due to hyperparameter selection and choice of evaluation epoch, yet MCA has higher recall metrics than MMA for most modal sparsities studied. The results show the utility of using MCA as a method of retrieving embedding containing information from multiple modalities, even when only a single modality is available at inference time and training examples contain only sparsely aligned modalities.

Figure: (a) Average AUPR value for multi-class classification of TCGA tumor type.($\uparrow$)

### 6.1 Task Performance

The analysis of the generated embeddings on downstream tasks is a crucial component of validating model performance. In this section the linear probing performance of embeddings as a function of modal sparsity is demonstrated. A linear layer is trained using embeddings produced from the dataset training split using either L1 loss for regression or cross entropy loss for classification. The metrics displayed in Figure 6 are calculated using the trained linear model to run inference on test dataset splits which have not been used to train the embeddings or the linear probe. Importantly, no pretrained model weights are relaxed in this evaluation. All tasks are thus a direct linear probing of the generated embedding spaces for the test dataset splits. In general MCA provides improved results over MMA for both datasets.

The task performed for the TCGA dataset is a multiclass problem with 32 classes. These correspond to the type of cancer present in the specimen from which a sample was generated. The class-averaged area under the precision-recall curve (AUPR) is presented in Figure 6. Both MCA and MMA perform well, while MCA shows an slightly increasing level of improvement over MMA as modal sparsity is increased to 0.4.

The CMU-MOSEI sentiment analysis task is a regression to a single value. This value ranges between 0 and 1, corresponding to negative and positive sentiment. In the initial study describing the CMU-MOSEI dataset, a correlation coefficient of 0.54 is achieved using an LSTM-based modal fusion architecture.[] While better correlation coefficients for this dataset have been demonstrated more recently, this result is taken as the baseline for evaluation. MCA is able to meet this baseline result with only linear regression of the produced embeddings when no modal sparsity is present. As modal sparsity is increased the correlation coefficient is reduced, and MCA and MMA results become similar with a minor increase in MCA vs. MMA.

## 7 Conclusion

Masked multimodal transformer architectures can learn embedding spaces robust to missing modalities. Multichannel attention was presented as an extension of the masked multimodal transformer model and incorporates modal-incomplete channels in the multihead attention mechanism. Using two well-known datasets, each with four modalities, both MMA and MCA are shown to learn high quality embedding spaces up to a modal sparsity of 0.4, corresponding to 1-2 modalities most frequently missing per sample for the datasets studied here. We find that MCA generally improves the quality of embedding spaces, recall metrics, and subsequent performance on downstream tasks.

Future directions for research could extend the architecture presented here to autoencoding or joint embedding predictive architectures for multimodal transformers, explore multimodal scaling laws with modal sparsity, and explore foundational model scale datasets for multimodal representation learning with sparsely aligned datasets.

## 8 Acknowledgments

The author thanks Clayton Rabideau, Gabrielle Griffin, Callum Birch-Sykes and Archis Joglekar for helpful conversations in the course of this study. The results shown here are in whole or part based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga.

## 9 Additional Figures

Figure: Figure 9.7: Uniformity of embeddings for CMU-MOSEI dataset.

Figure: Figure 9.8: Alignment of embeddings for CMU-MOSEI dataset.

Figure: Figure 9.9: Uniformity of embeddings for TCGA dataset.

Figure: Figure 9.10: Alignment of embeddings for TCGA dataset.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: The paper suggests novel contributions, but they are not sufficiently highlighted. Can you expand on how MCA distinguishes itself from existing methods in terms of theoretical innovations or novel mechanisms, beyond empirical performance? Providing explicit motivations for architectural choices would also make the contributions clearer and more impactful.
soundness: 3
strengths: 1. It addresses the challenge of missing modalities.

2. It extends the masked multimodal attention (MMA), improving its capability to handle incomplete data.

3. Improves the quality of learned embeddings.
summary: The paper proposes a method called Modal Channel Attention (MCA) to enhance masked multimodal transformer models for learning from multimodal data, even in the presence of missing modalities. The approach extends existing masked multimodal attention (MMA) by incorporating channels for incomplete modalities, which significantly improves the quality of learned embeddings.
weaknesses: 1. The contributions primarily emphasize performance improvements and extensions to the existing MMA. More detailed explanations of the specific contributions and the importance of the novelty would be beneficial. 

2. The technical section is quite brief (half a page), and it would be helpful if the figure showing the architecture and self-attention provided more detail. Moreover, part of the technical section is devoted to describing the architecture by listing details like activations, hidden size, and the number of attention heads. However, no explicit motivations are provided for these choices. Additionally, the figures should have been more curated to avoid text pixelation. Given the current shape of the paper, the novelties seem to be present, but the paper fails to properly highlight them, relying mostly on an empirical evaluation.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 1
questions: No
soundness: 1
strengths: The topic is very interesting and important.
summary: This paper is not ready for publication in ICLR
weaknesses: The presentation is poor.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Refer to the summary
soundness: 3
strengths: Refer to the summary
summary: **Summary of this paper:**

This work proposes an extension to the masked multimodal transformer model which incorporates modal-incomplete channels in the multihead attention mechanism called modal channel attention (MCA).

**Strengths:**

1.	This paper addresses an interesting problem by extending the masked multimodal attention model architecture to work with modality-incomplete datasets, a common scenario in real-world applications.

2.	The paper is straightforward.
 
3.	Experimental results demonstrate the effectiveness of the proposed method.

**Weakness:**

There is no visual comparison between the proposed architecture and previous structures, which would help clarify the improvements and differences.

**Comments, Suggestions And Typos:**

1.	It is recommended to add a figure in the introduction to briefly introduce the task of this paper, or to add a figure somewhere to compare the proposed model architecture with previous MMA architecture.

2.	Figure 1 is not good-looking, the authors are encouraged to improve the quality of the figure.

3.	For clarity and rigor, it is recommended to express important definitions or functions (i.e., modal sparsity) by formulas.

4.	The citation formats for figures differ in this paper.
weaknesses: Refer to the summary

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	In the MCA model, are important modalities treated differently from minor modalities? Or are different modalities given different weights in the attention mechanism?
2.	At which critical point does the performance of MCA significantly degrade as modal sparsity increases? Are there optimisation methods to maintain embedding quality at higher sparsity?
soundness: 3
strengths: 1. MCA provides a novel method for dealing with modal sparsity in multimodal transformers, addressing a common limitation in multimodal learning. The approach enhances the MMA model by adding a modality-aware attention channel, innovatively allowing robust embedding creation from incomplete datasets.
2. The study is rigorous, using two datasets (CMU-MOSEI and TCGA) to evaluate the model's effectiveness across varied tasks, including sentiment analysis and cancer classification. The experiments are thorough, with evaluations under multiple levels of modal sparsity and comparisons with established models.
3. This model has important implications for real-world applications where complete multimodal data is often unavailable, such as in medical data analysis and surveillance systems. MCA enables more resilient model applications by tolerating missing modalities without significant loss of performance, thus broadening the scope of multimodal transformers.
summary: Traditional multimodal learning methods often assume that all modes are uniformly present in training and reasoning, which may be impractical in practical applications. To solve this problem, the authors propose a modal channel attention (MCA) model, which is an extension of the masked multimodal attention (MMA) approach. By integrating a dedicated attention mechanism, MCA achieves efficient fusion and embedding of multimodal data, even in the case of sparse modes. We evaluate MCA on the CMU-MOSEI and TCGA datasets, and the results show that MCA outperforms MMA in creating aligned, uniform embeddings with different modal sparsity. In downstream tasks such as multimodal sentiment analysis and tumor type prediction, MCA also shows higher recall rates and classification/regression performance, making it a versatile tool for real-world multimodal data fusion.
weaknesses: 1. This paper does not explore in depth the interactions between modes to further explore the specific impact of different combinations of modes on MCA model performance. You may wish to consider exploring whether particular combinations are more conducive to improving model performance in order to gain a fuller understanding of the relative importance of each mode.
2. Although MCA is described as an efficient solution, this paper does not further quantify its computational efficiency and resource consumption on larger or more complex datasets to effectively assess the feasibility of its practical application.
3. The attention mechanism of the MCA model in dealing with missing modalities is still complex and not easy to understand intuitively. I would like to see the interpretability of the model in applications, which would help build user trust in the model's decisions.
4. The comparison experiments in this paper are limited to comparisons with the MMA model, which makes it difficult to fully demonstrate the relative strengths of the MCA model in multimodal approaches. You can evaluate the advantages and limitations of the MCA model more comprehensively by increasing the number of comparison experiments with multimodal methods other than MMA.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: * As far as I know, CMU-MOSEI primarily provides information in three modalities, including audio, video, and text. I am not sure what exactly the four modalities the authors are referring to when they use them.

* The core structure presented in this paper is very similar to previous work [1], leading to limited improvements and contributions. I suggest that the authors clarify the differences between them in detail to highlight their contribution.

[1] Shvetsova, N., Chen, B., Rouditchenko, A., Thomas, S., Kingsbury, B., Feris, R. S., ... & Kuehne, H. (2022). Everything at once-multi-modal fusion transformer for video retrieval. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 20020-20029).
soundness: 2
strengths: * This paper demonstrates the importance of how to cope with representation learning in the case of incomplete modalities in real applications.

* The authors provide extensive implementation details and interesting experiments to demonstrate the need for the proposed structure.

* The authors' review of the relevant literature is well organized.
summary: This paper presents a sparse multimodal fusion method in the presence of modal incompleteness. An extension to the masked multimodal transformer model is proposed which incorporates modal-incomplete channels in the multi-head attention mechanism called modal channel attention (MCA). The validity and usefulness of the method is verified by experiments on two multimodal datasets with different metrics and scenarios.
weaknesses: * This paper is clearly not ready for publication, including confusing typography, inadequate page counts for major sections, and formatting errors. For example, Fig and Figure are mixed up and line 221 is missing punctuation.

* The experimental section focuses on the comparative analysis of different metrics. However, the authors lacked an effective comparison with previous work to highlight the advantages of the proposed methodology.

* In addition, this study does not have enough qualitative analysis and ablation studies to explore the effect of different combinations of modal deletions in the proposed structure on performance.

Meta-review: It is clear to see that all the reviewers have concerns about the technical and experimental details of this paper, and all of them voted a low score for this paper, that is, one rejection, three weak rejections, yet one weak acceptance. Based on the overall scores of the whole ICLR papers, this paper will NOT be accepted for ICLR.

Decision: Reject