Learning Multimodal Data Augmentation in Feature Space
Authors: Zichang Liu
Zhiqiang Tang
Xingjian Shi
Aston Zhang
Mu Li
Anshumali Shrivastava
Andrew Gordon Wilson
Venue: ICLR 2023 poster (2023)

Abstract: The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is a defining feature of intelligent systems. While there have been promising advances in designing neural networks to harness multimodal data, the enormous success of data augmentation currently remains limited to single-modality tasks like image classification. Indeed, it is particularly difficult to augment each modality while preserving the overall semantic structure of the data; for example, a caption may no longer be a good description of an image after standard augmentations have been applied, such as translation. Moreover, it is challenging to specify reasonable transformations that are not tailored to a particular modality. In this paper, we introduce LeMDA, Learning Multimodal Data Augmentation, an easy-to-use method that automatically learns to jointly augment multimodal data in feature space, with no constraints on the identities of the modalities or the relationship between modalities. We show that LeMDA can (1) profoundly improve the performance of multimodal deep learning architectures, (2) apply to combinations of modalities that have not been previously considered, and (3) achieve state-of-the-art results on a wide range of applications comprised of image, text, and tabular data.

LEARNING MULTIMODAL DATA AUGMENTATION IN FEATURE SPACE
Zichang Liu
Zhiqiang Tang Amazon Web Services
Xingjian Shi
Department of Computer Science Rice University z171 8 erccu.edu
Amazon Web Services jishi Bamazon,
zgtang@amazon.coo com
Aston Zhang
Mu Li
Anshumali Shrivastava Department of Computer Science Rice University an hnsmumm i l ce. edu
Amazon Web Services
Amazon Web Services
astonz@amazon.comm
mli @ama zon, com
Andrew Gordon Wilson
New York University
Amazon Web Services
andrewgww cims. nyu.edu
ABSTRACT
The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is defining feature of intelligent systems. While there have been promis- ing advances in designing neural networks to harness multimodal data, the enor- success of data augmentation currently remains limited single-modality tasks like image classification. Indeed, it is particularly difficult to augment each modality while preserving the overall semantic structure of the data; for exam- ple, a caption may no longer be a good description of an image after standard augmentations have been applied, such translation. Moreover, it is challenging to specify reasonable transformations that are not tailored to particular modality. In this paper, we introduce LeMDA, Learning Multimodal Data Augmentation. an easy-to-use method that automatically learns to jointly augment multimodal data in feature space, with no constraints on the identities of the modalities or the rela- tionship between modalities. We show that LeMDA can (1) profoundly improve the performance of multimodal deep learning architectures, (2) apply to combi- nations of modalities that have not been previously considered, and (3) achieve state-of-tt ofte--f-the-rr art results on a wide range of applications comprised of image, text, and tabular data.
INTRODUCTION
Imagine watching a film with no sound, or subtitles. Our ability to learn is greatly enhanced through jointly processing multiple data modalities, such as visual stimuli, language, and audio. These in- formation sources are often so entangled that it would be near impossible to learn from only one modality in isolation cordingly, there have been substantial research efforts in recent years on developing multimodal deep learning jointly process and interpret information from different modalities once (Bal- trušaitis et al. 2017). Researchers studied multimodal deep learning from various perspectives such model architectures (Kim et al. 2021b Pérez-Rúa al. 2019 Nagrani al. 202T Choi & Lee, 2019), training techniques (Li al., 2021 Chen et al. (2119a). and theoretical analysis (Huang et al,,
significant constraint traditional machine learning approaches. Ac-
2021 Sun et al., (020b). However, data augmentation for multimodal learning remains relatively
unexplored (Kim et al. 2021a), despite its enormous practical impact in single modality settings.
Work done while interning Amazon Web Services.

---

Label
Description
(a)
Entailment
The plane is doing tricks while flying down. There is a plane in the air. The pilot is aware that the plane is doing loop. The plane is falling down.
(b)
Entailment
Entailment
(d)
Neutral
AntoAuggent Cifart0
Autoougment ImageNet
AutoAugmeni SVIIN
TrivialAugment
The top row shows four training samples drawn from SNLI-VE (Xie et al. 2019a). visual entailment dataset. Each text description paired with the image on the top lett. The task is to predict the relationship between the image and the text description, which can be "Entailment", "Neutral", or "Contradiction". The bottom row shows four augmented images generated by different image-only augmentation methods. If we pair the text description with the augmented images, observe mislabeled data. For example, the smoke loop is cropped out the image augmented TrivialAugment. The image does not match the description: "The pilot is aware that the plane doing a loop", as in data (c). However, the label of the augmented pair will still be "Entailment".
Figure 1:
Indeed, data augmentation has particularly proven its value for data efficiency, regularization, and Cubuk al. 2020 Müller & Hutter
improved performance in computer vision (Ho et al. 2019
2021 Zhang et al.) 2017 Yun et al. 2019) and natural language processing (Wei Zou 2019
Karimi al. 2002 Fadaee al. 2017 Sennrich et al.
2015 Wang & Yang 2015| Andreas
2020
Kobayashi 2018). These augmentation methods are largely tailored to a particular modality in
isolation. For example, for object classification in vision, we know certain transformations such as translations or rotations should leave the class label unchanged. Similarly, in language, certain sentence manipulations like synonym replacement will leave the meaning unchanged.
The most immediate way of leveraging data augmentation in multimodal deep learning is to sep- arately apply well-developed unimodal augmentation strategies to each corresponding modality. However, this approach can be problematic because transforming one modality in isolation may lead to disharmony with the others. Consider Figure which provides four training examples from SNLI-VE (Xie et al., 2019a). vision-language benchmark dataset. Each description is paired with the image on the top left, and the label refers to the relationship between the image and description.
The bottom row provides four augmented images generated by state-of-the-aar image augmentation methods (Cubuk et al. 2019 Müller & Hutter 202I). In the image generated by Auto-Augeent Cifar10 and AutoAugment- SVHN, the plane is entirely cropped out, which leads to mislabeling for data (a), (b), (c), and (d). In the image generated by AutoAugment-Imagee
due to the change in smoke color, this plane could be fire and falling down, which leads mislabeling for data (a) and (d). In the image generated by TrivialAugment (Müller & Hutter 2021). recent image aug- mentation method that randomly chooses one transformation with random magnitude, the loop cropped out, which leads to mislabeling for data (a) and (c). Mislabeling can be especially problem- atic for per-parameterized neural networks, which tend confidently fit mislabeled data, leading to poor performance (Pleiss et al. 2020).
There are two key challenges in designing general approach to multimodal data augmentation. First, multimodal deep learning takes input from diverse set of modalities. Augmentation trans- formations can obvious for some modalities such as vision and language, but not others, such sensory data which often numeric categorical. Second, multimodal deep learning includes a diverse set tasks with different cross-modal relationships. Some datasets have redundant or totally correlated modalities while others have complementary modalities. There is no reasonable assumption that would generally preserve labels when augmenting modalities isolation.
In this work, propose LeMDA (Learning Multimodal Data Augmentation) general multi- modal data augmentation method. LeMDA augments the latent representation and thus can be ap- plied to any modalities. We design the augmentation transformation learnable module such that
2

---

it is adaptive to various multimodal tasks and cross- relationships. Our augmentation module is learned together with multimodal networks to produce informative data through adversarial train- ing, while preserving semantic structure through consistency regularization. With no constraints over the modalities and tasks, one can simply plug-and-play LeMDA with different multimodal architectures. We summarize our contributions as follows.
In Section 3 we introduce LeMDA, novel approach to multimodal data augmentation. Section 3.1 shows how to use LeMDA with multimodal networks, and Section 3.2 de- scribes how to train the augmentation module to produce informative and label preserving data. The method notable for several reasons: (I) it can be applied to any modality combinations; (2) it is attractively simple and easy-to-use -ooee;;; (3) it is the first augmentation method to be applied to the joint of text, image, and tabular data, which is essentially un- charted territory.
In Section we show that LeMDA consistently boosts accuracy for multimodal deep learn- ing architectures compared a variety of baselines, including state-of-the-- input aug- mentation and feature augmentation methods.
In Section [..4 we provide an ablation study validating the design choices behind LeMDA. In particular, we study the architecture of the augmentation module, and the effects of con- sistency regularizer. We demonstrate that the consistency regularizer clearly outperforms 122 regularization (Tang al. 2020b
2 BACKGROUND AND RELATED WORK
Multimodal network architectures. Multimodal deep learning architectures are categorized as performing early or late fusion, depending on the stage of combining information from each modal- ity. In early fusion, the network combines the raw input or token embedding from all the modalities. Early fusion architectures can be designed to exploit the interaction between low-level features, making it good choice for multimodal tasks with strong cross-modal correlations (Barnum et al.) 2020 Gao et al. 2020). For example, there exist low-level correspondence in image captioning task because different words in the caption may relate to different objects in the image. We note that feature-space augmentation procedures are typically computationally intractable on farlinfusion chitectures, because early fusion would require combining a large number of latent features, such as long sequence of token embeddings. On the other hand, in late fusion, the focus of our work, input from each modality is independently processed by different backbones. The representations provided by different backbones are fused together in later layers. often just before the classifier layer (Shi et al., 2021a: Wang et al., 2017 Schönfeld et al. 2019 Mahajan et al. 2020). This design straightforward to apply to any new modality and any multimodal task. Late fusion often uses pre-trained networks as backbones in each modality, making it more computationally tractable. In both early and late fusion, there are a variety of methods to fuse information. Standard approaches include (I) feed all modalities as token embedding into the network, (2) perform cross-attention between modalities, (3) concatenate representations from all modalities, and (4) combine the pre- dictions from each modality in an ensemble (Baltrušaitis al. (017). Researchers usually design the multimodal network by considering the task objective, the amount of data available, and the computation budget (Shi et al. 22021b] Chen al. 2019b) [ al. 2022 Tsai et al. 2019 Mahajan
& Roth 2020). Baltrušaitis et al. (017) provides further readings.
Data augmentation for single modality tasks. Data augmentation is widely adopted in vision and natural language tasks. In vision, we can manually intervene on a per-task basis to apply trans- formations that should leave our label invariant - e.g., translations, rotations, flipping. cropping, and color adjustments. A transformation on one task may not be suitable for another: for example, flipping may be reasonable on CIFAR-10, 10, but would lose semantic information on MNIST, because flipped *6" becomes a '9'. Accordingly, there are a variety of works for automatic augmentations in vision, including neural architecture search (Ho et al. 2019; Cubuk et al., 2020), reinforcement learning (Cubuk et al. 2019), generative modelling (Ratner et al. 2017), mixing aspects of the ex- isting data (Zhang et al. 2017 Yun et al. 2019). and adversarial training for informative examples
Fawwi et al.; 20166] Goodfellow et al.) 2015; Zhang et al. 2019 Suzuki 2022; Tang al. 2020b
Tsai et al. (017)_ Similarly, in natural language processing there are a variety of standard interven-
tions (replacement, deletion, swapping) (Wei & /ou
2019 Karimi et al. 2021 Fadaee al. 2017).
3

---

Algorithm LeMDA Training
Input:
Task network before fusion before: Task network after fusion Fafter; Augmentation net-
work G: Training set X: Task loss function L; Consistency loss Leonsisti
while F not converged do
Sample mini-batch from X'
Compute < beeore (z:
Generate augment feature ((2)
f- Fafter (z), Dg <- (g
Update the augmentation network & by stochastic gradient - VV((e) + Leonsist 0g)
Update the task network JF by stochastic gradient L() + V (Úig)
end while
and more automatic approaches such as hack-translation (Sennrich et al.
2013), context augmen-
tation (Wang & Yang 2015 Andreas 2020; Kobayashi 2018), and linear interpolation of training data (Sun et al. 2020a). Data augmentation less explored for tabular data, but techniques in vision, such mixup (Zang et al. 2017) and adversarial training (Goodiellow et al. 2015) have recently been adapted to the tabular setting with promising results (Kadra et al.) 2021). Latent space augmen- tation is much less explored than input augmentation, it is less obvious what transformations to apply. To augment latent vectors produced by passing data inputs through a neural network (feature space augmentation), researchers have considered interpolation, extrapolation, noise addition, and generative models (Verma et al. 2019 Liu et al. 2018 Kumar et al. 2019).
Multimodal data augmentation. There are a small number of works considering multimodal data augmentation, primarily focusing on vision-text tasks. In visual question answering, Tang et al.] (2020a) proposes to generate semantic similar data by applying fack-translation on the text and adversarial noise on the image. [Wang et al. (202T) generates text based on images using variational autoencoder. In cross-modal retrieval, Gur et al. (2021) query similar data from external knowledge sources for cross-modal retrieval tasks. The state-of-the-art augmentation procedure for visual-language representation learning generates new image-text pairs by interpolating between images and concatenating texts a method called MixGen (Hao et al. 2022).
All prior work multimodal data augmentation relies on tailored modality -specii transforma- tions. By contrast, our proposed approach is fully automatic and can be applied to any arbitrary modality. Indeed, for the first time, consider augmentation jointly over the tabular, image, and language modalities. Moreover, even for image-text specific problems, we show that our approach outperforms MixGen, the state-of-the-arn tailored approach.
3 LEMDA: LEARNING MULTIMODAL DATA AUGMENTATION
We now introduce LeMDA, a simple and automatic approach to multi-modal data augmentation. LeMDA learns an augmentation network G, along with the multimodal task network F generate informative data that preserves semantic structure. In Sections 3.1 and 3.2 we describe how we learn the parameters the augmentation and task networks, respectively. We summarize the training algorithm for LeMDA in Figure 2 and Algorithm [ In Section 3.4 we provide intuition for the consistency loss. Finally, in Section 3.3 we describe how we design the augmentation network.
3.1 TRAINING THE TASK NETWORK
The task network can be divided into parts at the fusion layer F(x) Fafter (Foefore (z)) where Fbefore denotes the layers before fusion, Fatter denotes the layers after the fusion. Given a train- ing sample
pass x until the fusion layer and obtain the latent features for each modality { Fzefore(z) where is the number modalities. Taken {2x N as inputs, the augmenta-
tion network & generates additional latent vectors [({2),N 1N Both { and G({,) M )) are fed through the rest of target network Fafter distinct training data. Then, the task network is trained in the standard way, taking the task loss function on both original data and augmented data, to find min (l=()((()) + T(jg)) where Fatter (J Feefore (2:)) and ÿG Fatter (F (x))).
4

---

9
yG
VL + (Z)
Task Network After Fusion
(
((22)
((2))
Augmentation Network
21
Zz
Z
Modality A Encuder
Mudality Encoder
Modality C Encoder
Input from Modality
Input from Modality
Input Modality
(L) Vinonns (D.5g)
yg
Task Network After Fusion
((z))
((22)
((z))
Augmentation Network
Z2
Z3
Modality Encoder
Modality B Encoder
Modality Encoder
Input Modality from Modality Input from Modality
Figure 2: LeMDA training as described in Algorithm [ Top: the training process for the task network. Latent representations for each modality 2; passed into the augmentation network, which generates new latent vector for each modality. Both original features and augmented features passed into the rest of the task network. Bottom: the training process for the augmentation network. The augmentation network is trained to maximize task loss while minimizing consistency loss. We describe our standard choices for fusion Section [ and the design of our augmentation network in Section 3.3
3.2 TRAINING THE AUGMENTATION NETWORK
Inspired by adversarial data augmentation, to maximize the task loss such that the task network's representation is encouraged to be updated by the augmented data. At the same time, we introduce consistency regularizer that encourages similar output distribution given the original data and the augmented data to preserve the semantic structure. Formally, we find max Ex-x ([(gg) + min E
optimize parameters for the augmentation network
E [coossse (Ú, ûg) where Lecnsist (j), Úg) denotes divergence metric between the logit outputs on original data and on augmented data such as the Kullback-Leibler divergence.
Confidence masking. For classification problems, we apply the consistency term only to the sam- ples whose highest probability greater than threshold a. the task network can't make confident prediction, it is unlikely the prediction provides good reference to the ground truth label.
Design decisions. The simplicity and generality of this approach, combined with its strong empirical performance in Section 4 are LeMDA's most appealing features. The few design decisions for training involve how the consistency regularizer should be defined and to what extent it should be applied. For example, Le distance of the augmented feature vector to the original feature vector as a proxy for preserving the label of the augmentation. We provide ablations of these factors in Section 4.4
alternative a KL-based consistency regularizer,
could minimize the

---

3.3 THE DESIGN OF AUGMENTATION NETWORK
The augmentation network can take various forms depending on the multimodal learning tasks and the fusion strategies. In our experiments, tation network, since VAEs have generally been found effective for augmentation purposes (Tang al. 2020b). We consider two architectural choices:
variational autoencoder (VAE) as the augmen-
MLP-VAE: The encoder and decoder of VAF are MLPs. { are concatenated as the input.
Aitention-VAE: The encoder and decoder are made of self-attention and feedforward networks. {2i are treated as N tokens where each token has an embedding 2;.
There are two loss terms in the standard VAE, the reconstruction loss, and the KL divergence reg- ularizer. We only adopt the KL regularizer on the encoder distribution. The updating step for augmentation networks is [[(g) + V Leoosist (U: (jp) + + +V where TAAE refers to the KI. divergence regularizer on the latent encoding distribution.
The major deciding factor between MLP-VAE and Attention- VAE is the multimodal task network architectures. With late fusion architectures, which is the primary focus of this paper, 2; refers to the representation from a single modality backbone (e.g. CLS embedding from BERT model), and N is the number of modalities or the number of backbone models. We can concatenate { as one vector input to MLP-VAE, or we can treat )= as a sequence of N tokens to Attention- VAE. Attention- VAE may be less intuitive here because N usually small number in late fusion architectures( 2 or 3 in our experiment). We provide a performance comparison between these two architectures in Section 4.4 On the other hand, for early fusion architectures, 2; could be a sequence of token embedding for text or sequence of patch embedding for an image. Concatenation will result really high dimension input, which makes MLP-VAD less favorable.
3.4 INTUITION ON WHY CONSISTENCY REGULARIZER DISCOURAGES MISLABELED DATA
In Figure we provide intuition for the consistency regularizer using a simple illustrative binary classi- fication. Darker background corresponds to higher task training loss, the solid green line is the ac- tual decision boundary, and the dashed green line is the model's decision boundary. Starting from point in feature space, moving to DI and D2 would provide similar increase in task loss and thus are equally favored by the adversarial loss term. How- ever, D2 crosses the model's decision boundary, and thus would be heavily penalized by the consistency regularizer
0
we would hope, since such point is likely to have different class label. On the other hand, an L2 regularizer between the original and augmented points in feature space would have no preference between D1 and D2, as they are an equal distance away from the starting point. Empirically, in Section 4 we see the consistency loss confers accuracy improvements over both pure adversarial training and L2 regularizer.
Figure 3: Motivation for the consistency reg- ularizer. The solid and dashed green lines are the ground truth and model decision bound- aries, respectively. Darker background corre- sponds to higher loss for the task network. We intuitively prefer D1, because the aug- mented point should be informative but pre- serve the same label. The consistency loss will prefer DI over D2, because D2 crosses the model's decision boundary, even though both points incur the same training loss.
Similar intuition is in Suzuki 2022), which uses the logits distribution from the teacher model (an expo- nential moving average over the model's weights) as the soft target such that the augmented data is still recognizable by the teacher, and Xie al. (2019b), which designs the unsupervised training objective to encourage similar logits for augmented data.

---

Dataset
Train
#Test
Metric
Image
Text
Tabular
Hateful Memes
7134
1784
Accuracy
Food
67972
22716
Accuracy
SNL.-VE
529527
17901
Accuracy
Pettinder
11994
2999
Quadratic Kappa
Melbourne Aurbab
18316
4579
Accuracy
News Channel
20284
5071
Accuracy
7
Wine Reviews
84123
21031
Accuracy
4
Kick Starter Funding
86502
21626
ROC-AUC
Table 1: This table provides a summary of the source, statistic, and modality identity.
4 EXPERIMENTS
We evaluate LeMDA over diverse set of real-world multimodal datasets. We curate list of public datasets covering image, text, numerical, and categorical inputs. Table [ provides a summary of
the source, statistic, and modality identity. We introduce baselines in Section 4.1 and describe
experimental settings in Section [42) We provide the main evaluation result Section [.3 Finally, we investigate the effects of the consistency regularizer and the choices of augmentation model architecture in Section 4.4
4.1 BASELINES
To the best of our knowledge, there exist general purposse multimodal augmentation methods. We compare against a diverse set of otate-of-the-art data augmentation methods from vision, language, and vision-text tasks. We additionally consider baselines for feature augmentation, since LeMDA augments in the feature space. Finally, we compare with state-of-the-ard multimodal augmentation methods from the vision-text tasks, although we note, unlike LeMDA, these methods are not general purpose and cannot be directly applied to our datasets that have tabular inputs.
Input Augmentation. We apply state-of-the-an art input augmentation independently on the data from each modality. For images, we use TrivialAugment (Müller & Hutter (0211) simple and effective method for image classification tasks. For text, we apply EDA (Wei &
Zou 2019) and AEDA (Karimi et al. 2021). We randomly sample one transformation from transformations proposed in EDA and AEDA with a randomly generated magnitude.
Mixup. Mixup was originally proposed to perform interpolation between two images in the training data for image classification. We adopt the original Mixup for images and numerical features and extend for text and categorical features. Specifically, given pair of data, we construct the mixed data as follows. We generate random number j uniformly between 0.0 1.0. j < a, we use the first data, else we use the second data.
Manifold Mixup. Manifold Mixup (Verma et al., 2019) performs interpolation between hidden representations and thus can be applied to all modalities. We applied Manifold Mixup to the exact feature in the multimodal network as LeMDA.
MixGen. MixGen (Hao et al.] 2022) a of data augmentation designed specifically for vision-text tasks. Mix generates data by interpolating images and concatenating text. We apply MixGen to datasets only consisting of images and text.
4.2 EXPERIMENT SETUP
We use Multimodal-Net (Shi al. 2021a) la) for all the datasets except SNLI-VE. Multimodal-Net passes input from each modality through separate backbones, concatenates the representation(e.. the CLS embedding) from all backbones, and passes them through fusion MLP. We use the default hyper-parameters provided by Multimodal-Nae and plug LeMDA before the fusion layer. We use ConvNet as the image backbone and ELECTRA as the text backbone.
To further demonstrate LeMDA's generalizability, we evaluate LeMDA with early fusion architec- tures ALBEF (Li et al. 2021 on SNLI-VE, ALBEF performs cross-altention between image patch
Coode available http:///ttthh..oommccmmm///////

---

Multimodal Network
Input Augmentation
Manifold MixUp
Mixup
MixGen
LeMDA
Hateful Memes
0.6939
0.7057
0.6939
0.6878
0.7510
0.7562
Food101
0.9387
0.9432
0.9400
0.9390
0.9432
0.9452
Petfinder
0.2911
0.3236
0.3244
0.3492
0.3537
Melbourne Airbnb
0.3946
0.3978
0.3966
0.3840
0.4047
News Channel
0.4754
0.4745
0.4723
0.4757
0.4798
Wine Reviews
0.8192
0,8212
0.8143
0.8126
0.8262
Kick Starter Funding
0.7571
0,7572
0.7597
0.7578
0.7614
SNLI-VE
0.7916
0.7931
0.7957
0.7929
0.7950
0.7981
Table LeMDA not only significantly increases accuracy over the original architectures but also outperforms all baselines.
embeddings and text token embeddings. We keep all original configurations except the batch size due to limitations computation memory. We set the batch size to be half of the default. We load the 4M pre- praitrri checkpoints. In this setting, we apply LeMDA before the cross-attention layer. The augmentation network augments every image patch embedding and every text token embedding.
For LeMDA, we the confidence threshold for consistency regularizer a as 0.5, and study this choice in Section 4.4 For our baselines, we follow the recommended hyperparameters. For Mixup and Manifold Mixup, we set a as 0.8, and for MixGen, we À as 0.5.
4.3 MAIN RESULTS
We summarize the performance comparison in Table Plugging LeMDA in both Multimodal-Nee and ALBEF leads to consistent accuracy improvements. There are also some particularly notable improvements, such 6% increase in accuracy for both Hateful Memes and Petfinder. Table illustrates how LeMDA performs comparing to the baselines. We see that single modality input augmentation methods can hurt accuracy, for example, on News Channel, in accordance with the intuition from our introductory example in Figure [ Mixup also can hurt accuracy. for example, on Wine Reviews. Similarly, in the latent space. Manifold Mixup fails improve accuracy across datasets. On Melbourne Airbnb and Wine Reviews, Manifold Mixup results in accuracy drops. On the contrary, LeMDA consistently improves upon original architectures and provides clearly better performance than wide range of baselines.
4.4 ABLATION STUDY
We now perform three ablation studies to support the design choice of LcMDA.
Dataset
No Regularizer
Consistency
I.2
Consistency L.2
Hateful Memes
0.7433
0.7562
0.7472
0.7545
Food101
0.9433
0.9452
0.9415
0.9438
Petfinder
0.3369
0.3537
0.3420
0.3461
Melbourne Airbnb
0.3935
0.4047
0.3987
0.4051
News Channel
0,4851
0,4869
0.4869
0.4894
Wine Reviews
0.8228
0.8263
0.8255
0.8262
Kick Starter Funding
0.7609
0.7614
0.7604
0.7614
The effects of regularizer choice. Regularization over the augmentation network generally lead to better performance. Consistency regularizer consistently outperforms a standard L2 regu- larizer in feature space. Moreover, combining the consistency regularizer with an L2 regularizer improves over only using an L2 regularizer.
Table 3:
Augmentation Network Regularizer. We argue in Section 3.4 that consistency regularizer helps preserve the semantic structure of augmentations. In Table we see that this consistency regularizer significantly improves performance, and also outperforms L2 regularization in feature space. While L2 regularization attempts to keep augmented features close in distance the original as proxy for semantic similarity, the consistency regularization has access to the softmax outputs of the target and augmentation networks, providing direct information about labels.
8

---

Dataset
Augmentation
MLP-VAE
Attention VAE
Hateful Memes
0.6939
0.7562
0.7483
F0od101
0.9387
0.9452
0,9443
Petfinder
0.2911
0.3537
0.3456
Melbourne Airbnb
0.3946
0.4047
0.4031
News Channel
0,4754
0.4798
0,4733
Wine Reviews
0.8192
0.8262
0.8250
Kick Starter Funding
0.7571
0.7614
0.7586
Table 4: Both MLP-VAE and Attention-VAI augmentation networks provide significant gains no augmentation. MLP-VAE outperforms Autention- VAE in the late fusion setting because the input of augmentation networks is only 2 or 3 latent representations.
Dataset
0
= 0.3
0.5
0).8
Hateful Memes
0.7410
0.7443
0.7556
0.7371
Food101
0.9431
0,9438
0.9447
0,9438
Petfinder
0.3243
0.3462
0.3676
0.3497
Melbourne Airbnb
0.3988
0.3964
0.3988
0.3964
News Channel
0.4869
0,4851
0.4869
0,4695
Wine Reviews
0.8228
0.8274
0.8275
0.8274
Kick Starter Funding
0.7614
0.7617
0.7620
0.7618
Table 5: The influence of confidence-based masking. a = 0 indicates no masking such that consis- tency loss calculated with all data. We see that filtering out low-confidence data leads better end-to-end accuracy.
Architecture Difference. We consider the two augmentation architectures introduced in Sec- tion 3.3, MLP-VAE and Attention-VAL In Table we see both architectures increase performance augmentation. We also see that MLP-VAE generally outperforms Attention- VAF. We sus- pect the reason is that Multimodal-Net passes the concatenation of N latent vector into fusion layers, where N is the number of modalities or in our experiments), For Attention-VAE, this means that the input is only 2 or tokens. However, we note that MLP-VAN is not reasonable for ALBEF, since it would require concatenating thousands of tokens.
Confidence Masking, Here, we investigate the effect of confidence masking, as well as the choice of a Table 5 a means no masking, and all training data are used calculate consistency loss. We see that confidence masking generally leads to higher accuracy, and that the performance not particularly sensitive to the precise value of
4.5 THE RELATIONSHIP BETWEEN MODALITIES
We can categorize the relationship between available modalities by looking at P(ylx) where y y and y is the target domain. Let x (x1, T2,..., ,NN consist of N modalities.
Perfect Correlation P(y|z) P(y] Essentially, modality alone provides enough informa- tion to make the right prediction. Nevertheless, data still comprises multiple modalities for reasons such easier training (Huang al.) 2021). One example could be Food101, where the task is to predict the food from the text of a recipe and the photo of the food.
Complementary P(y|=) P -2,--..,T,, NNN }). This category suggests that information aggre- gated from all modalities is necessary to make the right prediction. Each modality is complementary each other, and missing one modality would lead information loss. One example could be Hate- ful Memes, where only the combined meaning of text and image indicates harmful content.
The design for LeMDA does exploit any assumption over the cross-modal relationship. We observe from Table 2 that LeMDA consistently improves performance regardless of the relationship.
5 CONCLUSION
Jointly learning from multiple different modalities will be crucial in our quest to build autonomous intelligent agents. We introduce the first method, LemDA, for jointly learning data augmentation

---

across arbitrary modalities. LeMDA is simple, automatic, and achieves promising results over wide range experiments. Moreover, our results provide several significant conceptual findings about multimodal data augmentation in general: (1) separately augmenting each modality performs much worse than joint augmentation; (2) although feature augmentation is less popular than in- put augmentation for single-modality tasks because it is less interpretable, feature augmentation is particularly promising for mogality-agnostic settings; (3) a learning-basee based multimodal augmenta- tion policy can outperform even tailored augmentations, and significantly improve accuracy when augmentation transformations are not obvious such as for categorical data.
Our investigation has primarily focused on late-fusion architectures, showing strong results over wide range of settings. In general, applying feature augmentation strategies to early-fusion ar- chitectures an open question. Early fusion combines large number of latent features (e.g., long sequence of token embeddings), resulting in typically intractable computational costs for aug- menting every latent feature. Our experiment with an early-fusion architecture shows however that developing more efficient augmentation networks, or selectively generating only few important latent vectors, is a promising direction future work.
10

---

REFERENCES
Jacoh Andreas. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7556-7566, Online, July 2020. URL https:
Association for Computational Linguistics. doi: 10218663/11/0220ac--mm //AclanthoLogy.orr//////3
main.676
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Moreney. Multimodal machine learning:
A survey and taxonomy, 2017. URL https:::////. //axxvv wttp://ogggab////aabbab 00077955094666 09406
George Barnum, Sabera Talukder, and Yisong Yue. On the benefits of early fusion in multimodal representation learning, 2020. URL/ttps::/ www.v.ogg/.....o.......o..o.o http://bb//21111 07191
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng. and Jingjing Liu. UNITER: learning universal image-text representations. CoRR, abs/1909.11744 2019a. URL http:htt:::attt
org. h/////bb..a 9099 11740.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning, 2019b. URL https:
11740
Jun-Ho Choi and Jong-Seok Lee. Embracenet: robust deep learning architecture for multimodal classification. Information Fusion, 51:259-277 270, 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 18613-18624. Curran Associates, Inc., 2020. URL httpps://proceed/ wew.innn.inniiinneneeeeeeeeeee.e 2tt.:////c..01aaaa1aa/////////00
Paper.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. Data augmentation for low-resource neu- ral machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 567-573, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 16655/11/17-220000 URL https: //aclanthology.orgr
Alhussein Fawzi, Horst Samulowitz, Deepak Turaga, and Pascal Frossard. Adaptive data augmenta- tion for image classification. In 2016 IEEE International Conference on Image Processing (ICIP), pp. 3688-3692 2016. doi: 10.1 109//CI12201..7533348
Jing Gao, Peng Li, Zhikui Chen, and Jianing Zhang. A Survey on Deep Learning for Multimodal Data Fusion. Neural Computation, 3265),829-864, 05 2020. ISSN 0899-7667. doi: 10.1162/ 2ttp://ccca__01277
neco_a.01273. URL htttp:://oo..rrg///
Ian J.
Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. CORR, 4at.//122...7, 2015.
Shir Gur, Natalia Neverova, Chris Stauffer, Ser-Nam Lim, Douwe Kiela, and Austin Reiter. Cross- modal retrieval augmentation for multi-modal classification, 2021. URL https://arxiv. org/abs 2104. 08108
Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. Mix- gen: A new multi-modal data augmentation, 2022. UtL/Rttps::/xxxvvorrr wttp:////r.bbb......//
08358
Daniel Ho, Eric Liang, Ion Stoica, P. Abbeel, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. In ICML, 2019.
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably), 2021. URL htthtt:s htt.tttsss///itttttst /ttp/ii..o.gg///////........ abs/ 21 06.066.66 04538
11

---

Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL net/ d
Akbar Karimi, Leonardo Rossi, and Andrea Prati. AEDA: easier data augmentation technique for text classification. CoRR, abs/2108. 13230, 2021. /tt.ppppe///avvv..or///b////////
13230
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convo- lution region supervision. In ICML, 202
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languagg transformer without convo-
lution or region supervision, 2021b. URL
Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic rela- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pupers), pp- 452 457-457, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: ac
10.18653/v 1/N18/ 2072. URL http:p////e.aaanoooooooooor/////
2072
Varun Kumar, Hadrien Glaude, Cyprien de Lichy, and William Camphell. A closer look at feature space data augmentation for few-shot intent classification. CoRR, abs/1910.04176, 2019. URL,
http::/arx:n wttp:/bs.///...o...////// 04176
Junnan Li, Ramprasaath Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. CoRR, abs/2107.07651, 2021. URL htttps://erxiv.orr///a
07657
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-tnaining for unified vision-language understanding and generation, 2022. URL https: //arxiv.orgy abs//ab///a/ httgg//bb/220... 12086
Xiaofeng Liu, Yang Zou, Lingsheng Kong, Zhihui Diao, Junliang Yan, Jun Wang, Site Li, Ping Jia, and Jane You. Data augmentation latent space interpolation for image classification. In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 728-733, 2018. doi: 10.1 N0.1109/ICPR.20 201.8555000...
Shweta Mahajan and Stefan Roth. Diverse image captioning with context-object split latent spaces. In Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 3613-3624. Curran Asso- ciates, Inc., 2020. URL http://ss/s/pprcpeeinneeee rewriiinnnnnuuuruuneeee neurips ps. http:/swwww.pp..pppppppeeeeeee cc//paper/222//file) (11.5222aaa1222233323322333
.ddf
Shweta Mahajan, Iryna Gurevych, and Stefan Roth. Latent normalizing flows for many-to-many cross-domain mappings, 2020. htl.https:/// /ttp/aaiiiv/..////v///ee////aata http://wwww....///////2202222
Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmen- tation, 2021. TRRLhtpps:/ /ttp:/pppa///axxi.ooo///////////
10158
Arsha Nagrani, Shan Yang, Anurag Arna,, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. CoRR, abs/2107.00135, 2021. URL https://arxiv:
org//bs/ 21 07. 00135
Geoff Pleiss,
Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:17044-17056, 2020.
MannMM Pérez-Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Ju- rie. Mfas: Multimodal fusion architecture search, 2019. URL htttt:t///:/aate //apxiiv////......//aa/aa org/ abs/ 1903. 06496
12

---

Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn- ing to compose nomain-specific transformations for data augmentation. Advances in neural in- formatiin processing systems, 30, 2017.
Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata, Cross-
linked variational autoencoders for generalized zero-shot learning, 2019. URL https://
openreview Seeenrreiee.eee/formmm
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data, 2015. URL BBLLLt/ss///aad...crr/////////
Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alex Smola. Multimodal automl on struc- tured tables with text fields. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021a.
Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alexander J Smola. Benchmarking multi- modal automl for tabular data with text fields. In Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks, 2021b.
Lichao Sun, Congying Xia, Wenpeng Yin, Tingting Liang, Philip S. Yu, and Lifang He. Mixup- transformer: Dynamic data augmentation for nlp tasks, 2020a. URL httpt:t://t.ii..rrrr
foos/201..o 02394
Xinwei Sun, Yilun Xu, Peng Cao, Yuqing Kong, Lingjing Hu, Shanghang Zhang, and Yizhou Wang- TCGM: an information-theoreti framework for sumi-supervised multi multimmmmlliiiiiii modality learning. CoRR,
abs/2007.06793, 2020b. (RRL/ttts:///xxiiii org absp//0/ss////0// 06793
Teppei Suzuki. Teachaugment: Data augmentation optimization using teacher knowledge, 2022.
URL https:://ts/iiii www.vvorg/aaaaa abtpabss.22.../ 12222.22111.1
Ruixue Tang, Chao Ma, W. Zhang, Qi Wu, and Xiaokang Yang. Semantic equivalent adversarial data augmentation for visual question answering. ArXiv, abs/2000709553 009002,,.999.......0 2020a.
Zhiqiang Tang, Yunhe Gao, Leonid Karlinsky, Prasanna Sattigeri, Rogerio Feris, and Dimitris
Metaxas. Onlineaugment: Online data augmentation with less domain knowledge, 2020b. URL
https: wttp://wwwwppp////////////
Yao-Hung Hubert Tsai, Liang- Kang Huang, and Ruslan Salakhutdinov. Learning robust visual- 05908
semantic embeddings, 2017. URL
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-PPillppe Morency, and Ruslan Salakhut- dinov. Learning factorized multimodal representations. In International Conference on Learning Representations, 2019. URL httthttss:. /tttps:///perrreveee....
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp- 6438 6447. PMLR, 09-15 Jun 2019. Vermal 9a. html
URL https: http:////cccceeeeeeeeee mir pww.s/prressvvvv
Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for 0347C
image-text matching tasks, 2017. URL http::ttt/s//iiiiaai www.oorg///////oo.///.......e/se
William Yang Wang and Diyi Yang. That's so annoying!!!: A lexical and frame-semantic em- bedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pp. 2557-2563, Lisbon, Portugal, September 2015, Association for Com- putational Linguistics. doi: 10. 00.18653v1/DD.... 1306. URL. https:: /ttp:://cclhattthoo ogy.org/
1305-1306]
Zixu Wang, Yishu Miao, and Lucia Specia. Cross generative aaggr augmetttion for visual question answering, 2021. URL /tt.a//tss////rrrr..r//////////0
13

---

Jason Wei and Kai Zou, Eda: Easy data augmentation techniques for boosting performance text classification tasks, 2019. URL htttps://axxiv..oot.t httpaaass/11.1... 111996
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- grained image understanding. arXiv preprint arXiv:1901.06776. 2019a.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data aug- mentation for consistency training, 2019b. URL https:: //axiv v org/ orggabse arggabbs1904. 12848
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe. and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceed- ings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rX1v:1710.99122, 2017.
Xinyu Zhang. Qiang Wang, Jian Zhang, and Zhao Zhong, Adversarial autoaugment. arXiv preprint arXiv: 19/2.1918818 2019.
14

---

A MORE DETAILS ON THE DESIGN
A.1 DETAILED ARCHITECTURES THE AUGMENTATION NETWORK
We consider VAE architectures in LeMDA, depending the architecture of the task network. The latent dimension in VAE is set as 8. We adopted the KL divergence regularizer on the encoder distribution. Note that do use the reconstruction loss between the input and the output. In MLP-VAE, the encoder and decoder are standard fully connected layers with ReLU the activa- tion function. Dropout is used with p 0.5. In Attention VAE, the encoder implemented as tor TransformerEncoder. We set numlayers as 4 and nhead as 8. One fully connected layer is used to The decoder is symmetric as the encoder. Features from all modalities are treated as token embedding with no cross-attention.
We use VAE for its simplicity. The main focus of this paper is demonstrate the effectiveness of a learnable augmentation network for multimodal learning. Other generative models, such as diffusion models and GANs, are also valid architectures. The main concern may lie in efficiency, and we leave this direction as future work.
A.2 IMPLEMENTATION DETAILS OVER THE TRAINING PROCEDURE
In practice, we iterative train the task and augmentation networks using the same batch of training data. Specifically, we perform two separate forward passes using Fatter for easy implementation with py pyTorch Autograd. We use two optimizers, one for the task network and one for the augmentation network.
B EXPERIMENT DETAILS
B. ADDITIONAL STUDIES ON THE TRAINING COST
One limitation of learning- based approach is the extra training cost. LeMDA optimizes the aug- mentation network along with the task network and does incur extra training costs. Here, we in- vestigate the training throughput to provide more complete understanding of the method. We summarize the training throughput(it/sccom for LcMDA compared to other baselines.
Table As expected, we observe lower throughput
Multimodal Network
Input Augmentation
Manifold MixUp
Mixup
MixGen
LeMDA
Hateful Memes
2.39
2.17
2.35
1.63
2.35
1.41
Foo
4.27
4.46
4.31
4.48
4.47
2.21
Petfinder
2.36
2.29
2.95
2.36
1.87
Melbourne Airbnb
5,66
5.94
5.59
5.69
4.13
News Channel
8.14
7.18
7.31
7.12
5.12
Wine Reviews
12.54
11.60
11.89
11.46
6.28
Kick Starter Funding
12.37
12.57
12.62
12.21
6.69
Table 6: This table summarizes the training throughput, measured it/second. Experiments were conducted on a server with 8 V100 GPU. As expected, learning-base based approach incur higher training cost.
However, efficiency can be improved. The most straightforward direction to reduce the frequency of updating the augmentation network. Currently, the augmentation network is updated every iter- ation. However, the parameters for our task network change slowly, especially the later stage of training. We leave this part future direction.
15

---

B.2 ADDITIONAL STUDIES ON THE HYPER- PARAMETERS
The optimization for the augmentation network is min-max game, which leads to hyperparameters to balance the contradicting loss. Specifically, -wi (ū(ū) + w2 V Leansist (Ú, ūc) + wg V LAAE. where LAVE refers to the KL. divergence regularizer on the latent encoding distribution.
In our main experiment, we use w1 0.0001, w2 0.1, W3 0.1 on all datasets except Melbourne Airbnb and SNLI-VE. On Melbourne Airbnb and SNLI-VE, we use = 0.001, W2 = 0.1, W3 = 0.1. Note that the hyperparameters are relative consistent across datasets.
Further, we investigate the influence of the different combinations of W1 W2, and Wg.. We summarize the result on Petfinder Table 7 We observe consistent improvements over the original multimodal network across various combinations.
W1
U2
W3
Accuracy
0.0001
0.1
0.1
0.3539
0.0001
0.01
0.01
0.3400
0.005
0.1
0.1
0.3482
0.005
0.01
0.01
0.3464
0.001
0.1
0.1
0.3371
0.001
0.01
0.01
0.3467
Multimodal Network
0.2911
Table 7:
LeMDA improves over Multimodal Network in accuracy with different sets of hyperpa- rameters on Petfinder. Specifically, ww (lGg) + w2 Leonniitt (Ú. (G) + wy LVAE
C MOTIVATIONAL EXAMPLES WHEN ONLY AUGMENTING ONE MODALITY
We have performed an additional set of experiments investigate the effect of augmenting a single modality on Hateful Memes using fhate-of-the-eee the augmentation techniques. In Hateful Memes, both text and image are required to decide if the content is hateful. Two modalities provide complemen- tary information to each other. We run baseline augmentation to only one modality or independently on two modalitics. We observe no consistent improvements. Essentially, performing augmentation naively to one modality or jointly without considering cross-modality relationships won't 't lead to effective augmentation.
Multimodal Network
Method
Image
Text
Image + Text
Trivial Augment
0.7040
0.6860
0.7057
MixUp
0.6855
0.6777
0.6939
0.6939
Manifold Mixup
0.6323
0.7444
0.6878
MixGen
0.7427
0.6872
0.7510
16


Reviews:
[{"review_id": "rk7TEsaByGp", "reviewer": "Reviewer_nNn5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes LeMDA (Learning Multimodal Data Augmentation) as a general multimodal data augmentation method. Specifically, the authors design the augmentation transformation as a learnable module applying on the latent representations of the inputs from different modalities. Since the module is end-to-end learned, it is naturally adaptive to different cross-modal relationships and can be applied to any modality combinations. The empirical experiments show that the proposed method consistently boosts accuracy.", "summary_of_the_review": "Overall, this is a timely paper with good empirical performance. However, I am not recommending acceptance because I am not sure I am understanding the main method correctly. Please do elaborate in the rebuttal on my questions above.  ", "strength_and_weaknesses": "The paper flows well. But I am confused at some major settings. First, how $\\mathcal{F}$ and $\\mathcal{G}$ is trained. For the optimization problem definition under section 3.2, do you mean this (as I inferred from the context),\n\nmax_$\\mathcal{G}$ $\\mathbb{E}$\\_{$x\\sim\\mathcal{X}$} (L($\\hat{y}$ \\_{$\\mathcal{G}$})+L\\_{consist}($\\hat{y}$, $\\hat{y}_{\\mathcal{G}}$)).\n\n(sorry there are some formatting issue from OpenReview) I am asking because I do not see why $L(\\hat{y})$ is dependent on $\\mathcal{G}$, and why this is a maxmin problem. If so, the notations in the algorithm and the figure are also wrong.\n\nAlso, for the VAE implementation, do you only adopt the architecture of VAE or both the architecture and VAE loss?\n\nIn practice, you do not have a coefficient to balance different losses? \n\nIn practice, how do you balance the training of the task network and augmentation network? For example, do you iteratively train either of the networks like GAN?\n\nThe experiment results look promising. The authors also provide comprehensive ablation study. \n\nAs an important motivation, it would be better if the claims on the relationships are verified via experiments. For example, what if in other methods/in the proposed method, only one of the complementary modalities is augmented?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jdgMn47jP4L", "reviewer": "Reviewer_cbnb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Multimodal Data Augmentation model, which can augment multimodal data in feature space.", "summary_of_the_review": "The idea is interesting.  However, the experiment is limited. The authors should do more experiments.", "strength_and_weaknesses": "The idea is interesting. However, the experiment is limited. The authors should consider the multimodal dataset with audio and visual modalities. And other multimodal data augmentation methods should be compared.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y5S30LOvM-", "reviewer": "Reviewer_huNS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work identifies two critical challenges in multimodal data augmentation: First, Non-trivial augmentation for certain modalities such as numerical or categorical, and preservation of labels when the different modalities are augmented in isolation. This work proposes an easy-to-adapt learnable multimodal augmentation technique( LeMDA) that applies augmentation in the latent space. These augmentations are learned through adversarial training and semantic correspondence is ensured using consistency regularization. \nExperiments on three modalities of images,  text, and tabular data show the approach's effectiveness and the improvement in accuracy with the proposed augmentation.", "summary_of_the_review": "Overall, the paper introduces an approach for generalized multimodal data augmentation for any modality. The experiments are performed on various tasks and datasets where the method outperforms the prior state of the art. There are some concerns with the overlapping related work in vision and language (which can be addressed).\n\n\n\n", "strength_and_weaknesses": "Strengths:\n+ The method is applicable to any modality. The general framework proposed in this work can be applied across different input modalities. This is certainly a strong point of the work and most of the prior work on multimodal data augmentation caters to specific modalities. \n\n+ Adequate experiments are performed to show the effectiveness of the approach. Experiments are performed for three modalities which is a positive. The performance improves across tasks and modalities. The usefulness of the consistency regularizer is also supported by ablations.\n\nWeaknesses:\n- The model formulation has limited novelty $f_{before}$ and is also present in the other related work [1,2,3,4,5] as pre-trained feature representations much like in this work. $\\mathcal{G}$ is usually a VAE [1,2,3,5] encoding the latents and then the outputs of the decoder are used to minimize a certain loss (a reconstruction for consistency or a classifier in the latent space[1]). What are the crucial advantages of the current approach over these existing methods and how can be term the current approach as an “augmentation” approach since we do not modify the input data? In essence, can we use the above-mentioned approaches to sample many “z” to get the same effect as with the current approach?\n- The architectural details are not provided. What kind of activations and VAEs are considered in work? What is the motivation/experimental reasoning behind using specific architectures? What is the design of the cross-attention module? This needs to be discussed in detail. \nSupport the following claims with an example:\n In related work: With early fusion, we capture the interaction between low-level features useful for multimodal tasks with strong cross-modal correlations.\nThe work in multimodal learning with late fusion is sparsely discussed. For example, the following work is relevant to the paper and should be adequately cited:\n   1. Cross-Linked Variational Autoencoders for Generalized Zero-Shot Learning. Schönfeld et. al. CVPR 2019.\n2. Latent normalizing flows for many-to-many cross-domain mappings. Mahajan et. al, ICLR 2020. \n3. Learning two-branch neural networks for the image-text matching tasks. Wang et al, TPAMI 2019.\n4. Learning robust visual semantic embeddings. Tsai et al, CVPR 2017. \n5. Context object-split latent spaces for diverse image captioning.  Mahajan et al, NeurIPS 2020. \n5 also introduces pseudo-captions for augmentation. How does it compare to the work proposed in the paper?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BEWr3Ihi3ft", "reviewer": "Reviewer_PrKb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposed a new data augmentation strategy applied on the feature space for multi-modal classification task. Specifically, the authors applied learnable augmentation network, in the form of VAE, to perturb the encoded embeddings from different modalities. Experiments are conducted on 8 multimodal datasets and compared with the baseline, the proposed augmentation strategy significantly improves the classification results (up to 6%).", "summary_of_the_review": "Overall I think this manuscript is a good paper. I give marginal accept for now and would like to hear authors' feedbacks on my comments in the weakness session.", "strength_and_weaknesses": "# Strength\nThe paper is well-written and easy to follow. The experimental results are strong and the improvements from the proposed method are significant. The ablation studies are well-designed and clearly illustrated the effectiveness of the design choices.\n\n# Weakness\n* I wonder if there are typos on the gradients for updating augmentation network G. Should the gradient be written as -\\partial L(y^hat_G) + \\partial L_{consistent} instead? Updating network G with the gradient -\\partial L(y^hat) doesn't make sense to me as i believe \\partial L(y^hat) \\partial G is zero? (this notation consistently appears in Fig. 2, Alg. 1 and main text).\n* I feel one table illustrating the training cost (FLOPS/Params/CPU wall time) when using different augmentation is necessary to better understand the proposed augmentation strategy.\n* Is the update step on augmentation network G and task network F happening within one iteration (using the same batch of training data) or the updating steps between G and F are alternating as in GAN training? \n* When updating these two networks, is there any hyperparameter that need to be adjusted (e.g. the weight between two losses). If there is any augmentation specific hyperparameter, is it consistently used across different experiments/datasets?\n* The overall improvements brought by the proposed augmentation across different datasets have large variance (from 0.4% - 6%). It would be better if the authors provides more discussions and insights on why the proposed method works better on certain dataset than others.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster