Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This manuscript proposed a new data augmentation strategy applied on the feature space for multi-modal classification task. Specifically, the authors applied learnable augmentation network, in the form of VAE, to perturb the encoded embeddings from different modalities. Experiments are conducted on 8 multimodal datasets and compared with the baseline, the proposed augmentation strategy significantly improves the classification results (up to 6%).
strength_and_weaknesses: # Strength
The paper is well-written and easy to follow. The experimental results are strong and the improvements from the proposed method are significant. The ablation studies are well-designed and clearly illustrated the effectiveness of the design choices.

# Weakness
* I wonder if there are typos on the gradients for updating augmentation network G. Should the gradient be written as -\partial L(y^hat_G) + \partial L_{consistent} instead? Updating network G with the gradient -\partial L(y^hat) doesn't make sense to me as i believe \partial L(y^hat) \partial G is zero? (this notation consistently appears in Fig. 2, Alg. 1 and main text).
* I feel one table illustrating the training cost (FLOPS/Params/CPU wall time) when using different augmentation is necessary to better understand the proposed augmentation strategy.
* Is the update step on augmentation network G and task network F happening within one iteration (using the same batch of training data) or the updating steps between G and F are alternating as in GAN training? 
* When updating these two networks, is there any hyperparameter that need to be adjusted (e.g. the weight between two losses). If there is any augmentation specific hyperparameter, is it consistently used across different experiments/datasets?
* The overall improvements brought by the proposed augmentation across different datasets have large variance (from 0.4% - 6%). It would be better if the authors provides more discussions and insights on why the proposed method works better on certain dataset than others.
clarity,_quality,_novelty_and_reproducibility: Clarity: There are some questions that I posted on the Weakness session. But overall the idea and logic is clear to me.
Quality: The written is in good quality and the figures/tables/experiments are well-designed.
Novelty: I think the proposed meothod is a novel idea.
Reproducibility: The manuscript provided some implementation details and promised to open-source code upon publication.
summary_of_the_review: Overall I think this manuscript is a good paper. I give marginal accept for now and would like to hear authors' feedbacks on my comments in the weakness session.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work identifies two critical challenges in multimodal data augmentation: First, Non-trivial augmentation for certain modalities such as numerical or categorical, and preservation of labels when the different modalities are augmented in isolation. This work proposes an easy-to-adapt learnable multimodal augmentation technique( LeMDA) that applies augmentation in the latent space. These augmentations are learned through adversarial training and semantic correspondence is ensured using consistency regularization. 
Experiments on three modalities of images,  text, and tabular data show the approach's effectiveness and the improvement in accuracy with the proposed augmentation.
strength_and_weaknesses: Strengths:
+ The method is applicable to any modality. The general framework proposed in this work can be applied across different input modalities. This is certainly a strong point of the work and most of the prior work on multimodal data augmentation caters to specific modalities. 

+ Adequate experiments are performed to show the effectiveness of the approach. Experiments are performed for three modalities which is a positive. The performance improves across tasks and modalities. The usefulness of the consistency regularizer is also supported by ablations.

Weaknesses:
- The model formulation has limited novelty $f_{before}$ and is also present in the other related work [1,2,3,4,5] as pre-trained feature representations much like in this work. $\mathcal{G}$ is usually a VAE [1,2,3,5] encoding the latents and then the outputs of the decoder are used to minimize a certain loss (a reconstruction for consistency or a classifier in the latent space[1]). What are the crucial advantages of the current approach over these existing methods and how can be term the current approach as an “augmentation” approach since we do not modify the input data? In essence, can we use the above-mentioned approaches to sample many “z” to get the same effect as with the current approach?
- The architectural details are not provided. What kind of activations and VAEs are considered in work? What is the motivation/experimental reasoning behind using specific architectures? What is the design of the cross-attention module? This needs to be discussed in detail. 
Support the following claims with an example:
 In related work: With early fusion, we capture the interaction between low-level features useful for multimodal tasks with strong cross-modal correlations.
The work in multimodal learning with late fusion is sparsely discussed. For example, the following work is relevant to the paper and should be adequately cited:
   1. Cross-Linked Variational Autoencoders for Generalized Zero-Shot Learning. Schönfeld et. al. CVPR 2019.
2. Latent normalizing flows for many-to-many cross-domain mappings. Mahajan et. al, ICLR 2020. 
3. Learning two-branch neural networks for the image-text matching tasks. Wang et al, TPAMI 2019.
4. Learning robust visual semantic embeddings. Tsai et al, CVPR 2017. 
5. Context object-split latent spaces for diverse image captioning.  Mahajan et al, NeurIPS 2020. 
5 also introduces pseudo-captions for augmentation. How does it compare to the work proposed in the paper?

clarity,_quality,_novelty_and_reproducibility: Clarity:
The design choices such as using a VAE in the augmentation is not adequately discussed. While VAEs are effective, generative models such as diffusion models, GANs and normalizing flows could be suitable for the tasks. The reason for choosing VAEs should be elaborated.

Quality:
The paper is well-written and easy to follow and most of the claims are supported by experiments. 

Novelty: 
The aspect of experimenting with more than two modalities with the same network is novel, however, the framework is similar to prior work pointed to above. 

Reproducaibility: 
The work is not reproducible with the details provided in the paper. The manuscript mentions that the code will be released upon publication. It is important to provide necessary architectural details in the supplemental if not the main paper.
summary_of_the_review: Overall, the paper introduces an approach for generalized multimodal data augmentation for any modality. The experiments are performed on various tasks and datasets where the method outperforms the prior state of the art. There are some concerns with the overlapping related work in vision and language (which can be addressed).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a Multimodal Data Augmentation model, which can augment multimodal data in feature space.
strength_and_weaknesses: The idea is interesting. However, the experiment is limited. The authors should consider the multimodal dataset with audio and visual modalities. And other multimodal data augmentation methods should be compared.
clarity,_quality,_novelty_and_reproducibility: The idea is interesting. The writing seems to be good. However, the experiment is limited. The authors should consider the multimodal dataset with audio and visual modalities. And other multimodal data augmentation methods should be compared.
summary_of_the_review: The idea is interesting.  However, the experiment is limited. The authors should do more experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes LeMDA (Learning Multimodal Data Augmentation) as a general multimodal data augmentation method. Specifically, the authors design the augmentation transformation as a learnable module applying on the latent representations of the inputs from different modalities. Since the module is end-to-end learned, it is naturally adaptive to different cross-modal relationships and can be applied to any modality combinations. The empirical experiments show that the proposed method consistently boosts accuracy.
strength_and_weaknesses: The paper flows well. But I am confused at some major settings. First, how $\mathcal{F}$ and $\mathcal{G}$ is trained. For the optimization problem definition under section 3.2, do you mean this (as I inferred from the context),

max_$\mathcal{G}$ $\mathbb{E}$\_{$x\sim\mathcal{X}$} (L($\hat{y}$ \_{$\mathcal{G}$})+L\_{consist}($\hat{y}$, $\hat{y}_{\mathcal{G}}$)).

(sorry there are some formatting issue from OpenReview) I am asking because I do not see why $L(\hat{y})$ is dependent on $\mathcal{G}$, and why this is a maxmin problem. If so, the notations in the algorithm and the figure are also wrong.

Also, for the VAE implementation, do you only adopt the architecture of VAE or both the architecture and VAE loss?

In practice, you do not have a coefficient to balance different losses? 

In practice, how do you balance the training of the task network and augmentation network? For example, do you iteratively train either of the networks like GAN?

The experiment results look promising. The authors also provide comprehensive ablation study. 

As an important motivation, it would be better if the claims on the relationships are verified via experiments. For example, what if in other methods/in the proposed method, only one of the complementary modalities is augmented?

clarity,_quality,_novelty_and_reproducibility: Quality: This is a timely work, proposing a data augmentation method under multi-modality setting. The method works well empirically. 

Clarity: The paper generally flows well, except for some major confusing/missing points.

Novelty: Although the idea of training a separate NN to do data augmentation is not new, the multimodality setting is comparably under-explored yet. 

Reproducibility: The authors promise they will release the code.

summary_of_the_review: Overall, this is a timely paper with good empirical performance. However, I am not recommending acceptance because I am not sure I am understanding the main method correctly. Please do elaborate in the rebuttal on my questions above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by four knowledgeable referees. Overall, the reviewers found the paper timely, the contribution interesting, and the evaluation rather extensive. Reviewers raised concerns about the missing discussion w.r.t. training costs (PrKb), the variance in the results across datasets (PrKb), the justification w.r.t. some design and hyper-parameter choices (PrKb, huNS, nNn5), the advantages over some existing methods (huNS), and the novelty (nNn5). The authors engaged with the reviewers during the discussion phase and adequately addressed most of their concerns. The authors ask the AC to disregard the review by cbnb. The AC goes over the review and agrees with the authors that the review is unsubstantial and does not provide detailed actionable feedback. The AC agrees with the other reviewers' assessment that the contribution is timely and well executed, and could stimulate future research. Therefore, the AC recommends to accept.
justification_for_why_not_higher_score: The contribution is sound and the experimental evidence extensive. Some choices could be better motivated - e.g. the use of a VAE instead of other generative models in this context. The authors leave this question as future work.
justification_for_why_not_lower_score: Relevant topic, sound contribution, convincing experimental evidence to support the claims.
Decision: Accept: poster