Title: Back-Modality: Leveraging Modal Transformation for Data Augmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: data augmentation

cross-modal
Abstract: We introduce Back-Modality, a novel data augmentation schema predicated on modal transformation. Data from an initial modality undergoes transformation to an intermediate modality, followed by a reverse transformation. This framework serves dual roles. On one hand, it operates as a general data augmentation strategy. On the other hand, it allows for other augmentation techniques, suitable for the intermediate modality, to enhance the initial modality. For instance, data augmentation methods applicable to pure text can be employed to augment images, thereby facilitating the cross-modality of data augmentation techniques. To validate the viability and efficacy of our framework, we proffer three instantiations of Back-Modality: back-captioning, back-imagination, and back-speech. Comprehensive evaluations across tasks such as image classification, sentiment classification, and textual entailment demonstrate that our methods consistently enhance performance under data-scarce circumstances.

Back-Modality: Leveraging Modal Transformation for Data Augmentation

Abstract

We introduce Back-Modality, a novel data augmentation schema predicated on modal transformation. Data from an initial modality undergo a transformation to an intermediate modality, followed by a reverse transformation. This framework serves dual roles. On one hand, it operates as a general data augmentation strategy. On the other hand, it allows for other augmentation techniques, suitable for the intermediate modality, to enhance the initial modality. For instance, data augmentation methods applicable to pure text can be employed to augment images, thereby facilitating the cross-modality of data augmentation techniques. To validate the viability and efficacy of our framework, we proffer three instantiations of BackModality: back-captioning, back-imagination, and back-speech. Comprehensive evaluations across tasks such as image classification, sentiment classification, and textual entailment demonstrate that our methods consistently enhance performance under data-scarce circumstances.

1 Introduction

Neural network-based deep learning models are often prone to overfitting, resulting in a loss of generalization capability due to the limited size of training data. To mitigate this issue, data augmentation techniques are routinely employed to generate an augmented pool of training samples. The recent past has seen significant strides in the application of data augmentation within diverse fields such as speech [Ko et al., 2015, Park et al., 2019], computer vision [Simard et al., 1996, Shorten and Khoshgoftaar, 2019, Szegedy et al., 2014], and natural language processing [Sennrich et al., 2016, Wei and Zou, 2019]. Nevertheless, most of these data augmentation algorithms are tailored for a specific modality.

Concomitant with the swift advancement of cross-modal methodologies, various dual cross-modal models, such as text-to-image generation and image captioning, have demonstrated impressive performance. This progress catalyzes our development of a novel paradigm for data augmentation: Back-Modality, a comprehensive data augmentation framework predicated on modal transformation. Data in the initial modality are transformed to an intermediate modality and subsequently reversed back. Theoretically, the initial and intermediate modalities can represent any modality, including but not limited to text, image, audio, and video. Our framework retains its universality, contingent on the availability of corresponding dual cross-modal models to execute the modal transformation. Leveraging pretrained cross-modal models, our method is capable of generating a wide array of high-quality and diverse data. A comparison illustrating the breadth and quality of the generated data is provided in Figure 1.

Simultaneously, our framework enables the augmentation method intended for the intermediate modality to benefit the initial modality. For instance, data augmentation methods designed for pure

Ground truth    RandomErasing        Mixups           AutoAugment

Ours

school bus

goldfish

text can be leveraged to augment images, fostering cross-modality of data augmentation techniques. In essence, our framework facilitates partial decoupling of data augmentation methods from specific modalities. Moreover, our framework broadens the application landscape of cross-modal models. Given that our methodology obviates the need to access model weights or undergo further fine-tuning, it can be seen as an application variant of Cross-Modal-Models-as-a-Service (CMMaaS).

To substantiate the feasibility and efficacy of our framework, we present three instantiations of Back-Modality: back-captioning, back-imagination, and back-speech. For back-captioning, the initial modality is an image, and we utilize an image-caption model to generate captions for each image. Subsequently, data augmentation methods designed for text are employed to augment these captions. Finally, a text-to-image model is invoked to generate variant images, which are then used as augmented samples. For back-imagination and back-speech, the initial modality is text, and the intermediate modality is image and speech, respectively.

Systematic evaluations across various tasks including image classification, sentiment classification, and textual entailment demonstrate that our methods can consistently improve performance, particularly under data-scarce conditions. Upon further analysis and comprehensive case studies, it has been observed that the data generated via our method typically exhibit increased diversity.

Our principal contributions and discoveries can be summarized as follows:

We introduce Back-Modality, a modality-agnostic data augmentation framework predicated on modal transformation. Within this framework, data in the initial modality undergo transformation to an intermediate modality and subsequently reverse back.

Our framework enables the cross-modality of data augmentation methods. For instance, data augmentation techniques developed for pure text can be deployed to augment images.

Our approach extends the application realms of cross-modal models. Our methodology eliminates the need to access model weights or conduct further fine-tuning, hence it can be perceived as a variant of the Cross-Modal-Models-as-a-Service (CMMaaS) application.

Experiments on a variety of tasks and datasets substantiate that our methods can consistently enhance performance, particularly in data-scarce scenarios. Further analysis reveals that data generated by our method typically exhibits a higher degree of diversity.

2 Method

2.1 Framework

Back-Modality presents a comprehensive, modality-agnostic data augmentation framework predicated on modal transformation. As shown in Figure 2, A symbolizes the initial modality, and B signifies the

Data in Modality A

Data in Modality B

Data in Modality A

Cross-Modal

F:

A

-->

B

Cross-Modal

G:

B

-->

A

H: Augmentation Method

intermediate modality. The framework incorporates two types of dual cross-modal tasks: F : A ↦→ B and G : B ↦→ A . Here, H denotes the augmentation methods applied to modality B .

Let X denote a sampled data point in the initial modality A , and X aug represents the augmented samples of X . Our framework can be succinctly expressed as:

Theoretically, A and B can represent any modality, such as text, image, audio, video, and so forth. Moreover, A and B can also be the same modality. For instance, back-translation augmentation methods can be perceived as a special case of our framework, where A and B are both text in different languages, and H is omitted. Essentially, our framework retains its universality, contingent upon the availability of corresponding dual models.

Within our framework, H can assist in augmenting data in modality A by augmenting data in the intermediate modality B . This implies that the augmentation method suitable for a specific modality can be transferred to other modalities via our framework. More formally, let us consider a scenario where F and G are fixed, with Y F,G denoting the corresponding framework. Here, H is treated as a variable. Consequently, the augmented samples X aug can be viewed as a function of X and H .

Given a specific X , varying H can yield a diverse set of augmented samples X aug . In essence, our framework facilitates partial decoupling of data augmentation methods from specific modalities.

2.2 Instantiations

In order to demonstrate the feasibility and efficacy of Back-Modality, we introduce three instantiations of this framework: back-captioning, back-imagination, and back-speech. These methods encompass three prevalent modalities: text, image, and speech.

Back-captioning In the proposed method, we define A as image modality and B as text modality. Here, F represents the task of image captioning, while G represents the task of generating images from text. H corresponds to the process of textual data augmentation.

Back-captioning, as a concept, can be deconstructed into a trifecta of augmentation methodologies: multi-captioning, caption augmentation, and multi-imagination. It is observed that various observers usually generate captions for the same image that, while being semantically analogous, exhibit subtle variances in detail. This phenomenon mirrors the capability of most image captioning models to generate multiple unique captions for a single image, a process we refer to as multi-captioning. Given a caption sentence, several text augmentation techniques can be employed to produce multiple related sentences, a process we term as caption augmentation. For a given sentence, the images that different individuals visualize generally share similarities, yet present unique details. We refer to this phenomenon as multi-imagination. A diffusion-based model, employing various random seeds, can effectively simulate this process. Given an image captioning model that generates l captions for each image, a caption augmentation process that generates m sentences for each caption, and a text-to-image model that generates n images for each sentence, the maximum total number of generated augmented sentences per original sentence amounts to l ∗ m ∗ n . This obviously represents

a highly efficient data augmentation approach. In practical implementation, we conduct uniform sampling at random on these augmented data to obtain the final augmentation dataset.

Back-speech In this proposed approach, we designate modality A as text, while B pertains to the audio modality. Here, F corresponds to the task of text-to-speech generation, and G denotes the automatic speech recognition task. The text-to-speech model is designed to generate one audio file for each input sentence. Following this, audio data augmentation techniques, such as pitch shifting, denoted by H , are applied to these audio files to produce augmented sound files. Subsequently, the automatic speech recognition model translates these augmented sound files back into text, thereby generating augmented sentences.

The notion of back-speech can be elucidated by the following observation: the sentence articulated by an intermediary often bears slight deviations from the original sentence. This is due to the fact that different individuals may have distinct pronunciations for the same sentence, influenced by factors such as speech rate, pitch, or regional accents. To simulate these variations, we employ augmentation techniques within the audio modality. As a consequence, in the resultant output, some words may be substituted by their homophones. Given that most homonyms bear morphological similarities, with only minor differences at the subword level, this transformation can be viewed as a form of subword regularization facilitated by the audio modality.

Back-imagination Back-imagination can be construed as the inverse operation of back-captioning. In this approach, we assign modality A to text, while B corresponds to the image modality. Here, F represents the task of generating images from text, and G signifies the image captioning task. Similarly, back-imagination amalgamates three types of augmentation techniques: multi-captioning, image augmentation, and multi-imagination.

Table 1: Statistics of configurations. "augmentation with GPT" means that we employ GPT model to augment captions. The symbol "-" indicates that we do not apply image augmentation in the process of back-imagination. Tiny ImageNet [Le and Yang, 2015] is an image classification dataset, SST-2 [Socher et al., 2013] is employed for sentiment classification, and TNCC, a textual entailment dataset, is based on Crisscrossed Captions [Parekh et al., 2020]. For further details, please refer to the appendix.

3 Experiments

In this section, we initially delineate the tasks, datasets, and models employed in our research. Subsequently, we assess the efficacy of back-captioning, back-speech and back-imagination in data-scarce scenarios, spanning a diverse range of tasks and datasets. Finally, we investigate the intrinsic mechanisms underlying the effectiveness of our proposed method from three perspectives: (1) conducting a quantitative analysis on the diversity and affinity of the generated samples, (2) performing ablation experiments to evaluate the contribution of individual components within the method, and (3) scrutinizing the differences between the original and augmented samples at an instance level and providing an in-depth analysis of the transformation effectuated by the augmentation process.

3.1 Baseline and Comparison

We use the performance of the base model, devoid of any data augmentation, as a baseline to evaluate the effectiveness of our augmentation method in enhancing model performance. For the image classification task, we employ the Resnet-18 model [He et al., 2015]. We use BERT [Devlin et al.,

2018] for sentiment classification and textual entailment tasks. To further substantiate the efficacy of our methods, we compare them against several other data augmentation strategies. For the image classification task, we consider four renowned methods: Random Erasing [Zhong et al., 2020], Autoaugment [Cubuk et al., 2019], Alignmixup [Venkataramanan et al., 2022], and Puzzle Mix [Kim et al., 2020]. For the sentiment classification and textual entailment tasks, we opt for the most popular task-agnostic data augmentation methods for comparison, as per Longpre et al. [2020], Yoo et al. [2021], including EDA [Wei and Zou, 2019] and Back-translation [Sennrich et al., 2016, Fadaee et al., 2017, Edunov et al., 2018, Ng et al., 2019]. We employ the large pre-trained EN-DE and DE-EN translation models [Ng et al., 2019, Yoo et al., 2021] for back-translation, which have a size order of magnitude comparable to the cross-model models used by Back-Modality methods, making them a robust baseline. In addition, we also consider some task-specific data augmentation methods for reference, including TMix [Chen et al., 2020], SSMix [Yoon et al., 2021], and Treemix [Zhang et al., 2022].

3.2 Datasets and Configurations

Datasets Tiny ImageNet [Le and Yang, 2015] and the Stanford Sentiment Treebank-2 (SST-2) [Socher et al., 2013] are widely used datasets. Given the scarcity of datasets for understanding natural language in visual scenes, we introduce a novel textual entailment dataset, named Textual Natural Contextual Classification (TNCC). And detailed descriptions can be found in the appendix.

Data-scarce scenarios To showcase our approach, we conduct experiments on artificially datascarce tasks by sub-sampling the training set. For image classification, sentiment classification, and textual entailment tasks, we perform a class-balanced sub-sample on the training set and we denote the number of data samples for each class in the downsampled dataset as "shot," before any data augmentation is applied. We adhere to statistical rigor in our experiments by executing the sub-samples to 5 different data seeds. Concurrently, for each sub-sample, we train all models using 5 different random seeds. We report the statistical mean for all results. All experiments were conducted using PyTorch and executed on RTX 6000 GPUs and the Atlas computing cluster.

Configurations Table 1 provides a comprehensive view of the configurations used in our experiments. Unless explicitly stated otherwise, all pretrained models utilized in our research are obtained from the Huggingface Transformers library 2 [Wolf et al., 2019]. The code used to generate augmented samples and the TNCC dataset are both accessible 3 . The default augmentation size used in our studies, which includes both our method and the comparison method, is set to 5. As an illustration, in our experiments with Back-captioning, when we have l = n = m = 2 , we create 8 augmentations for each image (2x2x2), and then, we randomly select 5 of these augmentations to form the final augmented dataset. For back-captioning, we utilize gpt-3.5-turbo 4 , augmenting captions using the following prompt: "Maintain the nouns in the following sentence intact and generate semantically diverse sentences." For the process of back-imagination, we consciously exclude image augmentation in our experiments. One reason is that images produced via certain augmentation techniques, such as random erasing [Zhong et al., 2020] and cutout [DeVries and Taylor, 2017], often present a substantial challenge to image captioning models. On the other hand, the combination of multi-imagination and multi-captioning appears to be sufficient to yield satisfactory results.

Detailed Strategies In our experimental process, we observed that simply constructing a pipeline does not necessarily ensure the quality of the augmented data. Consequently, we have devised several strategies aimed at minimizing the production of low-quality or even inaccurate augmented data.

In the case of back-captioning, while the model possesses the capacity to generate highly relevant image descriptions, when deployed for specific tasks such as image classification, more nuanced categories are often required. However, the model sometimes falls short in providing these nuanced labels within the generated descriptions. For instance, when given an image of a 'Persian cat', the model tends to generate descriptions mentioning 'cat' rather than the more specific 'Persian cat'. Several image captioning models, such as the OFA [Wang et al., 2022] model, have the capability to generate image captions conditional on both a text prompt and an image. To address this issue, we

explicitly inject the image labels into the text prompts, which leads to the generation of descriptions that incorporate these finer-grained labels.

During the back-imagination process, there are occasions when the text-to-image model generates black and white images. This can lead to subsequent image captioning models generating sentences like "A black and white photo of," which are clearly inappropriate for augmentation samples. Consequently, if the back-imagination process yields a black and white image, we directly discard it and proceed with a resampling operation. As for back-speech, if the sentences generated have an edit distance from the original sentence that exceeds 20% of the entire sentence length, we discard those sentences as well.

3.3 Results

Tables 2, 3, and Table 4, respectively, display the statistical average accuracy values on the Tiny ImageNet, TNCC, and SST-2 datasets. We carried out hypothesis testing on the results across all tasks, with all p-values falling below 0.05. This suggests that our findings are statistically significant. Our back-captioning, back-imagination, and back-speech methods consistently outperform both the base model and other data augmentation methods. This indicates that in data-scarce scenarios, data augmentation based on modal transformation can offer significant performance improvements. In particular, the results of back-imagination demonstrate that our architecture alone, even without the inclusion of additional data augmentation methods (H), is capable of generating effective augmented data.

3.4 Ablation

Table 5 shows the ablation studies of back-captioning, back-imagination and back-speech. The individual components of our framework make significant contributions to the overall performance of the model. In particular, augmentation with GPT, pitch shifting and time stretching all improve the model performance, which strongly supports the notion that our framework effectively accomplishes the cross-modality of data augmentation approaches.

3.5 Diversity and Affinity

Following the methodology outlined by Gontijo-Lopes et al. [2020], we further analyze our augmentation methods based on two key dimensions: Diversity and Affinity. Diversity is designed to quantify the notion that augmentations can prevent model overfitting by increasing the number of samples in the training set. As a metric of diversity, Gontijo-Lopes et al. [2020] employ the final training loss of a model trained with a given augmentation. The larger the loss function, the better the diversity of the augmented dataset. Affinity, on the other hand, quantifies how augmentation shifts data with respect to the decision boundary of the clean baseline model. Affinity is defined as the difference between the validation accuracy of a model trained on clean data and tested on clean data, and the accuracy of the same model when tested on an augmented validation set. Table 6 indicates that while ensuring comparable affinity metrics, our method consistently presents higher diversity metrics compared to other augmentation methods.

3.6 Case Study

Figure 1 showcases the images generated by our data augmentation method alongside other approaches. Upon further analysis, we have uncovered advantages in the back-captioning method that many traditional augmentation algorithms lack. As illustrated in Figure 3, we have listed several typical data augmentation patterns found in the back-captioning method:

Luminance Modulation : This involves altering the image's brightness by either decreasing or increasing it.

Luminance Modulation

goldfish

cash machine

candle

bathtub

school bus

lemon

ice cream

Structural Preservation

Attribute Adjustment

Entity Elimination

福

Structural Preservation : This technique focuses on preserving the primary subject's structure while making changes to the background elements.

Attribute Adjustment : Adjustments are made to the target's attributes, such as size, thickness, and quantity.

Entity Elimination : This technique involves the removal of non-target entities from the image.

We also perform an extensive case study on back-imagination and back-speech. As depicted in Figure 4, we notice that augmented sentences, generated by different modal transformations, display unique characteristics. Further analysis reveals three notable advantages of our methods:

Some augmented cases of back-imagination:

(1)

Add/ Remove detail

The black dog runs through the water .

A black dog running through the water in a lake .

A young man in a black and yellow jacket is gazing at something and smiling .

A young man in a black and yellow jacket is smiling .

(2) Replace word by synonym/ hypernym/ hyponym

A woman is throwing a frisbee on a beach .

A woman is playing a frisbee on a beach .

Aman is standing in front of a skyscraper .        A man standing in front of a building .

Two children sit on a small seesaw in the sand .        Two boys sit on a small seesaw in the sand .

(3) Reverse

the order

A boy strolls by a pond in a park .        A  boy walks in a park near a pond.

(4) Change /blur the quantity

Three people are walking on a path in a meadow .

Agroup of people are walking on a path in a meadow .

(5) Tense variation

Two children sit on a small seesaw in the sand .

Two children  is sitting on a small seesaw in the sand .

...

Some augmented cases of back-speech:

(1) Split /merge the words

That 's far too tragic to merit such superficial treatment .

That is far too tragic to merit such superficial treatment .

Remains utterly satisfied to remain the same throughout .

Remain sutterly satisfied to remain the same throughout .

(2). Remove word

He hoped there would be stew for dinner, turnips and carrots .       He hoped there would be stew for dinner, turnips .

(3)

Replace word by homonym

This is a shot of cannery row in 1932 .        This is a shot of cannery row in nineteen thirty-two .

Lend some dignity to a dumb story .        Lend some dignity to a dome story .

The year 's best and most unpredictable comedy .

The nears best and most unpredictable comedy .

(4)

Change punctuation

So,  we have this ability as well .

So we have this ability as well .

...

Leveraging cross-modal models, our back-imagination augmented sentences demonstrate exceptional fluency, readability, and minimal spelling or grammar errors. Correspondingly, the generated images consistently maintain a high level of quality.

Our approach introduces greater diversity in both augmented sentences and images. Notably, augmentation types such as adding details are challenging to achieve using popular text augmentation methods like EDA and back-translation.

Our method ensures better semantic consistency. For instance, while both EDA and our method involve deletion operations, our approach selectively removes details. But EDA performs random deletion, which increases the likelihood of altering the original semantics. A thorough analysis of back-speech augmentation reveals that certain words in augmented sentences are substituted with similar-sounding words, often sharing similar but not identical subwords. Additionally, the augmentation process involves splitting and merging certain words. Consequently, the augmentation of back-speech can be considered as a form of regularization akin to techniques such as BPE-Dropout [Provilkov et al., 2020] and SwitchOut [Wang et al., 2018].

4 Related Work

4.1 Data Augmentation

Data augmentation is widely utilized in various domains to address limited data scenarios, and it has achieved significant success. In computer vision, basic image manipulations such as translation and rotation can generate new samples [Ronneberger et al., 2015, Krizhevsky et al., 2017]. Techniques like random erasing [Zhong et al., 2020] and cutting [Devries and Taylor, 2017] aid in improving generalizability by occluding images. Pitch shifting and time stretching are popular methods in speech processing [Salamon and Bello, 2016, Moreno-Barea et al., 2018]. Text augmentation methods typically involve word-level text editing operations and sentence-level text generation.For word-level text editing operations, word substitution is commonly employed, including techniques such as

synonym replacement [Zhang et al., 2015], KNN replacement [Wang and Yang, 2015], Uniform and TF-IDF replacement [Xie et al., 2020], Bi-RNN Contextual replacement [Kobayashi, 2018], CBERT [Wu et al., 2019], and LAMBADA [Anaby-Tavor et al., 2019]. EDA [Wei and Zou, 2019] comprehensively explores text editing operations, including synonym replacement, insertion, swap, and deletion, for data augmentation. The popular text augmentation method based on generation is back-translation [Sennrich et al., 2016, Fadaee et al., 2017, Edunov et al., 2018]. It is evident that previous data augmentation methods have been primarily tailored for specific modalities.

4.2 Dual Cross-modal Tasks and Models

In recent years, large-scale pretrained foundation models have achieved remarkable success across a spectrum of tasks. Contemporary research [Zeng et al., 2022, Wu et al., 2023a, Shen et al., 2023] suggests the amalgamation of multiple large pretrained models for the execution of novel downstream multimodal tasks. However, our approach diverges from the aforementioned studies, as we employ dual cross-modal models to devise a fresh data augmentation paradigm. There exists a plenitude of dual cross-modal tasks and pretrained models, such as text-to-image [Ho et al., 2020, Rombach et al., 2022], image captioning [Wang et al., 2022, Li et al., 2022], text-to-video [Singer et al., 2022, Hong et al., 2022], video captioning [Yan et al., 2022, Yamazaki et al., 2022], text-to-speech [Ren et al., 2020], and automatic speech recognition [Baevski et al., 2020]. Dual Learning [He et al., 2016, Xia et al., 2017, 2018] protocols train the models of two dual tasks in tandem, while consciously leveraging the probabilistic correlation between them to regularize the training process. In contrast, our methodology primarily centers on the usage of the large pretrained models of two dual tasks for data augmentation, circumventing additional training.

5 Limitations

Compared to existing data augmentation techniques, Back-Modality typically requires additional computational resources and inference time due to its reliance on large pretrained cross-modal models. However, as advancements continue in the cross-modal field and efficient machine learning methodologies are further employed, we anticipate the development of more efficient techniques. As such, this limitation is expected to be progressively alleviated. Furthermore, while our framework is inherently versatile, specific instances may require the design of corresponding strategies akin to those discussed in Section 3.2, which ensure label invariance in the generated samples and maximize their affinity with the original data.

6 Conclusion

In this work, we introduce a novel data augmentation framework, predicated on modal transformation, and confirm its feasibility and effectiveness. We posit that as the field of cross-modal learning continues to evolve, modal transformation can present a fresh avenue for developing data augmentation techniques. This emerging field presents a plethora of intriguing research questions that merit further exploration. For instance, a comprehensive evaluation of our method across a broader spectrum of tasks and modalities would be insightful. In addition, investigating the impact of fine-tuning the cross-modal models on domain-specific datasets on the quality of the generated data could yield meaningful insights.

Acknowledgments

This work was supported by the NSFC project (No. 62072399), Zhejiang Provincial Natural Science Foundation of China under Grant No. LZ23F020009, China Research Centre on Data and Knowledge for Engineering Sciences and Technology, MoE Engineering Research Center of Digital Library, and the Fundamental Research Funds for the Central Universities. Thanks to the Hangzhou AI Computing Center for providing the Atlas computing cluster. We also express our sincere gratitude to anonymous reviewers for their invaluable feedback and constructive comments, which significantly contributed to the improvement of this paper.

References

Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. Audio augmentation for speech recognition. In Interspeech , 2015. URL https://api.semanticscholar.org/CorpusID: 7360763 .

Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Gernot Kubin and Zdravko Kacic, editors, Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019 , pages 2613-2617. ISCA, 2019. doi: 10.21437/Interspeech.2019-2680. URL https://doi.org/10. 21437/Interspeech.2019-2680 .

Patrice Y. Simard, Yann LeCun, John S. Denker, and Bernard Victorri. Transformation invariance in pattern recognition-tangent distance and tangent propagation. In Neural Networks , 1996. URL https://api.semanticscholar.org/CorpusID:16067356 .

Connor Shorten and Taghi M. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data , 6:1-48, 2019. URL https://api.semanticscholar.org/CorpusID: 195811894 .

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1-9, 2014. URL https://api.semanticscholar.org/CorpusID:206592484 .

Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 86-96, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1009. URL https://aclanthology.org/P16-1009 .

Jason W. Wei and Kai Zou. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , pages 6381-6387. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1670. URL https://doi.org/10.18653/v1/D19-1670 .

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing , 2013. URL https: //api.semanticscholar.org/CorpusID:990233 .

Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for ms-coco. arXiv preprint arXiv:2004.15020 , 2020.

Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2015. URL https://api.semanticscholar.org/CorpusID:206594692 .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 13001-13008, 2020.

Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 113-123, 2019. URL https://api.semanticscholar.org/ CorpusID:196208260 .

Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. Alignmixup: Improving representations by interpolating aligned features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19174-19183, 2022.

Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning , pages 5275-5285. PMLR, 2020.

Shayne Longpre, Yu Wang, and Chris DuBois. How effective is task-agnostic data augmentation for pretrained transformers? In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 4401-4411, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.394. URL https://aclanthology.org/2020. findings-emnlp.394 .

Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woo-Myoung Park. Gpt3mix: Leveraging large-scale language models for text augmentation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021 , pages 2225-2239. Association for Computational Linguistics, 2021. doi: 10.18653/v1/ 2021.findings-emnlp.192. URL https://doi.org/10.18653/v1/2021.findings-emnlp.192 .

Marzieh Fadaee, Arianna Bisazza, and Christof Monz. Data augmentation for low-resource neural machine translation. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 2: Short Papers , pages 567-573. Association for Computational Linguistics, 2017. doi: 10.18653/v1/P17-2090. URL https://doi.org/10.18653/v1/P17-2090 .

Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 489-500, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1045. URL https://aclanthology.org/D18-1045 .

Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) , pages 314-319, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-5333. URL https://aclanthology.org/W19-5333 .

Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification. arXiv preprint arXiv:2004.12239 , 2020.

Soyoung Yoon, Gyuwan Kim, and Kyumin Park. Ssmix: Saliency-based span mixup for text classification. arXiv preprint arXiv:2106.08062 , 2021.

Le Zhang, Zichao Yang, and Diyi Yang. TreeMix: Compositional constituency-based data augmentation for natural language understanding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5243-5258, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.385. URL https://aclanthology.org/2022.naacl-main.385 .

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 , 2019.

Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.03052 , 2022.

Raphael Gontijo-Lopes, Sylvia J Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.08973 , 2020.

Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. Bpe-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 1882-1892, 2020.

Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. Switchout: an efficient data augmentation algorithm for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 856-861, 2018.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computerassisted Intervention , pages 234-241. Springer, 2015.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. CoRR , abs/1708.04552, 2017. URL http://arxiv.org/abs/1708.04552 .

Justin Salamon and Juan Pablo Bello. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Processing Letters , 24:279-283, 2016. URL https://api.semanticscholar.org/CorpusID:3537408 .

Francisco J. Moreno-Barea, Fiammetta Strazzera, José M. Jerez, Daniel Urda, and Leonardo Franco. Forward noise adjustment scheme for data augmentation. 2018 IEEE Symposium Series on Computational Intelligence (SSCI) , pages 728-734, 2018. URL https://api.semanticscholar. org/CorpusID:54048603 .

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 649-657, 2015. URL https://proceedings.neurips.cc/paper/2015/ hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html .

William Yang Wang and Diyi Yang. That's so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 2557-2563, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1306. URL https://aclanthology.org/D15-1306 .

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems , 33:6256-6268, 2020.

Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 452-457, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-2072. URL https://aclanthology.org/N18-2072 .

Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. Conditional BERT contextual augmentation. In João M. F. Rodrigues, Pedro J. S. Cardoso, Jânio M. Monteiro, Roberto Lam, Valeria V. Krzhizhanovskaya, Michael Harold Lees, Jack J. Dongarra, and Peter M. A. Sloot, editors, Computational Science - ICCS 2019 - 19th International Conference, Faro, Portugal, June 12-14, 2019, Proceedings, Part IV , volume 11539 of Lecture Notes in Computer Science , pages 84-95. Springer, 2019. doi: 10.1007/978-3-030-22747-0\_7. URL https://doi.org/10.1007/ 978-3-030-22747-0_7 .

Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, N. Tepper, and Naama Zwerdling. Do not have enough data? deep learning to the rescue! In AAAI Conference on Artificial Intelligence , 2019. URL https://api.semanticscholar.org/ CorpusID:212821571 .

Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598 , 2022.

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 , 2023a.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 , 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by crossmodal skip-connections. arXiv preprint arXiv:2205.12005 , 2022.

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022.

Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 , 2022.

Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979 , 2022.

Kashu Yamazaki, Khoa Vo, Sang Truong, Bhiksha Raj, and Ngan Le. Vltint: Visuallinguistic transformer-in-transformer for coherent video paragraph captioning. arXiv preprint arXiv:2211.15103 , 2022.

Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558 , 2020.

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems , 33:12449-12460, 2020.

Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 820-828, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ 5b69b9cb83065d403869739ae7f0995e-Abstract.html .

Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie-Yan Liu. Dual supervised learning. In International conference on machine learning , pages 3789-3798. PMLR, 2017.

Yingce Xia, Xu Tan, Fei Tian, Tao Qin, Nenghai Yu, and Tie-Yan Liu. Model-level dual learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 5379-5388. PMLR, 2018. URL http://proceedings.mlr.press/v80/xia18a.html .

Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models. arXiv preprint arXiv:2205.15463 , 2022.

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7623-7633, 2023b.

Yunxiang Li, Hua-Chieh Shao, Xiao Liang, Liyuan Chen, Ruiqi Li, Steve Jiang, Jing Wang, and You Zhang. Zero-shot medical image translation via frequency-guided diffusion models. arXiv preprint arXiv:2304.02742 , 2023.

Gwanghyun Kim, Ji Ha Jang, and Se Young Chun. Podia-3d: Domain adaptation of 3d generative model across large domain gap using pose-preserved text-to-image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22603-22612, 2023.

Yasser Benigmim, Subhankar Roy, Slim Essid, Vicky Kalogeiton, and Stéphane Lathuilière. Oneshot unsupervised domain adaptation with personalized diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 698-708, 2023.

Appendix

7 Back-imagination and Back-speech

A black dog leaps over a log .

A black dog  leaps  over a log in the woods .

...

A black dog jumps over a log .

text to image

image caption

Back-speech

Back-imagination

...

A black dog leaps over a log .

text to speech

speech

augmentation

...

automatic speech

recognition

A black dog leap over a log .

...

A black doll leaps over a log .

As shown in Figure 5, we present illustrative examples to facilitate a better understanding of two proposed techniques: Back-imagination and Back-speech.

8 Datasets

Tiny ImageNet [Le and Yang, 2015] serves as a compact version of the comprehensive ImageNet dataset. It comprises 100,000 images spanning 200 classes, with 500 images per class, and these images are downsized to 64×64 pixels. Each class is furnished with 500 training images, 50 validation images, and 50 test images.

The Stanford Sentiment Treebank-2 (SST-2) [Socher et al., 2013] is a sentiment classification dataset populated with movie reviews gathered from Rotten Tomatoes, paired with their corresponding binary labels. The dataset is partitioned into training, validation, and testing sets, comprising 67,349, 872, and 1,821 instances, respectively.

Given the scarcity of datasets for understanding natural language in visual scenes, we introduce a novel textual entailment dataset, named Textual Natural Contextual Classification (TNCC). This dataset is formulated on the foundation of Crisscrossed Captions [Parekh et al., 2020], an image captioning dataset supplied with human-rated semantic similarity ratings on a continuous scale from 0 to 5. We tailor the dataset to suit a binary classification task. Specifically, sentence pairs with annotation scores exceeding 4 are categorized as positive (entailment), whereas pairs with scores less than 1 are marked as negative (non-entailment). The TNCC dataset is partitioned into training, validation, and testing sets, containing 3,600, 1,200, and 1,560 instances, respectively. This dataset will be made available alongside our source codes.

9 Configurations

In this work, we employ a uniform experimental configuration for both textual entailment and sentiment classification tasks. We adopt BERT-BASE [Devlin et al., 2018], a model pretrained using Masked Language Modeling (MLM), as our primary experiment subject. For each individual downstream classification experiment, the classification model is initialized with the pretrained parameters from the BERT-BASE model. The classifier component, comprising of two fully connected layers that deduce class labels from the output embeddings generated by the transformer architectures, is randomly initialized. During the training phase, we leverage the Adam optimization algorithm with a learning rate set at 5 e -5 , the first and second momentum terms, β 1 and β 2 , are respectively set to 0.9 and 0.999. Additionally, we introduce an L 2 weight decay of 0.01 to the model. We select a batch

size of 2 for all trials. We save model checkpoints during training and ultimately employ the best checkpoint-determined based on performance on the validation dataset-for testing. The results are presented as classification accuracies on both datasets under investigation.

For the image classification task, we employ the ResNet18 [He et al., 2015] model, which is considered more suitable for small datasets. We initialize all learnable layer parameters randomly. During the training process, we employ the SGD optimizer with a learning rate of 0 . 1 , momentum of 0 . 9 , and a weight decay of 0.0001.

10 Human Evaluation on Augmented Samples

In response to your suggestion, we conducted a human evaluation on the sampled augmented data. The results of the evaluation are as follows:

For the images generated using the back-captioning method:

Label Invariance Score: 99.2%

For the sentences generated using the back-imagination method:

Semantic Consistency Score: 98.8%

These high scores indicate that both methods performed exceptionally well in their respective evaluations. The results affirm that Back-Modality preserves the essential characteristics of the original data while introducing diversity, further validating our approach.

11 More Choices of Cross-Modal Generation Models

In our paper, for the Back-captioning with a 10-shot setting, we primarily used the OFA-large model, which yielded a top-1 accuracy of 20.07%. To assess the impact of different model sizes on the outcomes, we also conducted experiments with OFA-huge under the same conditions. The results showed a significant improvement, with the top-1 accuracy reaching 22.12%.

12 Cost of Obtaining the Augmented Samples

Table7 illustrates the additional computational overhead of various augmentation methods compared to the base model on RTX A6000. The primary cost of our method is related to generating images with the diffusion model, and the primary overhead of auto augment is associated with learning augmentation policies. In terms of text augmentation, on the textual entailment task, our method, back-imagination, took 4 hours 13 m 45 s, while the back-translation method took 5 h 38 m 12 s. On the sentiment analysis task, back-speech took 35 m 27 s, whereas the back-translation method took 5 h 22 m 4 s.

13 Discussion About Real-World Applicability

While current multi-modal and cross-modal models have achieved impressive results and continue to rapidly advance, it is worth noting that not all domains currently have easy access to open-source

cross-modal models. This limitation can, to some extent, restrict the effectiveness of our method in real-world applications. However, recent research has increasingly focused on the adaptability of diffusion-based cross-modal models in domains with limited data. This research encompasses areas such as few-shot [Giannone et al., 2022], one-shot [Wu et al., 2023b], zero-shot [Li et al., 2023], domain adaptation [Kim et al., 2023], and unsupervised domain adaptation [Benigmim et al., 2023]. These research directions will further expand the real-world applicability boundaries of our method.

0

5

10

15

20

25

30

35

40

Data Percentage

5

10

15

20

25

30

35

40

45

Back-captioning on Tiny ImageNet

Original Data

Augment Data

Difference

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Augmentation_size

0

5

10

15

20

25

30

35

Back-captioning on Tiny ImageNet

Original Data

Augment Data

Difference

14 Proportion and Size

We conducted an analysis of the proportion of augmented data and the augmentation size by sampling a subset of the data. Figure 6 illustrates the variation in the model's performance when increasing the volume of original data while keeping the augmentation multiple at 5. From the curve in this figure, it is evident that as the volume of original training data increases, augmented data continues to provide benefits. However, these benefits exhibit diminishing returns as more original data is added. Figure 7 addresses the aspect of extending data generation. It demonstrates that as the augmentation size increases, the model's performance improves. However, after a certain point, the gains tend to plateau. We believe the primary reason for the phenomena observed in these two figures is that the diversity introduced by augmented data leads to performance gains. However, this diversity may not match the affinity of the original data, which can result in diminishing returns.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach to data augmentation that leverages recent advances in generative models. It presents a general framework and experiments with three settings for image, text, and audio tasks, showing promising results for their Back-Modality approach.\n\nSTRENGTHS:\n- The idea is simple but interesting and effective based on the presented experiments.\n- The method is generic and can be extended to other modalities, and it can also incorporate other models as the field evolves. \n- Case and ablation studies show the benefits of using the method and support most of the claims in the paper. Authors also compute statistical significance, which is an important step to validate results.\n\nWEAKNESSES:\n- Experiments are only done with one small model for each task. It would be interesting to see the gains from increasing the model size and if the benefits would increase or diminish.\n- The setup explanation is a bit confusing. For example, when explaining the data-scarce scenarios, it seems like n-shot is used for the number of generated samples per class during training. Still, there is no information about the training set size after sub-sampling or the label distribution on the undersampled training set. It would be better to state the setup clearly, e.g., we have X instances in the training set, with Y classes, and we generate Z samples per class. While most of it can be inferred by reading the appendix and doing some math, the paper would be better presented with more clear numbers.\n- The paper could have more experiments testing the limits of how much data could be generated by these methods that would still yield good improvements. Also, a high resource scenario experiment would be useful.\n\nQUESTIONS:\nThere were two plots that I missed in this paper that would have made it stronger and more insightful. The first is a plot with less sub-sampling of the original data. Maybe with 5, 10, 20, 40, 60, 80, 100% of the training data + the generated instances. Would the proposed method still be useful if I have a lot of data? Or just if I don't have data at all? \n\nThe second plot would be to go further on the generation. Experiments go all the way to 10-shot, but what if we could generate 100, 1000? Is 10 the limit for this approach? \n\nDuring the detailed strategies, you share some tricks that made your model work better, like adding the image label to the prompt. How much of an issue to the final results were the problems you addressed in that section? Could it make it harder to expand the technique to other settings? It would have been interesting to add the original results to the appendix so we know how many problems that was causing.\n\nLIMITATIONS:\nYes, the authors have a limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduced a new data augmentation technique, called back-modality. The augmentation is based on modal transformations. Specifically, instances in the original modality (e.g., image) are transformed to an intermediate modality (e.g., text), augmented in the intermediate modality and then transformed back to the original modality. Experimental results in a few-shot setting on three benchmark datasets show that the proposed augmentation technique produces better results than the base model and also than other existing augmentation techniques.\n\nSTRENGTHS:\nThe proposed augmentation technique leverages recent advancements in transformations between different modalities. \n\nThe strategy is rather general and can be applied to a variety of applications for which cross-modal transformations can be obtained. It does not require access to model weights or fine-tuning, and can thus be seen as \"a variant of the Cross-Modal-Models-as-a-Service (CMMaaS) application\".\n\nExperimental results clearly suggest the advantage of the proposed technique as compared to existing augmentation approaches on the three datasets used.\n\nWEAKNESSES:\nWhile the authors aim to produce a general augmentation strategy based on cross-modal transformations, their actual process involves some decision specific to the modalities considered in the study, meant to minimize the production of low-quality augmentations (and no augmentations are used for images as an intermediate modality). This can make the strategy hard to apply without extensive analysis of the augmentations. \n\nSome of the choices made are rather arbitrary, e.g. why 5 augmentations per instance? Or 5% of the data used to balance the training data? Shouldn’t the size of the training data be given by the number  of shots per class? \n\nThe discussion on diversity and affinity is very brief in the main paper. Some details are provided in the appendix. It is not clear why they were not included in the paper (the current paper has only 8 pages). Same for the details of the case study regarding the back-imagination and back-speech augmentations.\n\nQUESTIONS:\nTo minimize the low-quality back-captioning augmentations, the authors “explicitly inject the image labels into the text prompts, which leads to the generation of descriptions  that incorporate these finer-grained labels. “ It is not clear if the label information is used during testing, especially when the task is to predict the image label. Please elaborate. \n\nSome of the choices made are rather arbitrary, e.g. why 5 augmentations per instance? Or 5% of the data used to balance the training data? Shouldn’t the size of the training data be given by the number  of shots per class?\n\nLIMITATIONS:\nThe authors emphasize some limitations related to the size of the cross-modal models. Other limitations related to generalizability could be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method to perform data augmentation: Back-Modality. The augmentation process involves translating the data into another modality, perform augmentation in that modality, and translate each augmented other-modality instance back into the original modality. Each of the three steps could produce multiple augmented data. The paper included three instantiations of Back-Modality: Back-captioning (image -> text -> image), Back-speech (text -> audio -> text), Back-imagination (image -> text -> image). The cross-modal translations are done by pre-trained models. These instantiations are then evaluated on few-shot learning settings (1-10 instances per class) within their domains, and the experiments showed that the models trained with Back-Modality-augmented data outperforms those trained with no data augmentation or with other baseline data augmentation techniques. Additional ablation studies showed that augmentations at each of the three steps are necessary to achieve best result.\n\nSTRENGTHS:\nThis paper presented a really interesting idea: to use cross-modal back-translations as data augmentation technique. The idea is straightforward and easy to follow, and the authors demonstrated that the approach works well when data is extremely scarce (1-10 examples per class), generating augmented data that is more diverse and helpful compared to other existing data augmentation methods.\n\nWEAKNESSES:\nA lot of key experiment details are missing (which I can't find in either main text or appendix), including but not limited to:\n(1) How many augmentations are generated at each cross-modal translation step and each other-modality-augmentation steps for every original data point (i.e. the multipliers at F,G,H for each instantiation)? The number of diverse augmentations is the most important detail for data augmentation experiments.\n(2) There is no description on how which caption augmentations are performed or which GPT model was used during the \"augmentation with GPT\" step.\n(3) How many augmentations are generated with each baseline augmentation methods? Do they match the number of augmentations with Back-Modality?\nWithout these key details, it is difficult to assess the soundness of the experiment results and whether they support the conclusion.\n\nAnother key concern for the new approach is how applicable it is in the real world. This technique relies heavily on the quality of the cross-modal translators, and these pre-trained models usually only work well in domains/tasks where data is not scarce in the first place (since training good cross-modal translators needs a lot of high-quality data). All experiments in this paper are done under artificially created data-scarce settings. In real life, if there is a task that really only have very scarce data (such as medical images for rare disease, etc), the existing cross-modal translators are likely not going to work well with them, as they were not trained with a lot of data like this. I would be more convinced about the practical usability of this technique if the paper included experiments (or just proof-of-concept ones) on performing Back-Modality on a real data-scarcity situation instead of artificially created data-scarcity on the most common domains.\n\nThe presentation of the paper also needs improvement. There are not enough concrete examples of the data augmentations in the main part of the paper. Since there is still plenty of space in the main paper, perhaps adding a few illustrations of one instantiation from beginning to end would help (like the ones in the appendix). Also, the font sizes of the tables are inconsistent. The font sizes of Table 2,3,4 are really big while Table 1,5 are normal sizes.\n\nQUESTIONS:\n(1) Please address the missing key experiment details as mentioned in the weakness section.\n\n(2) I assume that in Table 2,3,4, the number of \"Shot\" means how many real data points from each class are available. Is this correct? If so, what does the 5% subsampling mean under \"Data-scarce scenarios\" of section 3.2?\n\nLIMITATIONS:\nThe authors addressed the limitation of additional computation cost of Back-Modality. I believe there are a few additional limitations that may needs to be addressed, such as:\n(1) the requirement of existing cross-modal translation models that can handle the task-specific data well\n(2) the experiments in this paper are only done on artificially-created data scarcity situations and limited modalities", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pipeline called Back-Modality, which uses cross-modal generation models for data augmentation. In practice, the original data in source modality is first transformed into an intermediate modality using a cross-modal generation model. Then the typical augmentation strategy for the intermediate modality will be applied. The augmented data in intermediate modality will be transformed back into the source modality using another cross-modal generation model in the reverse generation direction. Specifically, three implementations of Back-Modality are explored, including back-captioning, back-imagination, and back-speech. On image and text classification tasks, the proposed augmentation pipeline shows effectiveness.\n\nSTRENGTHS:\nThis paper has the following strengths:\n1. The motivation of Back-Modality is straightforward and the method is easy-to-implement, which has the potential to be widely used in future research works.\n2. Three specific implementations of Back-Modality are proposed and experimented, proving the generalizability of this schema.\n3. The cross-modal generation models used in this work is easy-to-access and has SOTA performance (OFA, stable diffusion), guaranteeing the effectiveness and reproducibility of this pipeline.\n4. The experimental results are good compared with typical data augmentation methods, and data diversity is analyzed.\n\nWEAKNESSES:\nI think this paper will be much improved if the following issues can be more addressed:\n1. Human evaluation on the augmented samples: Clearly, the method of Back-Modality generates more diverse data samples. If human evaluation can be facilitated to verify that the semantic meaning of the original sample are kept and the label of samples are still correct, it will be much better.\n2. Verify robustness in adversarial setting (or some hard test samples): This work mainly conduct experiments on the scenario of limited training samples, this is definitely okay. Meanwhile, another aspect of data augmentation is the robustness on adversarial attacks. If more experimental results on adversarial setting or harder testing samples can be provided, the effectiveness of Back-Modality will be much more solid.\n3. More choices of cross-modal generation models, including different models and the same model with different model scales (like OFA-base compared with OFA-huge, how much performance on augmentation will be affected?).\n4. The cost of obtaining the augmented samples compared with previous (non-model-based) augment pipelines. (pointed in the limitation section for future research direction, but the clarification of current method is still needed)\n\nQUESTIONS:\nThe overall top-1 accuracy score on Tiny ImageNet is low in the experiment. Honestly, I am not so familiar with this ImageNet variant benchmark. Is this a proper choice or a sanity experimental setting here? It would be very good if more discussion can be provided of selecting this as the benchmark.\n\nLIMITATIONS:\nThe authors has pointed out the limitation of computation cost, which is reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper proposed a method to transform data between modalities so as to augment the data. Such a method makes it possible to leverage the multiple existing modalities to generate more useful data to train the model.

STRENGTHS:
1. The proposed method is modality-agnostic so that the initial modality can be transformed into any other modality and reverse back.
2. The generated data vary from the original data, introducing more variety in the training sources.
3. The performance on few-shot settings is largely improved.

WEAKNESSES:
1. The paper's presentation could be improved. The tables and citations are not in good shape.
2. More case study is needed. In the current version, the case study section is not clear, which simply describes the improvement but there are not concrete cases.

QUESTIONS:
See weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11