Back-Modality: Leveraging Modal Transformation for Data Augmentation
Authors: Zhi Li
Yifan Liu
Yin Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: data augmentation
cross-modal

Abstract: We introduce Back-Modality, a novel data augmentation schema predicated on modal transformation. Data from an initial modality undergoes transformation to an intermediate modality, followed by a reverse transformation. This framework serves dual roles. On one hand, it operates as a general data augmentation strategy. On the other hand, it allows for other augmentation techniques, suitable for the intermediate modality, to enhance the initial modality. For instance, data augmentation methods applicable to pure text can be employed to augment images, thereby facilitating the cross-modality of data augmentation techniques. To validate the viability and efficacy of our framework, we proffer three instantiations of Back-Modality: back-captioning, back-imagination, and back-speech. Comprehensive evaluations across tasks such as image classification, sentiment classification, and textual entailment demonstrate that our methods consistently enhance performance under data-scarce circumstances.

Back-Modality: Leveraging Modal Transformation for Data Augmentation
Zhi Li Zhejiang University, China zhili@zju. en
Yifan Liu Zhejiang University, China yifan. liu@zju. edu. en
Yin Zhang"
Zhejiang University, China zhangyin98@zju. edu.. cn
Abstract
We introduce Back-Modality, a novel data augmentation schema predicated on modal transformation. Data from an initial modality undergo transformation to an intermediate modality, followed by reverse transformation. This framework serves dual roles. On one hand, it operates as a general data augmentation strategy. On the other hand, allows for other augmentation techniques, suitable for the intermediate modality, to enhance the initial modality. For instance, data augmenta- tion methods applicable to pure text can be employed to augment images, thereby facilitating the cross- moddaityy of data augmentation techniques. To validate the viability and efficacy of our framework, we proffer three instantiations of Back- Modality: back-captioning, pack-imaginations and back-speech. Comprehensive evaluations across tasks such as image classification, sentiment classification, and textual entailment demonstrate that methods consistently enhance performance under data-scarce circumstances.
Introduction
Neural network-based deep learning models often prone to overfitting, resulting in loss of gener- alization capability due to the limited size of training data. To mitigate this issue, data augmentation techniques routinely employed to generate augmented pool of training samples. The recent past has seen significant strides in the application of data augmentation within diverse fields such speech [Ko al. 2015 Park et al., [209)] computer vision (Simard et al.) 1996, [Shorten and Szegedy et 2014]. and natural language processing [Sennrich et 2016
Khoshgoftaari 2019
Wei and Zoul 2019
Nevertheless, most of these data augmentation algorithms are tailored for
specific modality.
Concomitant with the swift advancement of cross-modal methodologies, various dual cross-modal models, such as text-to-image generation and image captioning, have demonstrated impressive performance. This progress catalyzes development of novel paradigm for data augmentation: Back-Modality, a comprehensive data augmentation framework predicated on modal transformation. Data in the initial modality transformed to intermediate modality and subsequently reversed back. Theoretically, the initial and intermediate modalities can represent any modality, including but not limited to text, image, audio, and video. Our framework retains its universality, contingent on the availability of corresponding dual cross-modal models to execute the modal transformation. Leveraging pretrained cross-modal models, our method is capable of generating wide array of high-quality and diverse data. A comparison illustrating the breadth and quality of the generated data provided in Figure !I
Simultaneously, our framework enables the augmentation method intended for the intermediate modality to benefit the initial modality. For instance, data augmentation methods designed for pure
"Correspondigg author: Yin Zhang.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

goldfish
school bus
Ground truth Random
Mixups
Auto Augment
Ours
Figure 1: Comparative evaluation of images generated by our data augmentation method and other approaches. Capitalizing on the inherent capability of pretrained cross- moosl models, our method excels in generating broad range of high high-quality and diverse images.
text can be leveraged to augment images, fostering cross-modality of data augmentation techniques. In essence, our framework facilitates partial decoupling of data augmentation methods from specific modalities. Moreover, our framework broadens the application landscape of cross-modal models. Given that our methodology obviates the need access model weights undergo further fine-tuning, it can be seen as an application variant of Cross-Modall -oadal--odell------Seeer (CMMaaS),
To substantiate the feasibility and efficacy of framework, present three instantiations of Back-Modality: back-captioning, back-imagination, and back-speech. For back-captioning, the initial modality is image, and utilize image-caption model to generate captions for each image. Subsequently, data augmentation methods designed for text are employed to augment these captions. Finally, text-to-image model is invoked generate variant images, which are then used as augmented samples. For iacgiimainntioon and back-speech, the initial modality is text, and the intermediate modality is image and speech, respectively.
Systematic evaluations across various tasks including image classification, sentiment classification, and textual entailment demonstrate that methods consistently improve performance, particu- larly under data-scarce conditions. Upon further analysis and comprehensive case studies, it has been observed that the data generated via method typically exhibit increased diversity.
Our principal contributions and discoveries can be summarized as follows:
We introduce Back-Modaliny a modality-agnostte data augmentation framework predicated modal transformation. Within this framework, data in the initial modality undergo transformation to an intermediate modality and subsequently reverse back.
Our framework enables the cross-modality of data augmentation methods. For instance, data augmentation techniques developed for pure text can be deployed to augment images.
Our approach extends the application realms of cross-modal models. Our methodology eliminates the need to access model weights or conduct further fine-tuning, hence can be perceived variant of the Molas-Moda-MMoeele -seess-------eecceeee (CMMaaS) application.
Experiments on variety of tasks and datasets substantiate that our methods can consistently enhance performance, particularly in data-scarce scenarios. Further analysis reveals that data generated by method typically exhibits higher degree of diversity.
2 Method
2.1 Framework
Mack-Mod presents a comprehensive, modality-agnostic data augmentation framework predicated on modal transformation. As shown in Figure A symbolizes the initial modality, and B signifies the

---

H: Augmentation Method
Data in Modality
Cross-Modal
Data in Modality B
Crosy-Modal
Data in Modality
A
G: B
Figure 2: Back-Modality constitutes universal data augmentation framework premised modal transformation. A represents the initial modality, and B denotes the intermediate modality. The framework integrates two types of dual cross- moddll tasks: F A + B and G B + A. H signifies the data augmentation methods applicable to the intermediate modality B.
intermediate modality. The framework incorporates two types of dual cross-modal tasks: F A B and G B + A. Here, H denotes the augmentation methods applied to modality B.
Let X denote sampled data point in the initial modality A, and Xaug represents the augmented samples of X. Our framework can be succinctly expressed as:
Xang (((((()))
(1)
Theoretically, AA and B can represent any modality, such text, image, audio, video, and forth. Moreover, AA and can also be the same modality. For instance, back-translation augmentation methods can be perceived as special case of our framework, where A and B are both text in different languages, and is omitted. Essentially, our framework retains its universality, contingent upon the availability of corresponding dual models.
Within our framework, H can assist in augmenting data in modality A by augmenting data in intermediate modality B. This implies that the augmentation method suitable for specific modality can be transferred to other modalities via our framework. More formally, Iet us consider a scenario where F and G' are fixed, with YF.G denoting the corresponding framework. Here, H is treated variable. Consequently, the augmented samples Xaug can be viewed as a function of and H.
X Xong~ (HI,
Given specific X, varying H can yield diverse set of augmented samples X Xaug. In essence, our framework facilitates partial decoupling of data augmentation methods from specific modalities.
Instantiations
In order to demonstrate the feasibility and efficacy of Back-Modalit, we introduce three instantiations of this framework: back-captioning, back boeck-maginntion, and back- speech. These methods encompass three prevalent modalities: text, image, and speech.
Back-captioning the proposed method, we define A as image modality and B as text modality. Here, F represents the task of image captioning, while G represents the task of generating images from text. H corresponds the process of textual data augmentation.
Back-captioning, as a concept, can be deconstructed into trifecta of augmentation methodologies: multi-captioning, caption augmentation, and multi-imagination. It is observed that various observers usually generate captions for the same image that, while being semantically analogous, exhibit subtle variances in detail. This phenomenon mirrors the capability of most image captioning models to generate multiple unique captions for single image, a process we refer to as multi-captioning. Given caption sentence, several text augmentation techniques can be employed to produce multiple related sentences, process we term as caption augmentation. For a given sentence, the images that different individuals visualize generally share similarities, yet present unique details. We refer to this phenomenon as multi mullii-magination. A diffusion based model, employing various random seeds, can effectively simulate this process. Given an image captioning model that generates / captions for each image, caption augmentation process that generates sentences for each caption, and a text-to-image model that generates n images for each sentence, maximum total number of generated augmented sentences per original sentence amounts to m This obviously represents

---

highly efficient data augmentation approach. In practical implementation, we conduct uniform sampling at random on these augmented data obtain the final augmentation dataset.
Back-speech In this proposed approach, we designate modality A as text, while B pertains to the audio modality. Here, F corresponds to the task of text-to-speech generation, and G denotes the automatic speech recognition task. The text-to-speech model is designed to generate one audio file for each input sentence. Following this, audio data augmentation techniques, such as pitch shifting, denoted by H, are applied to these audio files to produce augmented sound files. Subsequently, the automatic speech recognition model translates these augmented sound files back into text, thereby generating augmented sentences.
The notion of spek-s can be elucidated by the following observation: the sentence articulated by an intermediary often bears slight deviations from the original sentence. This is due to the fact that different individuals may have distinct pronunciations for the same sentence, influenced by factors such as speech rate, pitch, regional accents. To simulate these variations, we employ augmentation techniques within the audio modality. As consequence, in the resultant output, some words may be substituted by their homophones. Given that most homonyms bear morphological similarities, with only minor differences at the subword level, this transformation can be viewed as form of subword regularization facilitated by the audio modality.
Back -magiinmainntionn Back-imagination can be construed as the inverse operation of back-captioning. In this approach, we assign modality A to text, while B corresponds to the image modality. Here, F represents the task of generating images from text, and G signifies the image captioning task. Similarly, back-imagination amalgamates three types of augmentation techniques: multi-captioning. image augmentation, and multi calti-imagination..
Method
Back-captioning OFA stable-diffusine v2 augmentation with GPT image classifaction Tiny ImageNet
Back-speech fast-speech2 wav2vec2
Back iackiimagination stable-diffusion v2 OFA
F
G H
pitch shifting/time stretching
Task Dataset
sentiment classification
textual entailment
SST-2
TNCC
Table 1: Statistics of configurations. "augmentation with GPT" means that we employ GPT model augment captions. The symbol "_" indicates that do not apply image augmentation in the process of vack-imagination. Tiny ImageNet [Le and Yang 2013) is an image classification dataset, SST-2 [Socher et al., (2013) is employed for sentiment classification, and TNCC, a textual entailment dataset, is based on Crisscrossed Captions [Parekh et al. 2020). For further details, please refer to the appendix.
3 Experiments
In this section, we initially delineate the tasks, datasets, and models employed in our research. Subsequently, we assess the efficacy of back-captioning, back-speech and back-imagination in data-scarce scenarios, spanning a diverse range tasks and datasets. Finally, we investigate the intrinsic mechanisms underlying the effectiveness of our proposed method from three perspectives: (1) conducting quantitative analysis on the diversity and affinity of the generated samples, (2) performing ablation experiments evaluate the contribution of individual components within the method, and (3) scrutinizing the differences between the original and augmented samples at an instance level and providing in-depth analysis of the transformation effectuated by the augmentation process.
3.1 Baseline and Comparison
We use the performance of the base model, devoid of any data augmentation, baseline to evaluate the effectiveness of our augmentation method in enhancing model performance. For the image classification task, we employ the Resnet-18 18 model [He et al. 2015). We use BERT [Devlin et al.

---

2018] for sentiment classification and textual entailment tasks. To further substantiate the efficacy of our methods, we compare them against several other data augmentation strategies. For the image classification task, we consider four renowned methods: Random Frasing [Zhong et al. [020] Autoaugment [Cubuk et al. 2019), Alignmixup Venkaaaramanann et al. 2022]. and Puzzle Mix [Kim] et al.] 2020]. For the sentiment classification and textual entailment tasks, we opt for the most popular task- aask-agnottic data augmentation methods for comparison, as per Longpre et al. [2020], Yoo et al.
2019] and Back-translation (Sennrich et al. [0016] Fadace et al. [022]] including EDA [Wei and Zou
2017 Edunov et al.) 2018 Ng et al. 2019 We employ the large pre-trained EN-DE and DE-EN
translation models [gg et al. 2019 Yoo al. 2021] for back-translation, which have a size order of magnitude comparable to the cross-modee models used by Back- Modality methods, making them robust baseline. In addition, we also consider some task-specific data augmentation methods for reference, including TMix [Chen et al.
2020
SSMix Yoon et al. 2021]. and Treemix (Zhang et al.
2022
3.2 Datasets and Configurations
Datasets
Tiny ImageNet (Le and Yang. 2013] and the Stanford Sentiment Treebank-2 (SST-2) (Socher et al., 2013) are widely used datasets. Given the scarcity of datasets for understanding natural language in visual scenes, we introduce a novel textual entailment dataset, named Textul Natural Contextual Classification (TNCC). And detailed descriptions can be found in the appendix.
Data-scarce scenarios To showcase our approach, we conduct experiments on artificially data- scarce tasks by sub-sampling the training set. For image classification, sentiment classification, and textual entailment tasks, we perform a balsnbel sub-sample on the training set and we denote the number of data samples for each class in the downsampled dataset as "shot," before any data augmentation is applied. We adhere to statistical rigor in our experiments by executing the sub-samples to 5 different data seeds. Concurrently, for each sub-sample, we train all models using 5 different random seeds. We report the statistical mean for all results. All experiments were conducted using PyTorch and executed on RTX 6000 GPUs and the Atlas computing cluster.
Configurations Table provides a comprehensive view of the configurations used in our experi- ments. Unless explicitly stated otherwise, all pretrained models utilized in our research are obtained from the Huggingface Transformers library 27 WWlf et al. 2019j. The code used to generate augmented samples and the TNCC dataset are both accessible The default augmentation size used in our studies, which includes both our method and the comparison method, is set to 5. As an illustration, in our experiments with Back-captioning, when we have [ for each image (2x2x2), and then, we randomly select 5 of these augmentations to form the final augmented dataset. For back-captioning, we utilize -. augmenting captions using the following prompt: "Maintain the nouns in the following sentence intact and generate semantically diverse sentences." For the process of back -onginaagin,,,on, we consciously exclude image augmentation in our experiments. One reason is that images produced via certain augmentation techniques, such as random erasing (Zhong et al.) 2020 and cutout (DeVs and Taylor 2017), often present substantial challenge to image captioning models. On the other hand, the combination of multi-imagination and multi-captioning appears be sufficient yield satisfactory results.
2. we create 8 augmentations
Detailed Strategies In our experimental process, we observed that simply constructing a pipeline does not necessarily ensure the quality of the augmented data. Consequently, we have devised several strategies aimed at minimizing the production of low-quality or even inaccurate augmented data.
In the case of back-captioning, while the model possesses the capacity to generate highly relevant image descriptions, when deployed for specific tasks such as image classification, more nuanced categories are often required. However, the model sometimes falls short in providing these nuanced labels within the generated descriptions. For instance, when given an image of a 'Persian cat', the model tends to generate descriptions mentioning 'cat' rather than the more specific 'Persian cat'. Several image captioning models, such as the OFA [Wang et al., 2022 model, have the capability to generate image captions conditional on both text prompt and an image. To address this issue, we
Phupp://githubbu Phaps://gttuub.co//uuggggi wtg.:hhww....oooooogggooccconnnn
e hhups://githubbb. hhanp://gituuh.com//hiiiji http:h/wwww..iiiiii//aaeeeaaiiia http::///pttoooooopppiacoo//////
5

---

Shot
Method
10
Base-model
2.40
5.19
8.64
11.75
Random Erasing
2.61
5.77
9.11
12.59
Auto augment
3.41
6.35
9.84
13.23
Alignmixup
4.42
8.19
11.78
14.34
Puzzle Mix
4.48
10.26
12.52
15.66
Back-captioning 10.67
14.55
16.13
20.07
Table 2: The top-1 accuracy metrie for cack-captioning the Tiny ImageNet dataset is displayed
above. The overall accuracy is low, primarily because the model is trained from scratch using scarce data. "Shot" is used to indicate the quantity of images per label.
Shot
Method
10
Base-model
61.18
72.47
81.30
84.57
EDA
61.04
73.89
83.61
85.71
translaannn
62.35
74.07
84.30
85.28
TMix
60.11
71.54
82.77
84.60
SSMix
62.32
74.47
82.89
85.32
Treemix
62.88
73.27
85.15
87.41
mack-imagination
69.20
79.62
88.41
89.14
Table 3: The accuracy of the back-imagination method the TNCC dataset is indicated above. "Shot" signifies the quantity of sentence pairs corresponding to each label.
explicitly inject the image labels into the text prompts, which leads to the generation of descriptions that incorporate these finer-grained labels.
During the back-imaggeeee imagination process, there are occasions when the text-to-image model generates black and white images. This can lead subsequent image captioning models generating sentences like "A black and white photo of," which are clearly inappropriate for augmentation samples. Conse- quently, if the back-imagination process yields black and white image, directly discard it and proceed with a resampling operation. As for back-speech, if the sentences generated have an edit distance from the original sentence that exceeds 20% of the entire sentence length, we discard those sentences as well.
3.3 Results
Tables 3 and Table 4 respectively, display the statistical average accuracy values on the Tiny ImageNet, TNCC, and SST-2 datasets. We carried out hypothesis testing on the results across all tasks, with all p-values falling below 0.05. This suggests that our findings are statistically significant. Our back- back-captioning, back-imagination, and back-speech methods consistently outperform both the base model and other data augmentation methods. This indicates that in data-scarce scenarios, data augmentation based on modal transformation can offer significant performance improvements. In particular, the results of back- imaginatiion demonstrate that our architecture alone, even without the inclusion additional data augmentation methods (H), capable generating effective augmented data.
3.4 Ablation
Table shows the ablation studies of back-captioning, back-imagination and back-speech. The individual components our framework make significant contributions to the overall performance of the model. In particular, augmentation with GPT, pitch shifting and time stretching all improve the model performance, which strongly supports the notion that our framework effectively accomplishes the cross- modslity of data augmentation approaches.

---

Shot
Method
2
3
5
10
Base-model
51.46
52.31
54.42
57.88
61.10
EDA
50.35
52.17
54.87
58.32
61.72
fack-translation
51.30
52.09
55.84
57.69
61.94
TMix
49.71
51.59
54.18
57.62
61.31
SSMix
51.01
52.26
55.32
58.11
61.92
Treemix
51.35
52.18
55.81
58.73
62.37
Back-speech
52.13
52.78
56.11
59.03
63.21
Table 4: The accuracy of the back buck-speech on the SST-2 dataset is provided above." "Shot" refers to the count of sentences associated with each label.
Dataset
Method
Score
Base-model
11.75
+ +Back-captioning
20.07
Tiny ImageNet
w/o multi-captioning
17.21
w/o augmentation with GPT
18.49
w/o multi- aulri-imaginattin
18.92
Base-model
81.30
+ Back- BBack-imagination
88,41
TNCC
wfo multi- multi-imaginatiin
84.71
w/o multi-captioning
86.52
Base-model
57.88
+ +Back-speech
59.03
STS2
w/o pitch shifting w/o time stretching
58.45
58.60
Table 5: Ablation studies. We report the accuracy of shot 10 for back-captioning, shot 5 for both back-imagination and back-speech. Multi-captioning: generate multiple captions for each image. Multi-imagination: generate multiple images for each sentence. Augmentation with GPT: augment captions with GPT model. Pitch shifting and time stretching are different augmentation methods used for speech.
3.5 Diversity and Affinity
Following the methodology outlined by Gontijo-Lopes et al. 2020), we further analyze our augmen-
tation methods based two key dimensions: Diversity and Affinity. Diversity is designed to quantify
the notion that augmentations can prevent model overfitting by increasing the number of samples in the training set. As metric of diversity, [ontijo-Loppe et al. (2020) employ the final training loss of a model trained with a given augmentation. The larger the loss function, the better the diversity of the augmented dataset. Affinity, on the other hand, quantifies how augmentation shifts data with respect to the decision boundary of the clean baseline model. Affinity is defined as the difference between the validation accuracy of model trained on clean data and tested on clean data, and the accuracy of the same model when tested on an augmented validation set. Table indicatee that while ensuring comparable affinity metries, our method consistently presents higher diversity metries compared to other augmentation methods.
3.6 Case Study
Figure I showcases the images generated by our data augmentation method alongside other ap- proaches. Upon further analysis, we have uncovered advantages in the back-captioning method that many traditional augmentation algorithms lack. As illustrated in Figure 3] we have listed several typical data augmentation patterns found in the back-captioning method:
Luminance Modulation: This involves altering the image's brightness by either decreasing
increasing it.
7

---

Dataset
Method
Diversity
Affinity
Base-model
1.594
0
Random Erasing
1.621
-5.44
Tiny imagenet
Auto augment
1.615
-522
capkicaptioning
1.723
-6.04
Base-model
0.0279
EDA
0.0343
-1.14
TNCC
Back-translation
0.0301
-0.91
Back- -mnikianainnnion
0.0677
-0.82
Base-model
0.0089
EDA
0.0126
-7.57
SST-2
tack-translation
0.0104
-6.88
Buck-speech 0.0154
-7.93
Table 6: Statistics of diversity and affinity. The former measures the variety of outcomes, with higher values indicating increased diversity and thus considered favorable. The latter quantifies how augmentation shifts data with respect to the decision boundary of the clean baseline model, where value closer signifies greater affinity.
Luminance Modulation
Structural Preservation
goldfish
school
cash machine
tabby
Attribute Adjustment
Entity Elimination
candle
bathtub
lemon
ice cream
Figure 3: By comparing the original image with the augmented image using back-captioning method, we find typical data augmentation patterns: Luminance Modulation: Adjusting the image brightness either by decreasing or increasing Structural Preservation: Maintaining high similarity of the target structure while altering the background. Attribute Adjustment: Modifying the target's s attributes such as size, thickness, and quantity. Entity Elimination: Removing non-target entities.
2. Structural Preservation:
This technique focuses on preserving the primary subject's structure while making changes to the background elements.
3. Attribute Adjustment: Adjustments made the target's attributes, such size, thickness, and quantity.
4. Entity Elimination: image.
This technique involves the removal of non-target entities from the
We also perform extensive case study back-imagination and back-speech. As depicted in Figure we notice that augmented sentences, generated by different modal transformations, display unique characteristics. Further analysis reveals three notable advantages of our methods:

---

Some aigmented cases of tacck-imagination:
(1) Add' Remove detail
The black dog rums through the water. black dog namning through the water in like.
young
black and yellow jacket guzing something and smilingg -+ young man in black and yellow jacket is smiling.
(2) Replace word by synonyn' by heppenymm' hyponym
is throwing frishee bench.
woman playing frichee bench.
A man is stunding in frunt of alviccuper. - man standing in front a building
children on
secsow
Two bove small
(3) Reverse the order
boy strolls by a pond
boy walks near pond.
Change /blur the quantity
Three people are walking on a path in meadow. - A group of people are walking on path meadow.
Tense variation
Two children sit on a small seesow the sand... - Two children is sitting a small seesaw in the sand.
Some sugmentedd cases of back-spoolic
(1) Split imerge the words
That tragic
superficial treatment..
That trugic superficial treatment
Remains utterly satisfied remain the same throughout...
-
Remain suttery satisfied to remain the same throughout,
(2). Remove
He hoped there would be for dinner, turnigs and currats. Ile hoped there would be for dinner, turnps...
(3) Replace word by homonym
This shot of cumnery
1932. This shot of carmery
pineteen thirty-to..
and some dignity to dumb story - Lend some dignity to dome story.. yea''s best and unprodictable camedy. The best and most unpredictable comedy.
(4) Change punctuation
Se_
ability
ability
Figure 4: By comparing the original sentence with the augmented sentence, summarize several types from the perspective of linguistics. It is obvious that the linguistic transformations of our methods are diverse.
1. Leveraging cross-modal models, our back sock-imagination augmented sentences demonstrate exceptional fluency, readability, and minimal spelling or grammar errors. Correspondingly, the generated images consistently maintain a high level of quality.
2. Our approach introduces greater diversity in both augmented sentences and images. Notably, augmentation types such as adding details are challenging achieve using popular text augmentation methods like EDA and bock-translation.
3. Our method ensures better semantic consistency. For instance, while both EDA and our method involve deletion operations, our approach selectively removes details. But EDA performs random deletion, which increases the likelihood of altering the original semantics. A thorough analysis of back-speech augmentation reveals that certain words in augmented sentences are substituted with similar somidir- words, often sharing similar but not identical subwords. Additionally, the augmentation process involves splitting and merging certain words. Consequently, the augmentation of back-speech can be considered as form of regu- larization akin techniques such BPE-Dropout (Provilkov et al., 2020) and SwitchOut [Wang al. 2018
4
Related Work
4.1 Data Augmentation
Data augmentation widely utilized in various domains address limited data scenarios, and has achieved significant success. In computer vision, basic image manipulations such as translation and rotation can generate new samples (Ronneberger et al. 2015 Krizhevsky al., [017]. Techniques like random erasing [Zhong et al.] [020] and cutting [Devries and Taylor] 2017] aid in improving generalizability by occluding images. Pitch shifting and time stretching are popular methods in speech processing [Salamon and Bello 2016 Moreno- Mareno-BBreaa et al. 2018). Text augmentation methods typically involve word-level text editing operations and sentence-leve text generation. For word-level text editing operations, word substitution is commonly employed, including techniques such as

---

synonym replacement Zhang et al. 2015],
KNN replacement Wang and Yang, 2015]. Uniform Bi-RNN Contextual replacement [obbyashi 2018
and TF-IDF replacement Xie al. 2020
CBERT [Wu et al., 2 and LAMBADA (Anaby- Anaby-Tavor et al. 2019]. EDA (Wei and Zou 2019
comprehensively explores text editing operations, including synonym replacement, insertion, swap, and deletion, for data augmentation. The popular text augmentation method based on generation is nack-translation [Sennrich et al. 2016 Fadace et al. 2017 Edunov et al. 2018). It is evident that previous data augmentation methods have been primarily tailored for specific modalities.
4.2 Dual Cross-modal Tasks and Models
In recent years, large-scale pretrained foundation models have achieved remarkable success across spectrum of tasks. Contemporary research [Zeng et al., 2022 Wu et al., [202aa Shen et al., 2023) suggests the amalgamation of multiple large pretrained models for the execution of novel downstream multimodal tasks. However, our approach diverges from the aforementioned studies, as we employ dual cross-modal models devise a fresh data augmentation paradigm. There exists plenitude
of dual cross-modal tasks and pretrained models, such as lext-to-image [Ho et al., [2220] Rombach
al. 2022]. image captioning [Wang et al. 2022 Li al. [022]. text-to-video [Singer et al. 2022
Hong et al.
2022). video captioning Yan et al. 2022 Yamazaki et al. 2022), text-to-speech Ren
2000 and automatic speech recognition [aevski al. 2020 Dual Learning [He et al. 2016
Xia et al. 2017 2018) protocols train the models of two dual tasks in tandem, while consciously
leveraging the probabilistic correlation between them to regularize the training process. contrast,
our methodology primarily centers on the usage of the large pretrained models of two dual tasks for data augmentation, circumventing additional training.
Limitations
Compared existing data augmentation techniques, Back-Modality typically requires additional computational resources and inference time due to its reliance on large pretrained cross- modal models. However, as advancements continue in the cross-modal field and efficient machine learning methodologies are further employed, we anticipate the development of more efficient techniques. As such, this limitation is expected to be progressively alleviated. Furthermore, while our framework is inherently versatile, specific instances may require the design of corresponding strategies akin to those discussed in Section [32 which ensure label invariance the generated samples and maximize their affinity with the original data.
6 Conclusion
In this work, we introduce a novel data augmentation framework, predicated on modal transformation, and confirm its feasibility and effectiveness. We posit that as the field of cross-modal learning contin- ues to evolve, modal transformation can present a fresh avenue for developing data augmentation techniques. This emerging field presents plethora of intriguing research questions that merit further exploration. For instance, a comprehensive evaluation of our method across a broader spectrum of tasks and modalities would be insightful. In addition, investigating the impact of fine-tuning the cross-modal models on domain-specific datasets on the quality of the generated data could yield meaningful insights.
Acknowledgments
This work was supported by the NSFC project (No. 62072399), Zhejiang Provincial Natural Science Foundation of China under Grant No. LZ23F020009, China Research Centre on Data and Knowledge for Engineering Sciences and Technology, MoE Engineering Research Center of Digital Library, and the Fundamental Research Funds for the Central Universities. Thanks to the Hangzhou AI Computing Center for providing the Atlas computing cluster. We also express our sincere gratitude to anonymous reviewers for their invaluable feedback and constructive comments, which significantly contributed to the improvement of this paper.
10

---

References
Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur: Audio augmentation for speech recognition. In Interspeech, 2015. URLhhtps:://an http//pp...mmmnicsschllllaa.. org/
7360763
Daniel S. Park, William Chan, Yu Zhang, Chung-Chere Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Gernot Kubin and Zdravko Kacic, editors, Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 2613-2617. ISCA, 2019. doi: 10.2 10.21447/lntersppeec... 2019-2600 21437/Interspeech. -ttppeeecch.221---1 2680
URL https://doi.orr org/10
Patrice Y. Simard, Yann LeCun, John S. Denker, and Bernard Victorri. Transformation invariance in pattern recognition ttangent distance and tangent propagation. In Neural Networks, 1996. URL
https://api. semanti cscholar.
wochooaao.oo///orppposootooooeee wrg//orrpsssD........ 1a0isIDDD:6663335 60673566
Connor Shorten and Taghi M. Khoshgoftaar. A survey on image data augmentation for deep learn- wwwpiicccoooooooooooooooo
ing. Journal of Big Data, 6:1-48, 2019. URL https://api.
195811894
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan. Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1-9, 2014. URL https://api. wwmanticscoollaa orggCorppsID: 206592444
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany, Au- gust 2016. Association for Computational Linguistics. doi: 10. 10.18653/1//116-10999 URL https:://cllantoooogy.oorg orwp://-...//1611/909
Jason W. Wei and Kai Zou. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 6381-6387 Association for Computational Linguistics, 2019. doi: 10. 10.18653/v//11--11 1670. URL (tllstttps:: //ooi. www.eiorrrr... 18653/v1/019-1670
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionallity over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing, 2013. URL https
//api. www.stmsccchhoolrrrreoooo http:/oo....rrrrr/oorps/...99900 990233
Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for ms-coco. arXiv preprint arXiv:2004. 15020, 2020.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015. URL hhttts://app.
www.sscoolar.rrr.. wrg. ptrg/CorruslD: httpss///ap.semmtticchh.o.aaa
206594692
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep 2018.
bidirectional transformers for language understanding. arXiv preprint
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data aug- mentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation strategies from data. 2019 IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 113-123, 2019. URI hhtp::tttt/appppp wwwanticscholar.oogg
CorpusID: 1 166282660

---

Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. Alignmixup: Improving representations by interpolating aligned features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19174-19183, 2022.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pages 5275-5285.
PMLR, 2020.
Shayne Longpre, Yu Wang, and Chris DuBois. How effective is task-agnostic data augmenta- tion for pretrained transformers? In Findings of the Association for Computational Linguistics; EMNLP 2020, pages 4401-4411, Online, November 2020. Association for Computational Lin- guistics. doi: 10. 18653// http:/2....2nnseeeeeme http:///4..////ttts//////atatat /tai:::://aclaathooog)/) org/ 2020 findings-emnlp... 394
URL https::s::///
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang- Woo Lee, and Woo-Myoung Park. Gpt3mix: Leveraging large-scale language models for text augmentation. In
Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computa- tional Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 Novem- ber, 2021, pages 2225-2239. Association for Computational Linguistics, 2021, doi: 10. 1/ 2021.finding. www.iings--emlllppp 192. httpss/t::///o org/ 0. 18653/v1/2921, f findingsnnnnnneenneennnn emnlp. 192
Marzieh Fadace, Arianna Bisazza, and Christof Monz. Data augmentation for low-resource neural machine translation. In Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 Augustt 4, Volume 2: Short Papers, pages 567-573. Association for Computational I Linguistics. Jttps/tss:///o/ org/10 18653/v1/P17-2099
2017. doi: 10. 00.18653/11/11722000
Sergey Edunov, Myle Out, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. October-Novembee 2018. Association for Computational Linguistics. doi: 10.18553/w//D18 1045. URL htttps:://alaantoollgg wrwp://10g.ooo/1--1111/01
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Out, Michael Auli, and Sergey Edunov. Facebook FAIR's 's WMT19 news translation task submission. Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314-319, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/1/W119333 https///adaantooogg.orrg////
URL
Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistically--nfomeed interpolation of hidden
space for semi-supervised text classification. arXiv preprint arXiv:2004. 12239, 2020.
Soyoung Yoon, Gyuwan Kim, and Kyumin Park. Ssmix: Saliency-based span mixup for text
classification. arXiv preprini arXiv:2106. 08062, 2021.
Le Zhang, Zichao Yang, and Diyi Yang. TreeMix: Compositional constituency- based data augmenta- tion for natural language understanding. In Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5243-5258, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. htt5533/1120222aacclmmooo
URL htttt::////claatoologgg...o wwwp:/rgg2200002 2022. naacl-main. 385
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of -ft----rttt--tttt natural language processing. arXiv preprint arXiv:1910.03771 2019.
Terrance Dries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1xXiv:1708 04552, 2017.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sedd to-suueece learning framework. arXiv preprint arXiv:2202. 03052, 2022.
Raphael Montijo-Loppes Sylvia J Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.08973, 2020.
12

---

Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. Bpe-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882-1892, 2020.
Xinyi Wang. Hieu Pham, Zihang Dai, and Graham Neubig. Switchout: an efficient data augmentation algorithm for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 856-861, 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- assisted Intervention, pages 234-241. Springer, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. CoRR, htss/1708.0455 2017. URL /BLL/ttt:///arxx.orr///////// 04452
Justin Salamon and Juan Pablo Bello. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Processing Letters, 24:279-288 283, 2016. URL
https://api. http:////pppssmaatiiiiee. www.stmmiiii.... wwwpannccscchlll..cccoooooaaaaae wwtpaiiico.....oo//ro/o/ss///s/a 3537444 3537408
Francisco J. Moreno-Barea, Fiammetta Strazzera, José M. Jerez, Daniel Urda, and Leonardo Franco. Forward noise adjustment scheme for data augmentation. 2018 IEEE Symposium Series on
Compulational Intelligence (SSCI), pages 728-734, 2018. https:////ppp semanticscholar
Org/CorpusID: 54048603
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama. and Roman Garnett, editors, Advances in Neurul Informution Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657, 2015. httpssttpp::t....m p/r.ceedis///oooiiing cewpeeddiiinnnnneiissiiieeee cc/ cc/pappr//0p0 2015/ (ttp//220f8851772220 04771333388884444476770022220200
William Yang Wang and Diyi Yang. That's so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach automatic categorization of annoying behaviors using #pet- peeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2557-2563, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10. 10.18653/11/D15-13000
URL httptttt////aalantooolooo
org. org/ 1306
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33:6256-6268, 2020.
Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 2072
10.18653/v1/N18- 2072.. URL https: /
Xing Wu, Shangwen Lv. Liangjun Zang, Jizhong Han, and Songlin Hu. Conditional BERT contextual augmentation. In João M. F. Rodrigues, Pedro J. Cardoso, Jânio M. Monteiro, Roberto Lam, Valeria V. Krzhizhanovskaya, Michael Harold Lees, Jack J. Dongarra, and Peter M. A. Sloot, editors, Computational Science ICCS 2019- 19th International Conference, Faro Portugal, June 12-14 2019, Proceedings, Part IV, volume 11539 of Lecture Notes in Computer Science, pages 84-95. Springer, 2019. doi: 10.1007/978-3--000 007777788--000002277-000001 URL https:://dii.rrg// 1007/ 10--3-030-22777--0
22747-0
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, N. Tepper, and Naama Zwerdding. Do not have enough data? deep learning to the rescue! In AAAI Conference on Artificial Intelligence, 2019. URL, hhtpp:: //api. wwwanticschol.ar.oomm
CorpusID:212821551
13

---

Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveck Puro- hit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratie models: Composing shro multimodal reasoning with language. arXiv preprint arXiv:2204.005598 2022.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04677 2023a.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang, Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 17580, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision- Asiion-languag learning by cross- modal skip-connections. arXiv preprint arXiv:2205.12000 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14770 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-videe generation via transformers. arXiv preprint arXiv: 15868, 2022.
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:22/2.0497, 2022.
Kashu Yamazaki, Khoa Vo, Sang Truong, Bhiksha Raj, and Ngan Le. Vitint: Visual- linguistic -annsfoee for coherent video paragraph captioning. arXiv preprint arXiv:2211. 1,.152 2022.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text speech. arXiv preprint arXiv:2006.0455 2020.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 13:12449-12260 2020.
Di He. Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 820 828, 2016 URL https: 06669966336661003366799992288883 Abstrasttebbtttttt.t.. html
Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie- Yan Liu. Dual supervised learning. In International conference on machine learning, pages 7789-3779.. PMLR, 2017.
Yingce Xia, Xu Tan, Fei Tian, Tao Qin, Nenghai Yu, and Tie- Yan Liu. Model-level dual learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. ICML. 2018, Stockholmsmmsssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3379-5388 PMLR, 2018. URL http://proocedingss mlr, prwss//80//e xial html
Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models. arXiv preprint arXiv:2205.15463, 2022.
14

---

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023b.
Yunxiang Li, Cha-Chich Shao, Xiao Liang, Liyuan Chen, Ruiqi Li, Steve Jiang, Jing Wang, and You Zhang. Zero-shot medical image translation via frequency frequency-guided -uudedd diffusion models. arXiv preprint arXiv:2304.00742, 2023.
Gwanghyun Kim, Ji Ha Jang, and Se Young Chun. Podia-3d: Domain adaptation of 3d generative model across large domain gap using pose-preserved text-to-image diffusion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 22603-22612, 2023.
Yasser Benigmim, Subhankar Roy, Slim Essid, Vicky Kalogeiton, and Stéphane Lathuilière. One- shot unsupervised domain adaptation with personalized diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 698-708, 2023.
15

---

Appendix
7 Back-imagination and Back-speech
black dog leaps over log.
image caption
dog leaps
black dog jumps over log.
Roc-speck
A black dog leaps over log.
speech
speech augoeentatiu
aufomatic sprech
recegnition
A black dog leap over log.
back dell leaps log.
Figure 5: The illustrative examples for two proposed techniques: Back-imagination and Back-speech.
we present illustrative examples to facilitate better understanding of two proposed techniques: Back-imagination and Back-speech.
As shown Figure 5
8
Datasets
Tiny ImageNet [Le and Yang. 2015 serves compact version of the comprehensive ImageNet dataset. comprises 100,000 images spanning 200 classes, with 500 images per class, and these images are downsized to 64x64 pixels. Each class is furnished with 500 training images, 50 validation images, and 50 test images.
The Stanford Sentiment Treebank-2 (SST-2) Socher et al. 2013 is sentiment classification dataset populated with movie reviews gathered from Rotten Tomatoes, paired with their corresponding binary labels. The dataset is partitioned into training, validation, and testing sets, comprising 67, 67,349, 872, and 1,821 instances, respectively.
Given the scarcity datasets for understanding natural language in visual scenes, we introduce a novel textual entailment dataset, named Textual Natural Contextual Classification (TNCC). This dataset is formulated on the foundation of Crisscrossed Captions [Parekh et al. 2020]. an image captioning dataset supplied with human-rated semantic similarity ratings continuous scale from 0 to 5. We tailor the dataset to suit a binary classification task. Specifically, sentence pairs with annotation scores exceeding are categorized as positive (entailment), whereas pairs with scores less than 1 are marked as negative (non- monneentallmnt).. The TNCC dataset is partitioned into training, validation, and testing sets, containing 3,600, 1,200. and I. 1,560 instances, respectively. This dataset will be made available alongside our source codes.
9 Configurations
In this work, we employ uniform experimental configuration for both textual entailment and sen- timent classification tasks. We adopt BERT-BASE [Devlin al. 2018). model pretrained using Masked Language Modeling (MLM), as our primary experiment subject. For each individual down- stream classification experiment, the classification model initialized with the pretrained parameters from the BERT-BASE model. The classifier component, comprising of two fully connected layers that deduce class labels from the output embeddings generated by the transformer architectures, randomly initialized. During the training phase, leverage the Adam optimization algorithm with learning rate set at 5e 5, the first and second momentum terms, 31 and 32, are respectively set to 0.9 and 0.999. Additionally, introduce L2 weight decay of 0.01 to the model. We select a batch
16

---

size of 2 for all trials. We save model checkpoints during training and ultimately employ the best checkpoint- determined based on performance on the validation dataset-for testing. The results are presented as classification accuracies both datasets under investigation.
For the image classification task, employ the ResNet18 (He et al. 2015) model, which is considered more suitable for small datasets. We initialize all learnable layer parameters randomly. During the training process, employ the SGD optimizer with learning rate of 0.1, momentum of 0.9, and a weight decay of 0.0001.
10 Human Evaluation on Augmented Samples
In response your suggestion, we conducted human evaluation on the sampled augmented data. The results of the evaluation are as follows:
For the images generated using the back-captioning method:
Label Invariance Score: 99.2%
For the sentences generated using the back-imagination method:
Semantic Consistency Score: 98.8%
These high scores indicate that both methods performed exceptionally well in their respective evaluations. The results affirm that Back-Modality preserves the essential characteristics of the original data while introducing diversity, further validating our approach.
11 More Choices of Cross-Modal Generation Models
In our paper, for the Back-captioning with 10-shot setting, we primarily used the OFA-large model, which yielded top-1 accuracy of 20.07%. To assess the impact of different model sizes on the outcomes, we also conducted experiments with OFA-huge under the same conditions. The results showed significant improvement, with the top-1 accuracy reaching 22.12%.
12 Cost of Obtaining the Augmented Samples
Method
Additional Computational Time
Randigg
55 s
Puzzle Mix
h 29 m 25 h 59 m 45
Alignmixup
Back-captioning (our method)
11 35
Auto augment
About 49
Table The additional computational overhead of various augmentation methods compared the base model.
the additional computational overhead of various augmentation methods compared the base model on RTX A6000. The primary cost of our method related generating images with the diffusion model, and the primary overhead of auto augment is associated with learning augmentation policies. In terms of text augmentation, on the textual entailment task, our method, back-imagines ireeiiiaginntion, took 4 hours 13 m 45 s, while the back-translation method took 5 h 38 12 On the sentiment analysis task, back-speech took m 27 s, whereas the mack-rranslation method took h 22 m s.
13 Discussion About Real-World Applicability
While current multi-modal and cross-modal models have achieved impressive results and continue to rapidly advance, it is worth noting that not all domains currently have easy access to open-source
17

---

cross-mos cross-modal models. This limitation can, to some extent, restrict the effectiveness of our method in real-world applications. However, recent research has increasingly focused on the adaptability of duffusion-based cross-modal models in domains with limited data. This research encompasses areas such as few-shot [Giannone et al. 2022], one-shot [Wu et al.) [023b]. zero-shot [Li et al., 2023
domain adaptation |Kim et al. 2023], and unsupervised domain adaptation [Benigmim et al. 2023
These research directions will further expand the real-world applicability boundaries of our method.
Back-captioning Tiny ImageNet
Augment Data
40
35
30
Accuracy
25
20
15
10
5
10
15
20
25
30
35
40
Percentage
Figure 6: Effect of increasing original data.
Back-captioning Tiny ImageNet
Original
Augment Data
35
30
25
Accuracy
20
15
10
2.5
5.0
7.5
10.0 Augmentation,
12.5
15.0
17.5
20.0
Figure 7: Effect of increasing augmentation multiple.
18

---

14 Proportion and Size
We conducted an analysis of the proportion of augmented data and the augmentation size by sampling subset of the data. Figure 6 iillustrates the variation in the model's performance when increasing the volume original data while keeping the augmentation multiple at From the curve in this figure, it is evident that as the volume of original training data increases, augmented data continues to provide benefits. However, these benefits exhibit diminishing returns as more original data added. Figure adddesssess the aspect of extending data generation. It demonstrates that as the augmentation size increases, the model's performance improves. However, after certain point, the gains tend to plateau. We believe the primary reason for the phenomena observed in these two figures is that the diversity introduced by augmented data leads to performance gains. However, this diversity may not match the affinity of the original data, which can result in diminishing returns.
19


Reviews:
[{"review_id": "dVdq40zewZ", "reviewer": "Reviewer_WjzN", "summary": "The paper proposed a method to transform data between modalities so as to augment the data. Such a method makes it possible to leverage the multiple existing modalities to generate more useful data to train the model.", "questions": "See weaknesses", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method is modality-agnostic so that the initial modality can be transformed into any other modality and reverse back.\n2. The generated data vary from the original data, introducing more variety in the training sources.\n3. The performance on few-shot settings is largely improved.", "weaknesses": "1. The paper's presentation could be improved. The tables and citations are not in good shape.\n2. More case study is needed. In the current version, the case study section is not clear, which simply describes the improvement but there are not concrete cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WjzN,\n\nThank you for the time and effort you put into reviewing our paper. We appreciate your comments and your recognition of the strengths of our proposed method. Here, we would like to address the weaknesses and questions that you raised in your review:\n\n1. **Presentation**: We understand that you found the tables and citations not in good shape. We acknowledge this issue, and we will revise the paper to enhance the layout and formatting. For example, we will standardize the font sizes across all tables to enhance the visual coherence and readability of the paper. By organizing the tables and citations meticulously, we will make sure to provide a clearer and more accessible presentation. Additionally, if there are other particular formatting or stylistic concerns, such as alignment, captioning, or citation style, please do point them out. We are committed to adhering to the preferred presentation guidelines, and your precise guidance will enable us to address these aspects meticulously.\n\n2. **More Case Study Needed**: We value your suggestion to provide more concrete cases in our paper, and we would like to clarify the following two actions we will take to address this concern:\n    \n    a. **Moving Content from Appendix**: In fact, we have provided more extensive case studies, along with detailed analysis of back-imagination and back-speech augmentations, in the appendix of the current submission. Recognizing the importance of these details in the main text, we will relocate this content from the appendix to the relevant section in the paper. This move will ensure that readers have direct access to the comprehensive case studies without the need to refer to supplementary material.\n    \n    b. **Additional PDF for More Case Studies**: In response to the request for a more exhaustive exploration, we will create a PDF dedicated to additional case studies. This material will further illustrate the applications and potential benefits of our method.\n    \n\nBy implementing these measures, we believe we can provide a more comprehensive and    accessible understanding of our method's functionality and significance. We are committed to offering a thorough analysis and are confident that these adjustments will align with your expectations.\n\n3. **Rating**: While your rating indicates a borderline reject, we believe that addressing the above weaknesses will enhance the paper's quality and contribute to the overall field. Since the soundness and contribution have been rated as good, we kindly request reconsideration of the rating after we implement the proposed revisions.\n4. **Other Comments**: If there are additional specific concerns or suggestions not covered in the review, please do share them with us. We are committed to making all necessary improvements to ensure the paper meets the standards of the conference.\n\nIn conclusion, we believe that the concerns you raised can be addressed through targeted revisions. Thank you once again for your constructive feedback, and we look forward to hearing from you soon.\n\nBest regards,\n[Author(s) of Submission7875]"}, {"review_id": "ZnzjZmMhHo", "reviewer": "Reviewer_UnFZ", "summary": "This paper proposes a pipeline called Back-Modality, which uses cross-modal generation models for data augmentation. In practice, the original data in source modality is first transformed into an intermediate modality using a cross-modal generation model. Then the typical augmentation strategy for the intermediate modality will be applied. The augmented data in intermediate modality will be transformed back into the source modality using another cross-modal generation model in the reverse generation direction. Specifically, three implementations of Back-Modality are explored, including back-captioning, back-imagination, and back-speech. On image and text classification tasks, the proposed augmentation pipeline shows effectiveness.", "questions": "The overall top-1 accuracy score on Tiny ImageNet is low in the experiment. Honestly, I am not so familiar with this ImageNet variant benchmark. Is this a proper choice or a sanity experimental setting here? It would be very good if more discussion can be provided of selecting this as the benchmark.", "limitations": "The authors has pointed out the limitation of computation cost, which is reasonable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has the following strengths:\n1. The motivation of Back-Modality is straightforward and the method is easy-to-implement, which has the potential to be widely used in future research works.\n2. Three specific implementations of Back-Modality are proposed and experimented, proving the generalizability of this schema.\n3. The cross-modal generation models used in this work is easy-to-access and has SOTA performance (OFA, stable diffusion), guaranteeing the effectiveness and reproducibility of this pipeline.\n4. The experimental results are good compared with typical data augmentation methods, and data diversity is analyzed.\n", "weaknesses": "I think this paper will be much improved if the following issues can be more addressed:\n1. Human evaluation on the augmented samples: Clearly, the method of Back-Modality generates more diverse data samples. If human evaluation can be facilitated to verify that the semantic meaning of the original sample are kept and the label of samples are still correct, it will be much better.\n2. Verify robustness in adversarial setting (or some hard test samples): This work mainly conduct experiments on the scenario of limited training samples, this is definitely okay. Meanwhile, another aspect of data augmentation is the robustness on adversarial attacks. If more experimental results on adversarial setting or harder testing samples can be provided, the effectiveness of Back-Modality will be much more solid.\n3. More choices of cross-modal generation models, including different models and the same model with different model scales (like OFA-base compared with OFA-huge, how much performance on augmentation will be affected?).\n4. The cost of obtaining the augmented samples compared with previous (non-model-based) augment pipelines. (pointed in the limitation section for future research direction, but the clarification of current method is still needed)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UnFZ,\n\nThank you for your thoughtful review and constructive feedback on our submission. We appreciate your recognition of the method's potential and your careful assessment of its strengths. In response to the weaknesses and questions you highlighted, we would like to provide the following clarifications and details:\n\n1. **Human Evaluation on Augmented Samples**:\n\nIn response to your suggestion, we conduct a human evaluation on the augmented data. The evaluators consist of five crowdsourced workers, and the final scores are the average of them. The results from the evaluation were as follows:\n\nFor the images generated using the back-captioning method:\n\n- Label Invariance Score: 99.2%\n\nFor the sentences generated using the back-imagination method:\n\n- Semantic Consistency Score: 98.8%\n\nThese high scores suggest that both methods have performed remarkably well in their respective evaluations. The results affirm that Back-Modality maintains the essential characteristics of the original data while adding diversity, thus further validating our approach. \n\n1. **Robustness in Adversarial Settings**:\n\nWe agree that the robustness of Back-Modality in adversarial settings or with harder testing samples would solidify our claims. In the revised version, we will include results on adversarial settings, providing further evidence of the effectiveness of our method. To assess the robustness of our model, we reported both the accuracy before and after the adversarial attacks, along with the absolute drop in accuracy due to the attacks. For image classification tasks, we employed the *Universal adversarial perturbations* technique. The results are as follows:\n\n- Base model: 11.75% to 5.23% (a decrease of 6.52%)\n- Random Erasing: 12.59% to 5.60% (a decrease of 6.99%)\n- Auto augment: 13.23% to 8.60% (a decrease of 4.63%)\n- Alignmixup:  14.34% to 7.88% (a decrease of 6.46%)\n- Puzzle Mix:  15.66% to 8.49% (a decrease of 7.17%)\n- Back-captioning: 20.07% to 14.02% (a decrease of 6.05%)\n\nFrom the results, it's evident that Back-captioning retains a leading accuracy post-attack, showcasing good robustness. In terms of absolute value drop, it stays in the middle ground among the evaluated methods.\n\nFurthermore, for textual data augmentation techniques, we evaluated robustness using various adversarial attack methods. For the textual entailment task, we utilized the bert-attack method. The results were:\n\n- Base model: 84.57% to 7.23% (a decrease of 77.34%)\n- Back-imagination: 89.14% to 16.76% (a decrease of 72.38%)\n\nFor the SST-2 dataset, we employed the textbugger attack method:\n\n- Base model:  61.10% to 3.95% (a decrease of 57.15%)\n- Back-speech: 63.21% to 8.47% (a decrease of 54.74%)\n\nFrom these results, we can infer that our augmentation techniques, including Back-captioning, Back-imagination, and Back-speech, showcase a notable degree of resilience against adversarial attacks.\n\n3.**More Choices of Cross-Modal Generation Models**\n\nIn the paper, for the Back-captioning with a 10-shot setting, we primarily used the OFA-large model, which gave us a top-1 accuracy of 20.07%. Following your advice, we also experimented with OFA-huge, under the same conditions. The results showed a notable improvement, with the top-1 accuracy reaching 22.12%. \n\n4.**Cost of Obtaining the Augmented Samples**:\n\n The table below reflects the additional computational overhead of various augmentation methods compared to the base model. \n\n| Method | Additional Computational Overhead (RTX A6000) |\n| --- | --- |\n| RandErasing | 4 m 55 s |\n| Puzzle Mix | 1 h 29 m 25 s |\n| Alignmixup | 1 h 59 m 45 s |\n| Back-captioning (our method) | 11 h 35 m |\n| Auto augment | About 49 h |\nThe main cost of our method is in generating images with the diffusion model and the primary overhead of auto augment is in learning augmentation policies.   For the method of text augmentation, on the textual entailment task, our method, back-imagination, took 4 hours 13 m 45 s, while the back-translation method took 5 h 38 m12 s. On the sentiment analysis task, back-speech took 35 m 27 s, whereas the back-translation method took 5 h 22 m 4 s.\n\nQuestion：**Overall Top-1 Accuracy Score on Tiny ImageNet**\n\n1. **About Tiny ImageNet**: In the supplementary material, you'll find an in-depth introduction to Tiny ImageNet[1] from lines 13-16. For clarity, Tiny ImageNet is a compact version of the comprehensive ImageNet dataset. It comprises 100,000 images spanning 200 classes, with each class containing 500 training images, 50 validation images, and 50 test images. The resolution of these images is 64x64 pixels. By employing this dataset, we ensure that the evaluation process remains robust yet computationally manageable.\n2. **Relevance in Recent Research**: It has been used extensively in various studies, ranging from data augmentation [2], to self-supervised learning [3], and open-set recognition [4], among others. Its widespread adoption underscores its significance and suitability as a benchmark for various experimental setups.\n3. **Regarding the Accuracy Score**: We understand the concern regarding the \"low\" top-1 accuracy score on Tiny ImageNet. However, this was by design. The intent was to simulate a data-scarce scenario. Our primary aim was to demonstrate the efficacy of our method, while also ensuring a judicious use of computational resources. \nIn light of the above, we firmly believe that Tiny ImageNet was an appropriate choice for our experiments. \n\nReferences:\n- [1] Tiny ImageNet Visual Recognition Challenge\n- [2] Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup\n- [3] OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning\n- [4] Learning Placeholders for Open-Set Recognition.\n\nWe appreciate your “Weak Accept” rating and the confidence you placed in your assessment. We believe that the modifications and clarifications proposed above will address your concerns and elevate the contribution of our paper."}, {"review_id": "zcM6st7krq", "reviewer": "Reviewer_Usve", "summary": "This paper introduces a new method to perform data augmentation: Back-Modality. The augmentation process involves translating the data into another modality, perform augmentation in that modality, and translate each augmented other-modality instance back into the original modality. Each of the three steps could produce multiple augmented data. The paper included three instantiations of Back-Modality: Back-captioning (image -> text -> image), Back-speech (text -> audio -> text), Back-imagination (image -> text -> image). The cross-modal translations are done by pre-trained models. These instantiations are then evaluated on few-shot learning settings (1-10 instances per class) within their domains, and the experiments showed that the models trained with Back-Modality-augmented data outperforms those trained with no data augmentation or with other baseline data augmentation techniques. Additional ablation studies showed that augmentations at each of the three steps are necessary to achieve best result.", "questions": "(1) Please address the missing key experiment details as mentioned in the weakness section.\n\n(2) I assume that in Table 2,3,4, the number of \"Shot\" means how many real data points from each class are available. Is this correct? If so, what does the 5% subsampling mean under \"Data-scarce scenarios\" of section 3.2?\n", "limitations": "The authors addressed the limitation of additional computation cost of Back-Modality. I believe there are a few additional limitations that may needs to be addressed, such as:\n(1) the requirement of existing cross-modal translation models that can handle the task-specific data well\n(2) the experiments in this paper are only done on artificially-created data scarcity situations and limited modalities\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper presented a really interesting idea: to use cross-modal back-translations as data augmentation technique. The idea is straightforward and easy to follow, and the authors demonstrated that the approach works well when data is extremely scarce (1-10 examples per class), generating augmented data that is more diverse and helpful compared to other existing data augmentation methods.", "weaknesses": "A lot of key experiment details are missing (which I can't find in either main text or appendix), including but not limited to:\n(1) How many augmentations are generated at each cross-modal translation step and each other-modality-augmentation steps for every original data point (i.e. the multipliers at F,G,H for each instantiation)? The number of diverse augmentations is the most important detail for data augmentation experiments.\n(2) There is no description on how which caption augmentations are performed or which GPT model was used during the \"augmentation with GPT\" step.\n(3) How many augmentations are generated with each baseline augmentation methods? Do they match the number of augmentations with Back-Modality?\nWithout these key details, it is difficult to assess the soundness of the experiment results and whether they support the conclusion.\n\nAnother key concern for the new approach is how applicable it is in the real world. This technique relies heavily on the quality of the cross-modal translators, and these pre-trained models usually only work well in domains/tasks where data is not scarce in the first place (since training good cross-modal translators needs a lot of high-quality data). All experiments in this paper are done under artificially created data-scarce settings. In real life, if there is a task that really only have very scarce data (such as medical images for rare disease, etc), the existing cross-modal translators are likely not going to work well with them, as they were not trained with a lot of data like this. I would be more convinced about the practical usability of this technique if the paper included experiments (or just proof-of-concept ones) on performing Back-Modality on a real data-scarcity situation instead of artificially created data-scarcity on the most common domains.\n\nThe presentation of the paper also needs improvement. There are not enough concrete examples of the data augmentations in the main part of the paper. Since there is still plenty of space in the main paper, perhaps adding a few illustrations of one instantiation from beginning to end would help (like the ones in the appendix). Also, the font sizes of the tables are inconsistent. The font sizes of Table 2,3,4 are really big while Table 1,5 are normal sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer Usve,**\n\nThank you for taking the time to review this paper and provide detailed feedback. We sincerely appreciate the insightful comments and the critical examination of our work. However, it seems that there may have been parts that were missing or misunderstood. Below we provide responses to the concerns and questions you have raised:\n\n### **Weaknesses and Missing Experiment Details**\n\n1. **Number of Augmentations at Each Step:**\nYou may have missed certain specifics in the main text. On lines 104-105, we mention that \"In practical implementation, we conduct uniform sampling at random on these augmented data to obtain the final augmentation dataset.\" Furthermore, as stated on lines 155-156, our default augmentation size is set at 5. Theoretically, the product of l, n, and m should exceed the required augmentation multiplier.\nFor instance, in our experiments with Back-captioning, where l=n=m=2, for every image, we generate 8 (2x2x2) augmentations, and subsequently, we uniformly sample 5 of these as the final augmented data. For Back-imagination, where l=3, n=1, and m=3, for each sentence, we generate 9 (3x1x3) sentences, and then uniformly sample 5 as the final augmented data.\n2. **Caption Augmentations and GPT Model Used:**\nIn the main text, specifically on lines 156-158, we detail that for back-captioning, \"We utilize the gpt-3.5-turbo model to augment captions, with the prompt being: 'Maintain the nouns in the following sentence intact and generate semantically diverse sentences.'\"\n3. **Comparison with Baseline Augmentation Methods:**\nAs indicated in the main text on lines 153-156, unless explicitly stated otherwise, our default augmentation size is consistently set at 5, which is also applicable to the baseline methods, such as Random Erasing, Auto augment, EDA, Back-translation, etc.\n\n### **Concerns about Real-World Applicability**\n\n1. **Controlled and Reproducible Experimentation:** Firstly, the primary rationale behind our use of artificially created data-scarce settings was twofold: it provided a controlled environment for experimentation, and it ensured reproducibility. Reproducibility is a cornerstone of scientific research, and we wanted other researchers to be able to reliably reproduce our results without the inconsistencies of real-world situations. Our primary objective was to demonstrate the feasibility of our framework — showing that a data augmentation method for one modality can augment data in another modality. While the broader application in real-world scenarios is undeniably valuable, it wasn't the main focus of this paper. \n2. **Generalizability and Domain Adaptation:** You pointed out concerns regarding the quality of cross-modal translators in real data-scarce situations. However, it's crucial to note the generalizability and domain-adaptation capabilities of cross-modal models. As evident in machine learning, models trained on extensive data can sometimes capture patterns useful even in sparse-data situations. The striking generalizability of large language models stands as a testament. The potential for future larger cross-modal models to demonstrate enhanced generalization cannot be discounted. Moreover, recent research has increasingly honed in on the domain adaptability of diffusion-based cross-modal models in scenarios with limited data, encompassing Few-Shot[1], One-Shot[2], Zero-shot[3], Domain Adaptation[4], and Unsupervised Domain Adaptation[5]. The advancements in these areas hold promise in allaying the concerns you've raised. Analogously, when reverse translation techniques in natural language processing were initially introduced, they were primarily applicable in the domain of data-scarce language translation. As technology advanced and more robust translation models were developed, they emerged as popular data augmentation techniques across various NLP domains.\n3. **Proof-of-Concept Experiment:** Taking cue from your suggestion, we embarked on a proof-of-concept experiment. In collaboration with medical experts, we gained access to a diffusion model, akin to [6] and [7], that was trained on a public dataset. We use this model to generate X-ray images for Pulmonary Alveolar Proteinosis, a rare lung disorder. These synthesized images served as augmented data to train a lung disease image classification model, resulting in a performance uptick of 3.7%.\n\n[1]Few-Shot Diffusion Models\n\n[2]Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation\n\n[3]Zero-shot Medical Image Translation via Frequency-Guided Diffusion Models \n\n[4]PODIA-3D: Domain Adaptation of 3D Generative Model Across Large Domain Gap Using Pose-Preserved Text-to-Image Diffusion\n\n[5]One-shot Unsupervised Domain Adaptation with Personalized Diffusion Models\n\n[6]RoentGen: Vision-Language Foundation Model for Chest X-ray Generation\n\n[7]Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains\n\n### **Presentation Improvement**\n1. **Concrete Examples and Illustrations**\n \nRecognizing the importance of these details, we will relocate this content from the appendix to the relevant section in the paper. And we also create a new PDF dedicated to additional case studies.\n\n2.**Font Sizes of Tables**\n\n We will promptly correct this issue to ensure consistency in the font sizes across all tables.\n\n### **Questions**\n\n**Clarification on \"Shot\" and 5% Subsampling:** \n\nYes, the number of \"Shot\" does refer to how many real data points from each class are available. line 165-166 “keep the sub-sampled set within 5% of the number of the training dataset.” We sample various datasets corresponding to different shots, but meticulously ensure that the total quantity of each dataset stays within a 5% limit of the training dataset, thereby crafting an authentic simulation of data scarcity.\n\n We hope that these clarifications address your concerns and we kindly request reconsideration of the rating after we implement the proposed revisions."}, {"review_id": "nvNVOh9Hxd", "reviewer": "Reviewer_QqX9", "summary": "The paper introduced a new data augmentation technique, called back-modality. The augmentation is based on modal transformations. Specifically, instances in the original modality (e.g., image) are transformed to an intermediate modality (e.g., text), augmented in the intermediate modality and then transformed back to the original modality. Experimental results in a few-shot setting on three benchmark datasets show that the proposed augmentation technique produces better results than the base model and also than other existing augmentation techniques.  ", "questions": "To minimize the low-quality back-captioning augmentations, the authors “explicitly inject the image labels into the text prompts, which leads to the generation of descriptions  that incorporate these finer-grained labels. “ It is not clear if the label information is used during testing, especially when the task is to predict the image label. Please elaborate. \n\nSome of the choices made are rather arbitrary, e.g. why 5 augmentations per instance? Or 5% of the data used to balance the training data? Shouldn’t the size of the training data be given by the number  of shots per class? \n", "limitations": "The authors emphasize some limitations related to the size of the cross-modal models. Other limitations related to generalizability could be discussed. \n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed augmentation technique leverages recent advancements in transformations between different modalities. \n\nThe strategy is rather general and can be applied to a variety of applications for which cross-modal transformations can be obtained. It does not require access to model weights or fine-tuning, and can thus be seen as \"a variant of the Cross-Modal-Models-as-a-Service (CMMaaS) application\".\n\nExperimental results clearly suggest the advantage of the proposed technique as compared to existing augmentation approaches on the three datasets used. \n\n", "weaknesses": "While the authors aim to produce a general augmentation strategy based on cross-modal transformations, their actual process involves some decision specific to the modalities considered in the study, meant to minimize the production of low-quality augmentations (and no augmentations are used for images as an intermediate modality). This can make the strategy hard to apply without extensive analysis of the augmentations. \n\nSome of the choices made are rather arbitrary, e.g. why 5 augmentations per instance? Or 5% of the data used to balance the training data? Shouldn’t the size of the training data be given by the number  of shots per class? \n\nThe discussion on diversity and affinity is very brief in the main paper. Some details are provided in the appendix. It is not clear why they were not included in the paper (the current paper has only 8 pages). Same for the details of the case study regarding the back-imagination and back-speech augmentations. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## \n\nWe thank Reviewer QqX9 for the comprehensive and insightful review of our paper. Below, we address the specific weaknesses and questions raised in the review:\n\n### **Weaknesses:**\n\n**1. Decision Specific to Modalities:**\nWhile we understand the concern regarding the decisions specific to the modalities considered in the study, we would like to emphasize that the steps taken are important to ensure the quality of the augmented data. We include comprehensive guidelines in the manuscript, detailing how to adapt the method to various modalities. This inclusion facilitates the understanding and reproduction of our method and shows that a certain level of customization is often essential to adapt any machine learning method to specific applications. However, this does not impede the generalization of the method. \n\nAs stated in the paper (lines 159-162), one reason for this decision is that images produced via certain augmentation techniques, such as random erasing and cutout, often present a substantial challenge to image captioning models (Specifically, OFA models). On the other hand, when human evaluation is performed on augmented samples, the combination of multi-imagination and multi-captioning appears to be sufficient to yield satisfactory results. Moreover, if there are more robust image captioning models in the future, the image augmentation method can indeed be used as part of our framework.\n\n**2. Arbitrary Choices in Augmentation Parameters:**\n\nThe selection of these parameters was driven by our intention to provide a proof-of-concept that the proposed method is effective without overwhelming resource consumption. It's a deliberate trade-off:\n\n- **Too Small Augmentation Size**: If the augmentation size is too small, the effect of our back-modality technique may not be discernible. This could lead to a situation where the method's potential is underestimated or overlooked entirely.\n- **Too Large Augmentation Size**: Conversely, an excessively large augmentation size would consume more computational resources. While it might yield better results, it could also make the technique less accessible to researchers with limited resources, defeating the purpose of demonstrating a widely applicable method.\n\nOur choice of 5 augmentations per instance was therefore a calculated decision. We aimed to find a middle ground where the method's effectiveness could be demonstrated without significantly burdening computational resources. We were not pursuing the absolute best results but rather a pragmatic balance that underscores the method's viability and potential for various applications.\n\nWe will include these justifications in the revised manuscript, ensuring that readers understand the rationale behind these choices. Thank you again for pointing out this area for clarification, and we believe this amendment will strengthen the overall contribution of our paper.\n\n**3. Lack of Detail on Diversity, Affinity, and Case Studies:**\n\nWe appreciate the feedback on the brevity of certain sections in the main paper.  We agree that a more comprehensive discussion of diversity, affinity, and case studies in the main text would enhance understanding. We will revise the manuscript to include a more comprehensive discussion within the main body, as well as more details on back-imagination and back-speech augmentations.\n\n### **Questions:**\n\n**1. label information:**\n\nI would like to clarify the methodology regarding injecting image labels into text prompts, as it seems to have caused some confusion. The process of injecting these labels was employed strictly during the augmentation phase of the training dataset, and not during testing. Specifically, the label information was only used during training to guide the generation of descriptions for the data augmentation. We only augment the training dataset, ensuring that the original, unaltered testing set is used for evaluation. This method was carefully designed to ensure the augmentations were aligned with the correct classes, thereby enriching the training data with meaningful variations.\n\nDuring the testing phase, no such label information is injected. This was a deliberate choice to maintain a fair and unbiased evaluation of the model's performance on unseen data. It ensures that the experimental results are a genuine reflection of the model's ability to predict labels without any additional guidance.\n\n**2. why 5 augmentations per instance？**\n\nAddressed Above.\n\n**Or 5% of the data used to balance the training data?Shouldn’t the size of the training data be given by the number of shots per class?**\n\nIn response to your inquiry, the concept of \"shot\" in our work indeed refers to how many actual data points from each class are available for training. This is central to the notion of few-shot learning, where the emphasis is on learning from a limited number of examples.In our experiments, we sample various datasets corresponding to different shots but meticulously ensure that the total quantity of each dataset stays within(≤) a 5% limit of the training dataset.\n\nRegarding your specific question about the 5% limitation, as stated in lines 165-166 of our manuscript, we \"keep the sub-sampled set within 5% of the number of the training dataset.\" This approach was carefully devised to reflect real-world scenarios where data scarcity is a prevalent challenge.\n\nWe hope that these clarifications address your concerns. We believe that with these details, the validity of our experiments and conclusions is evident. We greatly appreciate your feedback and are looking forward to your further comments."}, {"review_id": "HCTRsQDSPn", "reviewer": "Reviewer_44qx", "summary": "The paper introduces a novel approach to data augmentation that leverages recent advances in generative models. It presents a general framework and experiments with three settings for image, text, and audio tasks, showing promising results for their Back-Modality approach. \n", "questions": "There were two plots that I missed in this paper that would have made it stronger and more insightful. The first is a plot with less sub-sampling of the original data. Maybe with 5, 10, 20, 40, 60, 80, 100% of the training data + the generated instances. Would the proposed method still be useful if I have a lot of data? Or just if I don't have data at all? \n\nThe second plot would be to go further on the generation. Experiments go all the way to 10-shot, but what if we could generate 100, 1000? Is 10 the limit for this approach? \n\nDuring the detailed strategies, you share some tricks that made your model work better, like adding the image label to the prompt. How much of an issue to the final results were the problems you addressed in that section? Could it make it harder to expand the technique to other settings? It would have been interesting to add the original results to the appendix so we know how many problems that was causing.", "limitations": "Yes, the authors have a limitations section.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The idea is simple but interesting and effective based on the presented experiments.\n- The method is generic and can be extended to other modalities, and it can also incorporate other models as the field evolves. \n- Case and ablation studies show the benefits of using the method and support most of the claims in the paper. Authors also compute statistical significance, which is an important step to validate results.", "weaknesses": "- Experiments are only done with one small model for each task. It would be interesting to see the gains from increasing the model size and if the benefits would increase or diminish.\n- The setup explanation is a bit confusing. For example, when explaining the data-scarce scenarios, it seems like n-shot is used for the number of generated samples per class during training. Still, there is no information about the training set size after sub-sampling or the label distribution on the undersampled training set. It would be better to state the setup clearly, e.g., we have X instances in the training set, with Y classes, and we generate Z samples per class. While most of it can be inferred by reading the appendix and doing some math, the paper would be better presented with more clear numbers.\n- The paper could have more experiments testing the limits of how much data could be generated by these methods that would still yield good improvements. Also, a high resource scenario experiment would be useful.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 44qx,\n\nWe would like to thank you for your thoughtful review and constructive feedback on our paper. We appreciate your recognition of the contributions and strengths of our work. Here, we address the weaknesses and questions you have raised.\n### **Weaknesses:**\n\n1. **Experiments with One Small Model for Each Task:** We agree that experimenting with different model sizes could provide valuable insights. To mitigate this concern, we performed additional experiments with varying model sizes and include them in the final version of the paper. For instance, in image classification tasks using Resnet50, we observed a relative improvement greater than what was observed with a smaller version of Resnet. Similarly, in textual tasks, our experiments with bert-large showcased a relative improvement that exceeded that of bert-base. These findings indicate that in scenarios with sparse data, the data produced by our augmentation technique offers greater benefits to larger models.\n\n2. **Setup Explanation and Clarity in Numbers:**\nI sincerely apologize for any confusion this might have caused.To clarify:\nWhen constructing the sparse dataset, we sample from each category. For instance, when we create a 10-shot dataset, we sample 10 unique data points from each class of the original data and then augment each data point five times.\n\nUsing the 10-shot, back-captioning approach as an example:\n\n- In the original tiny-imagnet dataset, there are 200 classes, with each class containing 500 data points.\n- After sub-sampling, the dataset consists of 200 classes, with 10 data points per class.\n- With a 5-fold augmentation (as stated on lines 155-156, our default augmentation size is set at 5), this dataset still consists of 200 classes, with each class now having 50 data points.\n\nThe term \"shot\" denotes how many samples are drawn from each class during sub-sampling, and the default augmentation multiple is 5.\nWe understand the importance of clear presentation and will strive to state our setups more transparently in future iterations. Once again, I apologize for the oversight, and thank you for bringing it to our attention. We hope that this clarification will help in understanding our method better.\n### **Questions:**\n\n1&2. **Plots for Different Sub-sampling and Generation Rates:**\nWe conducted experiments on our back-captioning method and have showcased the results in Figure 2 of the newly added PDF.\n\nFor the first question, regarding the impact of sub-sampling: Due to computational and time constraints, the highest percentage of the original data used for experiments was 40%. Figure 2, sub-figure (a) illustrates the model's performance variation when increasing the amount of the original data while maintaining the augmentation multiple at 5. From the difference curve in this sub-figure, it is evident that as the volume of original training data increases, augmented data continues to yield benefits. However, the benefit shows a diminishing return as more original data is added.\n\nFor the second question, on extending the generation:\nIn the same Figure 2, sub-figure (b) represents that as the augmentation multiple grows, the model's performance improves. However, after a certain point, the gains tend to plateau. The maximum augmentation multiple tested in our experiments was 20 times.\n\nThus, in summary: (1). Augmented data provides benefit even with an increase in original training data, but this advantage decreases as more original data is used.  (2). Increasing the augmentation multiple improves model performance up to a certain point, after which the improvements plateau. The current experiment went up to 20 times, indicating that there's room for further exploration.\n\n\n3. **Detailed Strategies and Potential Issues:** \n\nFirstly, to gauge the extent to which our strategies affected the model's performance, we conducted a comprehensive human evaluation on the augmented data. The evaluators consisted of five crowdsourced workers, with the final scores being an average of their evaluations. The results were indeed telling:\n\nFor the images generated using the back-captioning method:\n\n- **With** our proposed strategies, the Label Invariance Score reached an impressive 99.2%.\n- **Without** the said strategies, the Label Invariance Score dipped to 84.7%.\n\nSimilarly, for the sentences generated using the back-imagination method:\n\n- **With** our strategies, the Semantic Consistency Score was 98.8%.\n- In the absence of these strategies, the score fell to 94.1%.\n\nUpon closer inspection, the primary reason for the back-captioning method's inability to retain labels was that the image-captioning model failed to accurately describe the key labels for certain images. On the other hand, the back-imagination method's inconsistency in retaining semantics was because the diffusion model generated black and white images at times. This led to the subsequent back-captioning generation of descriptions like \"A black and white photo of.\"\n\nThe crux of the issue here lies in the limitations of the current open-source cross-modal models we utilized. While they've achieved significant success, they are not flawless and occasionally necessitate strategy-assisted filtering to improve their output. It's also worth noting that as the research in this domain rapidly progresses and cross-modal capabilities strengthen, the importance of such strategies will likely diminish.\n\n"}]

Author rebuttal:
Dear Reviewers,

Firstly, I would like to express my profound gratitude for the time and effort you invested in reviewing our manuscript. We are deeply appreciative of the recognition most reviewers gave to our methods and experiments. Your constructive feedback is invaluable.

From the collective feedback, we identified two primary and common concerns:

1. The suggestion that case study examples should be moved from the appendix to the main content.
2. The need for clearer explanations about experimental setup. Using the 10-shot, back-captioning approach as an example:

- In the original tiny-imagnet dataset, there are 200 classes, with each class containing 500 data points.
- After sub-sampling, the dataset consists of 200 classes, with 10 data points per class.
- In our experiments with Back-captioning, where l=n=m=2, (see line 100-102) for every image, we generate 8 (2x2x2) augmentations, and subsequently, we uniformly sample 5 of these as the final augmented data.
- With a 5-fold augmentation (as stated on lines 155-156, our default augmentation size is set at 5), this dataset expands to 200 classes, with each class now having 50 data points.
The term "shot" denotes how many samples are drawn from each class during sub-sampling

We have  carefully considered and responded to the individual queries and points raised by each reviewer. Our detailed responses for each concern are outlined in our responses.

Again, thank you for your constructive insights. We believe that with your feedback, our work has been significantly improved.



Meta-review: Solid paper on data augmentation via modality transformation. This paper could have a significant impact in the community. The authors have responded to comments from the reviewers and will incorporate the feedback in the next version. I recommend acceptance.
Decision: Accept (poster)