SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1 showcases the images generated by our data augmentation method alongside other approaches. Upon further analysis, we have uncovered advantages in the back-captioning method that many traditional augmentation algorithms lack. As illustrated in Figure 3, we have listed several typical data augmentation patterns found in the back-captioning method:
1. Luminance Modulation: This involves altering the image's brightness by either decreasing or increasing it.
Dataset	Method	Diversity	Affinity
TinyImagenet	Base-model	1.594	0
Random Erasing	1.621	-5.44
Auto augment	1.615	-5.24
Back-captioning	1.723	-6.04
TNCC	Base-model	0.0279	0
EDA	0.0343	-1.14
Back-translation	0.0301	-0.91
Back-imagination	0.0677	-0.82
SST-2	Base-model	0.0089	0
EDA	0.0126	-7.57
Back-translation	0.0104	-6.88
Back-speech	0.0154	-7.93
Table 6: Statistics of diversity and affinity. The former measures the variety of outcomes, with higher values indicating increased diversity and thus considered favorable. The latter quantifies how augmentation shifts data with respect to the decision boundary of the clean baseline model, where a value closer to 0 signifies greater affinity.
Luminance Modulation	Structural Preservation
goldfish	school bus
cash machine	tabby
Attribute Adjustment	Entity Elimination
candle	bathtub
lemon	ice cream
Figure 3: By comparing the original image with the augmented image using back-captioning method, we find typical data augmentation patterns: Luminance Modulation: Adjusting the image brightness either by decreasing or increasing it. Structural Preservation: Maintaining a high similarity of the target structure while altering the background. Attribute Adjustment: Modifying the target's attributes such as size, thickness, and quantity. Entity Elimination: Removing non-target entities.
2. Structural Preservation: This technique focuses on preserving the primary subject's structure while making changes to the background elements.
3. Attribute Adjustment: Adjustments are made to the target's attributes, such as size, thickness, and quantity.
4. Entity Elimination: This technique involves the removal of non-target entities from the image.
We also perform an extensive case study on back-imagination and back-speech. As depicted in Figure 4, we notice that augmented sentences, generated by different modal transformations, display unique characteristics. Further analysis reveals three notable advantages of our methods:
Some augmented cases of back-imagination:
(1) Add/Remove detail
The black dog runs through the water.  $\longrightarrow$  A black dog running through the water in a lake.  
A young man in a black and yellow jacket is gazing at something and smiling.  $\longrightarrow$  A young man in a black and yellow jacket is smiling.
(2) Replace word by synonym/ hypernym/ hyponym
A woman is throwing a frisbee on a beach.  $\longrightarrow$  A woman is playing a frisbee on a beach.
A man is standing in front of a skyscraper. → A man standing in front of a building.
Two children sit on a small seesaw in the sand.  $\longrightarrow$  Two boys sit on a small seesaw in the sand.
(3) Reverse the order
A boy strolls by a pond in a park.  $\longrightarrow$  A boy walks in a park near a pond.
(4) Change /blur the quantity
Three people are walking on a path in a meadow.  $\longrightarrow$  A group of people are walking on a path in a meadow.
(5) Tense variation
Two children sit on a small seesaw in the sand.  $\longrightarrow$  Two children is sitting on a small seesaw in the sand.
Some augmented cases of back-speech:
(1) Split /merge the words
That's far too tragic to merit such superficial treatment.  $\longrightarrow$  That is far too tragic to merit such superficial treatment. Remains utterly satisfied to remain the same throughout.  $\longrightarrow$  Remain utterly satisfied to remain the same throughout
(2). Remove word
He hoped there would be stew for dinner, turnips and carrots.  $\longrightarrow$  He hoped there would be stew for dinner, turnips.
(3) Replace word by homonym
This is a shot of cannery row in 1932.  $\longrightarrow$  This is a shot of cannery row in nineteen thirty-two.
Lend some dignity to a dumb story.  $\rightarrow$  Lend some dignity to a dome story
The year's best and most unpredictable comedy.  $\longrightarrow$  The nears best and most unpredictable comedy.
(4) Change punctuation
So, we have this ability as well.  $\longrightarrow$  So we have this ability as well.
Figure 4: By comparing the original sentence with the augmented sentence, we can summarize several types from the perspective of linguistics. It is obvious that the linguistic transformations of our methods are diverse.
1. Leveraging cross-modal models, our back-imagination augmented sentences demonstrate exceptional fluency, readability, and minimal spelling or grammar errors. Correspondingly, the generated images consistently maintain a high level of quality.
2. Our approach introduces greater diversity in both augmented sentences and images. Notably, augmentation types such as adding details are challenging to achieve using popular text augmentation methods like EDA and back-translation.
3. Our method ensures better semantic consistency. For instance, while both EDA and our method involve deletion operations, our approach selectively removes details. But EDA performs random deletion, which increases the likelihood of altering the original semantics. A thorough analysis of back-speech augmentation reveals that certain words in augmented sentences are substituted with similar-sounding words, often sharing similar but not identical subwords. Additionally, the augmentation process involves splitting and merging certain words. Consequently, the augmentation of back-speech can be considered as a form of regularization akin to techniques such as BPE-Dropout [Provilkov et al., 2020] and SwitchOut [Wang et al., 2018].
4.1 Data Augmentation
Data augmentation is widely utilized in various domains to address limited data scenarios, and it has achieved significant success. In computer vision, basic image manipulations such as translation and rotation can generate new samples [Ronneberger et al., 2015, Krizhevsky et al., 2017]. Techniques like random erasing [Zhong et al., 2020] and cutting [Devries and Taylor, 2017] aid in improving generalizability by occluding images. Pitch shifting and time stretching are popular methods in speech processing [Salamon and Bello, 2016, Moreno-Barea et al., 2018]. Text augmentation methods typically involve word-level text editing operations and sentence-level text generation. For word-level text editing operations, word substitution is commonly employed, including techniques such as
synonym replacement [Zhang et al., 2015], KNN replacement [Wang and Yang, 2015], Uniform and TF-IDF replacement [Xie et al., 2020], Bi-RNN Contextual replacement [Kobayashi, 2018], CBERT [Wu et al., 2019], and LAMBADA [Anaby-Tavor et al., 2019]. EDA [Wei and Zou, 2019] comprehensively explores text editing operations, including synonym replacement, insertion, swap, and deletion, for data augmentation. The popular text augmentation method based on generation is back-translation [Sennrich et al., 2016, Fadaee et al., 2017, Edunov et al., 2018]. It is evident that previous data augmentation methods have been primarily tailored for specific modalities.
4.2 Dual Cross-modal Tasks and Models
In recent years, large-scale pretrained foundation models have achieved remarkable success across a spectrum of tasks. Contemporary research [Zeng et al., 2022, Wu et al., 2023a, Shen et al., 2023] suggests the amalgamation of multiple large pretrained models for the execution of novel downstream multimodal tasks. However, our approach diverges from the aforementioned studies, as we employ dual cross-modal models to devise a fresh data augmentation paradigm. There exists a plenitude of dual cross-modal tasks and pretrained models, such as text-to-image [Ho et al., 2020, Rombach et al., 2022], image captioning [Wang et al., 2022, Li et al., 2022], text-to-video [Singer et al., 2022, Hong et al., 2022], video captioning [Yan et al., 2022, Yamazaki et al., 2022], text-to-speech [Ren et al., 2020], and automatic speech recognition [Baevski et al., 2020]. Dual Learning [He et al., 2016, Xia et al., 2017, 2018] protocols train the models of two dual tasks in tandem, while consciously leveraging the probabilistic correlation between them to regularize the training process. In contrast, our methodology primarily centers on the usage of the large pretrained models of two dual tasks for data augmentation, circumventing additional training.
Compared to existing data augmentation techniques, Back-Modality typically requires additional computational resources and inference time due to its reliance on large pretrained cross-modal models. However, as advancements continue in the cross-modal field and efficient machine learning methodologies are further employed, we anticipate the development of more efficient techniques. As such, this limitation is expected to be progressively alleviated. Furthermore, while our framework is inherently versatile, specific instances may require the design of corresponding strategies akin to those discussed in Section 3.2, which ensure label invariance in the generated samples and maximize their affinity with the original data.
In this work, we introduce a novel data augmentation framework, predicated on modal transformation, and confirm its feasibility and effectiveness. We posit that as the field of cross-modal learning continues to evolve, modal transformation can present a fresh avenue for developing data augmentation techniques. This emerging field presents a plethora of intriguing research questions that merit further exploration. For instance, a comprehensive evaluation of our method across a broader spectrum of tasks and modalities would be insightful. In addition, investigating the impact of fine-tuning the cross-modal models on domain-specific datasets on the quality of the generated data could yield meaningful insights.
This work was supported by the NSFC project (No. 62072399), Zhejiang Provincial Natural Science Foundation of China under Grant No. LZ23F020009, China Research Centre on Data and Knowledge for Engineering Sciences and Technology, MoE Engineering Research Center of Digital Library, and the Fundamental Research Funds for the Central Universities. Thanks to the Hangzhou AI Computing Center for providing the Atlas computing cluster. We also express our sincere gratitude to anonymous reviewers for their invaluable feedback and constructive comments, which significantly contributed to the improvement of this paper.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023b.
Yunxiang Li, Hua-Chieh Shao, Xiao Liang, Liyuan Chen, Ruiqi Li, Steve Jiang, Jing Wang, and You Zhang. Zero-shot medical image translation via frequency-guided diffusion models. arXiv preprint arXiv:2304.02742, 2023.
Gwanghyun Kim, Ji Ha Jang, and Se Young Chun. Podia-3d: Domain adaptation of 3d generative model across large domain gap using pose-preserved text-to-image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22603-22612, 2023.
Yasser Benigmim, Subhankar Roy, Slim Essid, Vicky Kalogeiton, and Stéphane Lathuilière. One-shot unsupervised domain adaptation with personalized diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 698-708, 2023.
7 Back-imagination and Back-speech
Figure 5: The illustrative examples for two proposed techniques: Back-imagination and Back-speech.
As shown in Figure 5, we present illustrative examples to facilitate a better understanding of two proposed techniques: Back-imagination and Back-speech.
8 Datasets
Tiny ImageNet [Le and Yang, 2015] serves as a compact version of the comprehensive ImageNet dataset. It comprises 100,000 images spanning 200 classes, with 500 images per class, and these images are downsized to  $64 \times 64$  pixels. Each class is furnished with 500 training images, 50 validation images, and 50 test images.
The Stanford Sentiment Treebank-2 (SST-2) [Socher et al., 2013] is a sentiment classification dataset populated with movie reviews gathered from Rotten Tomatoes, paired with their corresponding binary labels. The dataset is partitioned into training, validation, and testing sets, comprising 67,349, 872, and 1,821 instances, respectively.
Given the scarcity of datasets for understanding natural language in visual scenes, we introduce a novel textual entailment dataset, named Textual Natural Contextual Classification (TNCC). This dataset is formulated on the foundation of Crisscrossed Captions [Parekh et al., 2020], an image captioning dataset supplied with human-rated semantic similarity ratings on a continuous scale from 0 to 5. We tailor the dataset to suit a binary classification task. Specifically, sentence pairs with annotation scores exceeding 4 are categorized as positive (entailment), whereas pairs with scores less than 1 are marked as negative (non-entailment). The TNCC dataset is partitioned into training, validation, and testing sets, containing 3,600, 1,200, and 1,560 instances, respectively. This dataset will be made available alongside our source codes.
9 Configurations
In this work, we employ a uniform experimental configuration for both textual entailment and sentiment classification tasks. We adopt BERT-BASE [Devlin et al., 2018], a model pretrained using Masked Language Modeling (MLM), as our primary experiment subject. For each individual downstream classification experiment, the classification model is initialized with the pretrained parameters from the BERT-BASE model. The classifier component, comprising of two fully connected layers that deduce class labels from the output embeddings generated by the transformer architectures, is randomly initialized. During the training phase, we leverage the Adam optimization algorithm with a learning rate set at  $5e - 5$ , the first and second momentum terms,  $\beta_{1}$  and  $\beta_{2}$ , are respectively set to 0.9 and 0.999. Additionally, we introduce an  $L_{2}$  weight decay of 0.01 to the model. We select a batch
size of 2 for all trials. We save model checkpoints during training and ultimately employ the best checkpoint—determined based on performance on the validation dataset—for testing. The results are presented as classification accuracies on both datasets under investigation.
For the image classification task, we employ the ResNet18 [He et al., 2015] model, which is considered more suitable for small datasets. We initialize all learnable layer parameters randomly. During the training process, we employ the SGD optimizer with a learning rate of 0.1, momentum of 0.9, and a weight decay of 0.0001.
10 Human Evaluation on Augmented Samples
In response to your suggestion, we conducted a human evaluation on the sampled augmented data. The results of the evaluation are as follows:
For the images generated using the back-captioning method:
- Label Invariance Score:  $99.2\%$
For the sentences generated using the back- imagination method:
- Semantic Consistency Score:  $98.8\%$
These high scores indicate that both methods performed exceptionally well in their respective evaluations. The results affirm that Back-Modality preserves the essential characteristics of the original data while introducing diversity, further validating our approach.
11 More Choices of Cross-Modal Generation Models
In our paper, for the Back-captioning with a 10-shot setting, we primarily used the OFA-large model, which yielded a top-1 accuracy of  $20.07\%$ . To assess the impact of different model sizes on the outcomes, we also conducted experiments with OFA-huge under the same conditions. The results showed a significant improvement, with the top-1 accuracy reaching  $22.12\%$ .
12 Cost of Obtaining the Augmented Samples
Method	Additional Computational Time
RandErasing	4 m 55 s
Puzzle Mix	1 h 29 m 25 s
Alignmixup	1 h 59 m 45 s
Back-captioning (our method)	11 h 35 m
Auto augment	About 49 h
Table 7: The additional computational overhead of various augmentation methods compared to the base model.
Table7 illustrates the additional computational overhead of various augmentation methods compared to the base model on RTX A6000. The primary cost of our method is related to generating images with the diffusion model, and the primary overhead of auto augment is associated with learning augmentation policies. In terms of text augmentation, on the textual entailment task, our method, back-imagination, took 4 hours  $13\mathrm{m}45\mathrm{s}$ , while the back-translation method took  $5\mathrm{h}38\mathrm{m}12\mathrm{s}$ . On the sentiment analysis task, back-speech took  $35\mathrm{m}27\mathrm{s}$ , whereas the back-translation method took  $5\mathrm{h}22\mathrm{m}4\mathrm{s}$ .
13 Discussion About Real-World Applicability
While current multi-modal and cross-modal models have achieved impressive results and continue to rapidly advance, it is worth noting that not all domains currently have easy access to open-source
cross-modal models. This limitation can, to some extent, restrict the effectiveness of our method in real-world applications. However, recent research has increasingly focused on the adaptability of diffusion-based cross-modal models in domains with limited data. This research encompasses areas such as few-shot [Giannone et al., 2022], one-shot [Wu et al., 2023b], zero-shot [Li et al., 2023], domain adaptation [Kim et al., 2023], and unsupervised domain adaptation [Benigmim et al., 2023]. These research directions will further expand the real-world applicability boundaries of our method.
Figure 6: Effect of increasing original data.
Figure 7: Effect of increasing augmentation multiple.
14 Proportion and Size
We conducted an analysis of the proportion of augmented data and the augmentation size by sampling a subset of the data. Figure 6 illustrates the variation in the model's performance when increasing the volume of original data while keeping the augmentation multiple at 5. From the curve in this figure, it is evident that as the volume of original training data increases, augmented data continues to provide benefits. However, these benefits exhibit diminishing returns as more original data is added. Figure 7 addresses the aspect of extending data generation. It demonstrates that as the augmentation size increases, the model's performance improves. However, after a certain point, the gains tend to plateau. We believe the primary reason for the phenomena observed in these two figures is that the diversity introduced by augmented data leads to performance gains. However, this diversity may not match the affinity of the original data, which can result in diminishing returns.