SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhi Li
Zhejiang University, China zhili@zju.edu.cn
Yifan Liu
Zhejiang University, China  
yifan.liu@zju.edu.cn
Yin Zhang*
Zhejiang University, China zhangyin98@zju.edu.cn
We introduce Back-Modality, a novel data augmentation schema predicated on modal transformation. Data from an initial modality undergo a transformation to an intermediate modality, followed by a reverse transformation. This framework serves dual roles. On one hand, it operates as a general data augmentation strategy. On the other hand, it allows for other augmentation techniques, suitable for the intermediate modality, to enhance the initial modality. For instance, data augmentation methods applicable to pure text can be employed to augment images, thereby facilitating the cross-modality of data augmentation techniques. To validate the viability and efficacy of our framework, we proffer three instantiations of Back-Modality: back-captioning, back-imagination, and back-speech. Comprehensive evaluations across tasks such as image classification, sentiment classification, and textual entailment demonstrate that our methods consistently enhance performance under data-scarce circumstances.
Neural network-based deep learning models are often prone to overfitting, resulting in a loss of generalization capability due to the limited size of training data. To mitigate this issue, data augmentation techniques are routinely employed to generate an augmented pool of training samples. The recent past has seen significant strides in the application of data augmentation within diverse fields such as speech [Ko et al., 2015, Park et al., 2019], computer vision [Simard et al., 1996, Shorten and Khoshgoftaar, 2019, Szegedy et al., 2014], and natural language processing [Sennrich et al., 2016, Wei and Zou, 2019]. Nevertheless, most of these data augmentation algorithms are tailored for a specific modality.
Concomitant with the swift advancement of cross-modal methodologies, various dual cross-modal models, such as text-to-image generation and image captioning, have demonstrated impressive performance. This progress catalyzes our development of a novel paradigm for data augmentation: Back-Modality, a comprehensive data augmentation framework predicated on modal transformation. Data in the initial modality are transformed to an intermediate modality and subsequently reversed back. Theoretically, the initial and intermediate modalities can represent any modality, including but not limited to text, image, audio, and video. Our framework retains its universality, contingent on the availability of corresponding dual cross-modal models to execute the modal transformation. Leveraging pretrained cross-modal models, our method is capable of generating a wide array of high-quality and diverse data. A comparison illustrating the breadth and quality of the generated data is provided in Figure 1.
Simultaneously, our framework enables the augmentation method intended for the intermediate modality to benefit the initial modality. For instance, data augmentation methods designed for pure
*Corresponding author: Yin Zhang.
golfish
schoo
Ground truth
Figure 1: Comparative evaluation of images generated by our data augmentation method and other approaches. Capitalizing on the inherent capability of pretrained cross-modal models, our method excels in generating a broad range of high-quality and diverse images.
RandomErasing
Mixups
AutoAugment
text can be leveraged to augment images, fostering cross-modality of data augmentation techniques. In essence, our framework facilitates partial decoupling of data augmentation methods from specific modalities. Moreover, our framework broadens the application landscape of cross-modal models. Given that our methodology obviates the need to access model weights or undergo further fine-tuning, it can be seen as an application variant of Cross-Modal-Models-as-a-Service (CMMaaS).
To substantiate the feasibility and efficacy of our framework, we present three instantiations of Back-Modality: back-captioning, back-imagination, and back-speech. For back-captioning, the initial modality is an image, and we utilize an image-caption model to generate captions for each image. Subsequently, data augmentation methods designed for text are employed to augment these captions. Finally, a text-to-image model is invoked to generate variant images, which are then used as augmented samples. For back-imagination and back-speech, the initial modality is text, and the intermediate modality is image and speech, respectively.
Systematic evaluations across various tasks including image classification, sentiment classification, and textual entailment demonstrate that our methods can consistently improve performance, particularly under data-scarce conditions. Upon further analysis and comprehensive case studies, it has been observed that the data generated via our method typically exhibit increased diversity.
Our principal contributions and discoveries can be summarized as follows:
- We introduce Back-Modality, a modality-agnostic data augmentation framework predicated on modal transformation. Within this framework, data in the initial modality undergo transformation to an intermediate modality and subsequently reverse back.
- Our framework enables the cross-modality of data augmentation methods. For instance, data augmentation techniques developed for pure text can be deployed to augment images.
- Our approach extends the application realms of cross-modal models. Our methodology eliminates the need to access model weights or conduct further fine-tuning, hence it can be perceived as a variant of the Cross-Modal-Models-as-a-Service (CMMaaS) application.
- Experiments on a variety of tasks and datasets substantiate that our methods can consistently enhance performance, particularly in data-scarce scenarios. Further analysis reveals that data generated by our method typically exhibits a higher degree of diversity.
2.1 Framework
Back-Modality presents a comprehensive, modality-agnostic data augmentation framework predicated on modal transformation. As shown in Figure 2,  $\mathcal{A}$  symbolizes the initial modality, and  $\mathcal{B}$  signifies the
Figure 2: Back-Modality constitutes a universal data augmentation framework premised on modal transformation.  $\mathcal{A}$  represents the initial modality, and  $\mathcal{B}$  denotes the intermediate modality. The framework integrates two types of dual cross-modal tasks:  $F:\mathcal{A}\mapsto \mathcal{B}$  and  $G:\mathcal{B}\mapsto \mathcal{A}$ . H signifies the data augmentation methods applicable to the intermediate modality  $\mathcal{B}$ .
intermediate modality. The framework incorporates two types of dual cross-modal tasks:  $F: \mathcal{A} \mapsto \mathcal{B}$  and  $G: \mathcal{B} \mapsto \mathcal{A}$ . Here,  $H$  denotes the augmentation methods applied to modality  $\mathcal{B}$ .
Let  $X$  denote a sampled data point in the initial modality  $\mathcal{A}$ , and  $X_{\text{aug}}$  represents the augmented samples of  $X$ . Our framework can be succinctly expressed as:
$$
X _ {a u g} = G (H (F (X))) \tag {1}
$$
Theoretically,  $\mathcal{A}$  and  $\mathcal{B}$  can represent any modality, such as text, image, audio, video, and so forth. Moreover,  $\mathcal{A}$  and  $\mathcal{B}$  can also be the same modality. For instance, back-translation augmentation methods can be perceived as a special case of our framework, where  $\mathcal{A}$  and  $\mathcal{B}$  are both text in different languages, and  $H$  is omitted. Essentially, our framework retains its universality, contingent upon the availability of corresponding dual models.
Within our framework,  $H$  can assist in augmenting data in modality  $\mathcal{A}$  by augmenting data in the intermediate modality  $\mathcal{B}$ . This implies that the augmentation method suitable for a specific modality can be transferred to other modalities via our framework. More formally, let us consider a scenario where  $F$  and  $G$  are fixed, with  $Y_{F,G}$  denoting the corresponding framework. Here,  $H$  is treated as a variable. Consequently, the augmented samples  $X_{aug}$  can be viewed as a function of  $X$  and  $H$ .
$$
X _ {a u g} = Y _ {F, G} (H, X) \tag {2}
$$
Given a specific  $X$ , varying  $H$  can yield a diverse set of augmented samples  $X_{aug}$ . In essence, our framework facilitates partial decoupling of data augmentation methods from specific modalities.
2.2 Instantiations
In order to demonstrate the feasibility and efficacy of Back-Modality, we introduce three instantiations of this framework: back-captioning, back-imagination, and back-speech. These methods encompass three prevalent modalities: text, image, and speech.
Back-captioning In the proposed method, we define  $\mathcal{A}$  as image modality and  $\mathcal{B}$  as text modality. Here,  $F$  represents the task of image captioning, while  $G$  represents the task of generating images from text.  $H$  corresponds to the process of textual data augmentation.
Back-captioning, as a concept, can be deconstructed into a trifecta of augmentation methodologies: multi-captioning, caption augmentation, and multi-imagination. It is observed that various observers usually generate captions for the same image that, while being semantically analogous, exhibit subtle variances in detail. This phenomenon mirrors the capability of most image captioning models to generate multiple unique captions for a single image, a process we refer to as multi-captioning. Given a caption sentence, several text augmentation techniques can be employed to produce multiple related sentences, a process we term as caption augmentation. For a given sentence, the images that different individuals visualize generally share similarities, yet present unique details. We refer to this phenomenon as multi-imagination. A diffusion-based model, employing various random seeds, can effectively simulate this process. Given an image captioning model that generates  $l$  captions for each image, a caption augmentation process that generates  $m$  sentences for each caption, and a text-to-image model that generates  $n$  images for each sentence, the maximum total number of generated augmented sentences per original sentence amounts to  $l * m * n$ . This obviously represents
a highly efficient data augmentation approach. In practical implementation, we conduct uniform sampling at random on these augmented data to obtain the final augmentation dataset.
Back-speech In this proposed approach, we designate modality  $\mathcal{A}$  as text, while  $\mathcal{B}$  pertains to the audio modality. Here,  $F$  corresponds to the task of text-to-speech generation, and  $G$  denotes the automatic speech recognition task. The text-to-speech model is designed to generate one audio file for each input sentence. Following this, audio data augmentation techniques, such as pitch shifting, denoted by  $H$ , are applied to these audio files to produce augmented sound files. Subsequently, the automatic speech recognition model translates these augmented sound files back into text, thereby generating augmented sentences.
The notion of back-speech can be elucidated by the following observation: the sentence articulated by an intermediary often bears slight deviations from the original sentence. This is due to the fact that different individuals may have distinct pronunciations for the same sentence, influenced by factors such as speech rate, pitch, or regional accents. To simulate these variations, we employ augmentation techniques within the audio modality. As a consequence, in the resultant output, some words may be substituted by their homophones. Given that most homonyms bear morphological similarities, with only minor differences at the subword level, this transformation can be viewed as a form of subword regularization facilitated by the audio modality.
Back-imagination Back-imagination can be construed as the inverse operation of back-captioning. In this approach, we assign modality  $\mathcal{A}$  to text, while  $\mathcal{B}$  corresponds to the image modality. Here,  $F$  represents the task of generating images from text, and  $G$  signifies the image captioning task. Similarly, back-imagination amalgamates three types of augmentation techniques: multi-captioning, image augmentation, and multi-imagination.
Method	Back-captioning	Back-speech	Back-imagination
F	OFA	fast-speech2	stable-diffusion v2
G	stable-diffusion v2	wav2vec2	OFA
H	augmentation with GPT	pitch shifting/time stretching	-
Task	image classification	sentiment classification	textual entailment
Dataset	Tiny ImageNet	SST-2	TNCC
Table 1: Statistics of configurations. "augmentation with GPT" means that we employ GPT model to augment captions. The symbol "-" indicates that we do not apply image augmentation in the process of back-imagination. Tiny ImageNet [Le and Yang, 2015] is an image classification dataset, SST-2 [Socher et al., 2013] is employed for sentiment classification, and TNCC, a textual entailment dataset, is based on Crisscrossed Captions [Parekh et al., 2020]. For further details, please refer to the appendix.
In this section, we initially delineate the tasks, datasets, and models employed in our research. Subsequently, we assess the efficacy of back-captioning, back-speech and back-imagination in data-scarce scenarios, spanning a diverse range of tasks and datasets. Finally, we investigate the intrinsic mechanisms underlying the effectiveness of our proposed method from three perspectives: (1) conducting a quantitative analysis on the diversity and affinity of the generated samples, (2) performing ablation experiments to evaluate the contribution of individual components within the method, and (3) scrutinizing the differences between the original and augmented samples at an instance level and providing an in-depth analysis of the transformation effectuated by the augmentation process.
3.1 Baseline and Comparison
We use the performance of the base model, devoid of any data augmentation, as a baseline to evaluate the effectiveness of our augmentation method in enhancing model performance. For the image classification task, we employ the Resnet-18 model [He et al., 2015]. We use BERT [Devlin et al.,
2018] for sentiment classification and textual entailment tasks. To further substantiate the efficacy of our methods, we compare them against several other data augmentation strategies. For the image classification task, we consider four renowned methods: Random Erasing [Zhong et al., 2020], Autoaugment [Cubuk et al., 2019], Alignmixup [Venkataramanan et al., 2022], and Puzzle Mix [Kim et al., 2020]. For the sentiment classification and textual entailment tasks, we opt for the most popular task-agnostic data augmentation methods for comparison, as per Longpre et al. [2020], Yoo et al. [2021], including EDA [Wei and Zou, 2019] and Back-translation [Sennrich et al., 2016, Fadaee et al., 2017, Edunov et al., 2018, Ng et al., 2019]. We employ the large pre-trained EN-DE and DE-EN translation models [Ng et al., 2019, Yoo et al., 2021] for back-translation, which have a size order of magnitude comparable to the cross-model models used by Back-Modality methods, making them a robust baseline. In addition, we also consider some task-specific data augmentation methods for reference, including TMix [Chen et al., 2020], SSMix [Yoon et al., 2021], and Treemix [Zhang et al., 2022].
3.2 Datasets and Configurations
Datasets Tiny ImageNet [Le and Yang, 2015] and the Stanford Sentiment Treebank-2 (SST-2) [Socher et al., 2013] are widely used datasets. Given the scarcity of datasets for understanding natural language in visual scenes, we introduce a novel textual entailment dataset, named Textual Natural Contextual Classification (TNCC). And detailed descriptions can be found in the appendix.
Data-scarce scenarios To showcase our approach, we conduct experiments on artificially data-scarce tasks by sub-sampling the training set. For image classification, sentiment classification, and textual entailment tasks, we perform a class-balanced sub-sample on the training set and we denote the number of data samples for each class in the downsampled dataset as "shot," before any data augmentation is applied. We adhere to statistical rigor in our experiments by executing the sub-samples to 5 different data seeds. Concurrently, for each sub-sample, we train all models using 5 different random seeds. We report the statistical mean for all results. All experiments were conducted using PyTorch and executed on RTX 6000 GPUs and the Atlas computing cluster.
Configurations Table 1 provides a comprehensive view of the configurations used in our experiments. Unless explicitly stated otherwise, all pretrained models utilized in our research are obtained from the Huggingface Transformers library $^2$  [Wolf et al., 2019]. The code used to generate augmented samples and the TNCC dataset are both accessible $^3$ . The default augmentation size used in our studies, which includes both our method and the comparison method, is set to 5. As an illustration, in our experiments with Back-captioning, when we have  $l = n = m = 2$ , we create 8 augmentations for each image (2x2x2), and then, we randomly select 5 of these augmentations to form the final augmented dataset. For back-captioning, we utilize gpt-3.5-turbo $^4$ , augmenting captions using the following prompt: "Maintain the nouns in the following sentence intact and generate semantically diverse sentences." For the process of back-imagination, we consciously exclude image augmentation in our experiments. One reason is that images produced via certain augmentation techniques, such as random erasing [Zhong et al., 2020] and cutout [DeVries and Taylor, 2017], often present a substantial challenge to image captioning models. On the other hand, the combination of multi-imagination and multi-captioning appears to be sufficient to yield satisfactory results.
Detailed Strategies In our experimental process, we observed that simply constructing a pipeline does not necessarily ensure the quality of the augmented data. Consequently, we have devised several strategies aimed at minimizing the production of low-quality or even inaccurate augmented data.
In the case of back-captioning, while the model possesses the capacity to generate highly relevant image descriptions, when deployed for specific tasks such as image classification, more nuanced categories are often required. However, the model sometimes falls short in providing these nuanced labels within the generated descriptions. For instance, when given an image of a 'Persian cat', the model tends to generate descriptions mentioning 'cat' rather than the more specific 'Persian cat'. Several image captioning models, such as the OFA [Wang et al., 2022] model, have the capability to generate image captions conditional on both a text prompt and an image. To address this issue, we
$^{2}$ https://github.com/huggingface/transformers
<sup>3</sup>https://github.com/zhilizju/Back-Modality
4 https://platform.openai.com/docs/models/gpt-3-5
Method	Shot
1	3	7	10
Base-model	2.40	5.19	8.64	11.75
Random Erasing	2.61	5.77	9.11	12.59
Auto augment	3.41	6.35	9.84	13.23
Alignmixup	4.42	8.19	11.78	14.34
Puzzle Mix	4.48	10.26	12.52	15.66
Back-captioning	10.67	14.55	16.13	20.07
Table 2: The top-1 accuracy metric for back-captioning on the Tiny ImageNet dataset is displayed above. The overall accuracy is low, primarily because the model is trained from scratch using scarce data. "Shot" is used to indicate the quantity of images per label.
Method	Shot
1	2	5	10
Base-model	61.18	72.47	81.30	84.57
EDA	61.04	73.89	83.61	85.71
Back-translation	62.35	74.07	84.30	85.28
TMix	60.11	71.54	82.77	84.60
SSMix	62.32	74.47	82.89	85.32
Treemix	62.88	73.27	85.15	87.41
Back-imagination	69.20	79.62	88.41	89.14
Table 3: The accuracy of the back-imagination method on the TNCC dataset is indicated above. "Shot" signifies the quantity of sentence pairs corresponding to each label.
explicitly inject the image labels into the text prompts, which leads to the generation of descriptions that incorporate these finer-grained labels.
During the back-imagination process, there are occasions when the text-to-image model generates black and white images. This can lead to subsequent image captioning models generating sentences like "A black and white photo of," which are clearly inappropriate for augmentation samples. Consequently, if the back-imagination process yields a black and white image, we directly discard it and proceed with a resampling operation. As for back-speech, if the sentences generated have an edit distance from the original sentence that exceeds  $20\%$  of the entire sentence length, we discard those sentences as well.
3.3 Results
Tables 2, 3, and Table 4, respectively, display the statistical average accuracy values on the Tiny ImageNet, TNCC, and SST-2 datasets. We carried out hypothesis testing on the results across all tasks, with all p-values falling below 0.05. This suggests that our findings are statistically significant. Our back-captioning, back-imagination, and back-speech methods consistently outperform both the base model and other data augmentation methods. This indicates that in data-scarce scenarios, data augmentation based on modal transformation can offer significant performance improvements. In particular, the results of back-imagination demonstrate that our architecture alone, even without the inclusion of additional data augmentation methods (H), is capable of generating effective augmented data.
3.4 Ablation
Table 5 shows the ablation studies of back-captioning, back-imagination and back-speech. The individual components of our framework make significant contributions to the overall performance of the model. In particular, augmentation with GPT, pitch shifting and time stretching all improve the model performance, which strongly supports the notion that our framework effectively accomplishes the cross-modality of data augmentation approaches.
Method	Shot
1	2	3	5	10
Base-model	51.46	52.31	54.42	57.88	61.10
EDA	50.35	52.17	54.87	58.32	61.72
Back-translation	51.30	52.09	55.84	57.69	61.94
TMix	49.71	51.59	54.18	57.62	61.31
SSMix	51.01	52.26	55.32	58.11	61.92
Tremix	51.35	52.18	55.81	58.73	62.37
Back-speech	52.13	52.78	56.11	59.03	63.21
Table 4: The accuracy of the back-speech on the SST-2 dataset is provided above."Shot" refers to the count of sentences associated with each label.
Dataset	Method	Score
Tiny ImageNet	Base-model	11.75
+Back-captioning	20.07
w/o multi-captioning	17.21
w/o augmentation with GPT	18.49
w/o multi-imagination	18.92
TNCC	Base-model	81.30
+Back-imagination	88.41
w/o multi-imagination	84.71
w/o multi-captioning	86.52
STS2	Base-model	57.88
+Back-speech	59.03
w/o pitch shifting	58.45
w/o time stretching	58.60
Table 5: Ablation studies. We report the accuracy of shot 10 for back-captioning, shot 5 for both back-imagination and back-speech. Multi-captioning: generate multiple captions for each image. Multi-imagination: generate multiple images for each sentence. Augmentation with GPT: augment captions with GPT model. Pitch shifting and time stretching are different augmentation methods used for speech.
3.5 Diversity and Affinity
Following the methodology outlined by Gontijo-Lopes et al. [2020], we further analyze our augmentation methods based on two key dimensions: Diversity and Affinity. Diversity is designed to quantify the notion that augmentations can prevent model overfitting by increasing the number of samples in the training set. As a metric of diversity, Gontijo-Lopes et al. [2020] employ the final training loss of a model trained with a given augmentation. The larger the loss function, the better the diversity of the augmented dataset. Affinity, on the other hand, quantifies how augmentation shifts data with respect to the decision boundary of the clean baseline model. Affinity is defined as the difference between the validation accuracy of a model trained on clean data and tested on clean data, and the accuracy of the same model when tested on an augmented validation set. Table 6 indicates that while ensuring comparable affinity metrics, our method consistently presents higher diversity metrics compared to other augmentation methods.
3.6 Case Study