Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selection (2024)

Abstract: Large language models (LLMs) are very proficient text generators. We leverage this capability of LLMs to generate task-specific data via zero-shot prompting and promote crosslingual transfer for low-resource target languages. Given task-specific data in a source language and a teacher model trained on this data, we propose using this teacher to label LLM generations and employ a set of simple data selection strategies that use the teacher’s label probabilities. Our data selection strategies help us identify a representative subset of diverse generations that help boost zero-shot accuracies while being efficient, in comparison to using all the LLM generations (without any subset selection). We also highlight other important design choices that affect cross-lingual performance such as the use of translations of source data and what labels are best to use for the LLM generations. We observe significant performance gains across sentiment analysis and natural language inference tasks (of up to a maximum of 7.13 absolute points and 1.5 absolute points on average) across a number of target languages (Hindi, Marathi, Urdu, Swahili) and domains.1

heading: 1 Introduction
text: Multilingual pretrained models are a mainstay in modern NLP. To create highly-performant taskspecific models across different languages, a commonly adopted paradigm is to finetune a multilingual pretrained model like XLM-R (Conneau et al., 2020) using task-specific labeled data. In the absence of labeled data for a target language, pretrained models finetuned on task-specific data in a source language (such as English) have been shown to facilitate zero-shot crosslingual transfer (Yu and Joty, 2021; Zheng et al., 2021; Liu et al., 2021).
*These authors contributed equally to this work. 1The code and data for this work is available at
https://github.com/LLM-Based-Augmentations
Given large language models (LLMs) and their superior generation capabilities, a natural question is whether they can be used to generate synthetic task-specific data in English. To create synthetic data in a (non-English) target language, these LLM generations could be further translated into target language data using existing machine translation systems. In this work, we examine the following central question: How do we make best use of LLM generations to improve zero-shot2 crosslingual transfer to target languages without any labeled data? We stress here that we are interested in the realistic setting where task-specific data in the source language might vary in domain from the target language tasks; this setting is largely absent in zero-shot evaluations in prior work.
Our overall data augmentation pipeline is illustrated in Figure 1. We use an open-source LLM such as Llama-2 (Touvron et al., 2023) and prompt it to generate task-specific text in English. For all target languages, we assume access to task-specific data in English that may not be in the same domain as the target-language tasks. When domain information is available for the target tasks, we add this information in the prompt to generate text that appears to be more in-domain.
2Here, by zero-shot we mean that we have no access to in-domain labeled data for a given target language. Automatic translations of English into the target language, derived from an NMT system as shown in Figure 1, can be used during training.
13406
Pseudolabels for synthetic data. We examine two choices to generate pseudolabels for the LLM generations: 1. Via prompting the LLM, to appear in the output as part of the generation. 2. Via a teacher model trained on task-specific English data.
We explore two ways in which the abovementioned pseudolabels can be used when training models with synthetic data. We train a single model on task-specific English data augmented with generations pseudolabeled via LLM prompts. We also adopt teacher-student training where the teacher is trained on task-specific English data and the student model is trained on synthetic data with soft teacher labels (i.e. label distributions). Given label noise, we find training a student model with soft teacher labels yields significantly more accurate models compared to training with hard teacher labels.
Data selection. There are two main arguments in favour of data selection:
1. Efficiency: A smaller subset of the generated data can be used as augmentation, thus reducing the training cost compared to using all the generated data.
2. Accuracy: Data selection helps identify training instances that are likely to aid learning more, and hence generalize better to yield overall performance improvements on downstream tasks.
We explore different selection strategies and show that careful data selection yields stable performance improvements, unlike random selections that lead to higher variance runs and do not guarantee gains in performance. While generating synthetic data has been studied in prior work, in this work we investigate using filtering with LLMbased augmentations in a zero-shot setting.
We note here that the role of the teacher model is critical for data selection. The teacher model gives label probabilities for every LLM generation, that is used in our data selection techniques. (With the pseudolabels derived via LLM prompts, we do not have such confidence estimates.) The utility of the teacher is mainly in identifying a suitable subset based on label probabilities. Once we identify such a subset, either LLM labels or teacher labels can be used for the instances in the subset.

heading: 2 Methodology

heading: 2.1 Generation
text: Consider a scenario where we have task-specific labeled data for a high-resource source language such as English (denoted as Den). Our final downstream task is in a low-resource target language for which we have no labeled data. In this work, we experiment with two classification tasks: sentiment analysis (SA) and natural language inference (NLI). We aim to achieve improved cross-lingual transfer for these two tasks to different target languages by augmenting Den with (labeled) LLM generations denoted as Gen. This is motivated by recent work on boosting task performance via data augmentation techniques (Vu et al., 2022; He et al., 2022a; Liu et al., 2022; Whitehouse et al., 2023; De Raedt et al., 2023a). Gen is generated by prompting an LLM with a compact target domain description and the intended class label to produce class-conditioned, task-specific generations in the target domain. We utilize the open-source 13b llama-2-chat-hf model (Touvron et al., 2023) for all our generations. The prompt is composed of two sub-prompts: 1) A system prompt that specifies a generic set of rules that the generator should obey, and 2) an instruction prompt that specifies more targeted instructions for generation. More details about data generation using llama-2 and the prompts for all target tasks are specified in Appendix D and Appendix E.

heading: 2.2 Pseudolabeling and Training Methods
text: Teacher-student Training (T ). A teacher model is trained on the source data (Den) using crossentropy loss. The teacher is used to pseudolabel the generations in Gen. A subset of Gen is chosen via various selection techniques described in Section 2.3. We will refer to this subset as D′en. A student model is trained on both Den and D′en combined, using cross-entropy loss with the gold labels in Den and a KL-divergence loss with the soft pseudolabels derived from the teacher model. Equation (1) refers to the overall loss computed, where yc(x) is the one-hot label corresponding to each x ∈ Den, qc is the student model probability for class c (with temperature 1), pc(x) is the teacher model probability for each x ∈ D′en for class c ∈ C and q∗c is the student model probability for class c (scaled by temperature value 1.5). This is the standard teacher-student paradigm, and we will refer to the trained student model as T in our
experiments.
Len = 1 |Den| ∑
x∈Den
∑ c∈C −yc(x) log qc(x)+
1 |C||D′en| ∑
x∈D′en
∑ c∈C pc(x) · log
( pc(x)
q∗c (x)
) (1)
Rather than using English source data and English generations, we can also adopt the translatetrain setting (Artetxe et al., 2020) where Den and Gen are translated to the target language using an off-the-shelf neural machine translation system to yield Dtg and Gtg, respectively. The rest of the above-mentioned teacher-student training pipeline stays the same, except with using translated data everywhere.
Teacher-driven Training with Prompt Labels (Tpl). Instead of using a teacher model to pseudolabel the generations in Gen/Gtg, we use the teacher’s label probabilities for data selection (detailed in Section 2.3) after which we label the data using the labels in the LLM prompts that we use for class-conditional generation. A single model is trained using cross-entropy loss on both source data in Den/Dtg and prompt-labeled data sampled from Gen/Gtg. The main difference from teacherstudent training is the use of hard prompt labels for the sampled generations with a cross-entropy loss instead of soft pseudolabels from a teacher model with a KL-divergence loss. Here, we first utilize the teacher for data selection and subsequently use the LLM prompt labels for the generations. This model will henceforth be referred to as Tpl. Similar to T , even with Tpl, we can adopt the translate-train setting and use translated source data and LLM generations.

heading: 2.3 Data Selection Strategies
text: Around 130K instances are generated for each target task from which a small subset is sampled using various data selection techniques described below. In all experiments, we uniformly sample across positive, negative, and neutral class labels for sentiment analysis (and entailment, contradiction, and neutral class labels for NLI) by choosing 2500 instances from the full set of instances for each class to create D′en/D ′ tg.
• rand-k: We select a random subset of 2500 instances from the data generated for each class in Gen/Gtg.
• top-k: Instances specific to each class in Gen/Gtg are sorted in descending order using the teacher model’s predicted probability for that class. The top-k (k = 2500) instances from each class are then selected.
• div-k: We aim to select a diverse set of sentences from each target class using div-k. The sentences belonging to each class (based on teacher labels) are encoded using LABSE sentence embeddings (Feng et al., 2022). The embeddings for each class are then clustered using NLTK’s Kmeans clustering algorithm3. We create 25 clusters for each class and select the top 100 instances using the probabilities assigned by the teacher model (as in top-k) per cluster to get a total of 2500 instances per class. With this simple cluster-then-topk technique, we hope to identify samples that offer good coverage and capture the diversity of samples within each class.
• amb-k and easy-k: We design two additional selection techniques amb-k and easy-k by drawing inspiration from prior work on data cartography (Swayamdipta et al., 2020) where data points are characterized as ambiguous, easy or hard based on the training dynamics across epochs. We first compute predicted probabilities for each class for each instance across checkpoints of the teacher model saved after each training epoch. Next, we compute the mean and standard deviation across probabilities for each instance across training epochs. For each class, instances with the top-k (k = 2500) mean and standard deviation values are chosen as easy-k and amb-k, respectively. High standard deviation values signify larger variability in predictions across training; these instances are characterized as ambiguous examples that the model is unsure about. High mean values signify higher confidence in predictions; these instances are characterized as easy examples that the model is confident about. This selection technique is expensive in having to maintain checkpoints for all training epochs; we evaluate this only for NLI.
3https://tedboy.github.io/nlps/generated/nltk.cluster.html

heading: 3 Experimental Setup

heading: 3.1 Datasets
text: Source data refers to labeled task-specific data in English, while target data refers to evaluation sets in the target languages for which there is no labeled data. Unless specified otherwise, we choose source data to be from a different domain compared to the target data. This is different from most prior work in zero-shot evaluations where the source data is typically chosen to be consistent in the domain to the target tasks (Whitehouse et al., 2023; Li et al., 2021; Du et al., 2021; Vu et al., 2022). We assume a more realistic setting where the source and target domains can be mismatched.
Source data. We use SST5 (Socher et al., 2013) and SNLI (Bowman et al., 2015) datasets for sentiment analysis (SA) and natural language inference (NLI), respectively. SST5 is a sentiment classification dataset featuring five distinct labels: negative, very negative, positive, very positive, and neutral, that we collapse into three labels: positive, negative and neutral to match the target tasks. Similar to (Li et al., 2021), we consider a random subset of the SNLI train set (15K training sentences, 5K per class) to simulate a low-resource setting and for quicker experimental turnaround.
Target data. Our target SA tasks include Marathi Sentiment (Pingle et al., 2023), GLUECoS HindiEnglish code-switched Sentiment (Khanuja et al., 2020), and Hindi Product Reviews (Akhtar et al., 2016). For NLI, we evaluate on Hindi, Urdu, and Swahili from the XNLI (Conneau et al., 2018) corpus; these are some of the the least-represented XNLI languages. Appendix A provides more details about the source and target tasks. Appendix C shows how we generate code-mixed data for the translate-train setting of the GLUECos task.

heading: 3.2 Model and Training details
text: For all our experiments, we use the xlm-robertalarge model (Conneau et al., 2019) for modeling both the student and teacher. It is a 561M parameter model.4 Our choice of XLM-R for classification tasks was motivated by recent work on cross-lingual classification (Artetxe et al., 2023) that uses only XLM-R for all its evaluations. There is also prior work (Zhang et al., 2023) that shows that compared to much larger multilingual LMs
4https://huggingface.co/xlm-roberta-large
like BLOOMZ, etc., fine-tuned models of smaller scale like XLM-R are at par or superior on many cross-lingual classification tasks for low-resource languages. Both the student and teacher models are trained for 15 epochs, with a learning rate of 5e-6, AdamW as the optimizer, batch size of 32, and gradient accumulation step size of 4. The student model uses a temperature of 1.5 for the KL-divergence loss. We use the best checkpoint model for all the evaluations, where the best checkpoint is selected based on accuracy over the source dev set. Translations are obtained using IndicTrans2 (AI4Bharat et al., 2023) for all the languages except Swahili, for which we use NLLB (Team et al., 2022).

heading: 3.3 Baselines
text: Source only (SRC). Here, the model is trained on the train set of the source tasks (refer Section 3.1). No synthetic data is used to train the model. 8,544 and 15K instances are used for SA and NLI tasks, respectively.
Source+Generations (SRC+GEN). Here, the model is trained on a mixture of source and synthetic datasets. The synthetic dataset (7.5K) is sampled randomly from among the generations and is not selected via any data selection technique or with the help of a teacher resulting in 16K and 22.5K instances for SA and NLI tasks, respectively.
Generations only (GEN). Here, we train the model only on the synthetically generated data. The labels come from the prompts that we used for class conditional generation. For a fair comparison with SRC+GEN, we maintain the same total size of 16K and 22.5K instances for SA and NLI tasks, respectively

heading: 4 Results and Analysis

heading: 4.1 Main Results
text: We characterize the training data used for each model along two axes: source of the task-specific text and source of labels assigned to the data. GEN, SRC and SRC+GEN in Table 1 represent the baseline models as described in Section 3.3. Generated text used in all the baseline models is combined with the corresponding prompt labels used during generation. We observe that substituting a portion of generated instances with source instances (SRC+GEN) yields better performance, as
anticipated, compared to using GEN alone. Further augmenting the source data with generations (SRC+GEN) boosts the SRC baseline across all evaluated tasks/languages.
The results in Table 1 are all translate-train accuracy values since they are found to largely outperform the zero-shot numbers, thus highlighting the benefits of using (machine) translations for cross-lingual evaluations (as reported in prior work (Artetxe et al., 2023)). Please refer to Appendix B for zero-shot results. Our reported numbers are averaged across models trained on two different random seeds.5 Following the three rows of baseline numbers in Table 1, we show results using models that are trained across two different levels of supervision from the SRC baseline (acting as the teacher). T indicates that data selection is done using teacher pseudolabels while Tpl indicates that after data selection using teacher pseudolabels, for each instance, prompt labels are used for subsequent training (instead of retaining the teacher-assigned labels). Each of the listed models is trained on data selected using various selection strategies detailed in Section 2.3. The T models are trained with soft pseudolabels derived from the teacher while the baselines and Tpl models are trained with hard prompt labels. For a given
5For the Hindi SA task, due to the considerably smaller size of the evaluation sets, we trained the models using six different seeds to obtain more reliable evaluations.
strategy (top-k, rand-k, etc.), we note that the same unlabeled data subsets are used with one of the two kinds of labels (teacher soft vs. prompt hard).
Across all tasks, we observe that data selection strategies yield consistent performance improvements over the best baseline with absolute accuracy gains of up to 7% for Hindi Product SA. Tpl appears to do better overall i.e., prompt labels after teacherbased data selection; the teacher labels perform much better just on the Hindi Product task. We note here that unlike prior work that uses LLM-based augmentations for cross-lingual tasks (Whitehouse et al., 2023) with access to some target data, all our models are trained without any access to real target data.
We find that the delta values in Table 1 using our data selection techniques for XNLI and Marathi SA (that have a similar number of test instances) are statistically significant at p < 0.01 using the Wilcoxon signed rank test. Since the Hindi product review task has a significantly smaller number of test instances, we treat it separately across different random seeds and find that top-k data selection results in a statistically significant improvement (compared to SRC+GEN) at p < 0.05 using the Wilcoxon signed rank test.
Other than the five target sets in Table 1, in Table 2 we also evaluate on a code-switched HindiEnglish sentiment analysis task which is yet another challenging low-resource domain. Unlike
Table 1 where translate-train was more effective, we show zero-shot scores since the presence of English words in code-switched En-Hi text is found to benefit more from SRC+GEN (teacher) trained on original generations in English. More details about the code-mixed text generation and the corresponding Translate-train numbers are provided in Appendix C.

heading: 4.2 Experimental Analysis
text: Ambiguous/Easy Data Selection. Table 3 shows XNLI results of student models/prompt-based models trained on data selected using amb-k and easyk selection techniques. Augmenting the source data with prompt-labeled ambiguous instances benefits the model the most. Ambiguous instances are ones that the model is most uncertain about and are likely to help the model generalize well. This is consistent with observations about ambiguous instances in prior work (Swayamdipta et al., 2020; Liu et al., 2022).
Soft Labels vs. Hard Labels. Table 4 shows the translate-train accuracies of a student model (T ) trained using teacher hard pseudo labels and soft pseudo labels. CE implies training using teacher hard labels and cross-entropy loss, KLD implies training using teacher soft labels and KLdivergence loss. Large delta values highlight the significance of using soft teacher labels instead of hard labels. If the teacher labels are noisy, soft labels help the student model generalise better to unseen data.
Effect of Varying Sizes of Augmented Data. To study the effect of augmented data size on crosslingual transfer, we experiment with div-k selection (T model) and SRC+GEN model for the Marathi SA task. Models are trained in the translate-train
setting over varying amounts of augmented data. Table 5 shows that increasing k leads to a decrease in accuracy. The consistent downward trend in the div-k selection technique underscores the importance of data selection; the best accuracies were obtained using div-k with k = 7500. Also, augmenting synthetic data also results in increased training time. Determining the optimal augmentation size for each target task is left as future work.
Augmenting with Target Train Data. To explore whether LLM generations boost performance even in the presence of source data that matches in domain to the target task (henceforth referred to as target training data), we train teacher models on a subset of 15K sentences from the XNLI train set. (Note that the numbers in Table 1 were obtained using SNLI as the source data.) As expected, we see significant improvements in teacher accuracies in Table 6 when using target train data. Student models trained on generations pseudolabeled with this superior teacher further boost accuracies; the best results are obtained using a combination of
translate-train and div-k selection. Also, the accuracies obtained using SRC+GEN models are inferior to the best scores obtained using T models, again highlighting the importance of data selection.
Generations Uniform across Classes. By default, we create class-balanced augmentations by sampling 2500 instances from each class based on teacher labels. To analyse the effect of class imbalance on downstream evaluation, we augment the sentiment source data with class-imbalanced augmented data sets for the Marathi SA task. The total size of the augmented data remains constant, while the class distribution is altered by eliminating neutral sentences from 2500 to 0, thereby transitioning towards sentiment-rich augmentations. We see in Table 7 that students trained on data that is uniformly distributed across classes along with the top-k selection strategy exhibits superior performance, compared to those trained on a subset of the generated data with imbalanced class proportions. This suggests that employing a class-balanced augmentation is an important consideration.
Quality of prompt labels. To evaluate whether the LLM prompt labels are truly reflected in the filtered generated text or not, we ran a human evaluation on a set of 100 generations each for sentiment analysis and NLI. These instances were randomly
selected generations from among a set of pseudolabels predicted with high probability by the teacher model for the respective tasks. The average accuracy of label alignment between annotator-provided labels and prompt-derived labels was found to be 87.88% and 71.72%, with Cohen’s kappa coefficients of 0.752 and 0.749, respectively for the two tasks. This suggests that the prompt labels in the subset obtained after teacher-based filtering are of fairly high quality. Please refer to Appendix G for more details.
Analyzing Diversity. We introduce a simple metric that we call a “diversity score" to capture the dissimilarity in text embeddings across sentences in a dataset. This is computed by encoding each instance using LABSE (Feng et al., 2022), taking the average of the cosine distance of the LABSE embedding with every other instance in the data sample and finally taking an average of these mean distances across all data samples. To check if the data selected using the div-k selection technique is indeed diverse, we compute the diversity score for each task and data selection strategy. Figure 2 shows the trend of diversity scores. It is clear that the diversity of the 7500 sentences selected using div-k technique is greater than the diversity of the sentences selected via top-k and rand-k across all tasks.
Cross-Domain Analysis. Recall that the prompts to the LLM also contained domain information of the target task. To evaluate the impact of domainspecificity of the generations on zero-shot performance, we create two cross-domain datasets in the medical and law domain (unrelated to the target task domains). Table 8 shows zero-shot results of
the student models trained on the in-domain and out-of-domain (law, medical) augmented datasets for the Marathi sentiment and XNLI Hi tasks. We observe that the student model trained on out-ofdomain datasets perform comparably to the student models trained on in-domain data. This shows that models can effectively learn task information from augmented data even if it comes from domains that differ from the target task.
To further disentangle the roles of task and domain, we made an attempt to capture sentiment richness present in the in-domain vs out-of-domain data for the Marathi SA task. We use a sentiment lexicon6 consisting of a positive and negative sentiment score for each word along with its POS tag, word sense, etc. We simply add the corresponding scores for each word in the corpus found in the lexicon (along with matching the POS tag) and normalize the sums by the word count of the corpus. Table 9 shows that the computed sentiment scores for the generated out-of-domain datasets are much better than the in-domain dataset for Marathi SA. Here, overall score is calculated by summing both the positive and negative scores for each word. Hence, it is plausible that the sentiment richness in the generated out-of-domain datasets compensates for the domain mismatch, thus yielding comparable
6We used the SentiWordNet 3.0 sentiment lexicon available at: https://github.com/aesuli/SentiWordNet? tab=readme-ov-file).
or slightly better results in Table 8. (Tables 28, 29 in Appendix E show examples of generations from different domains.)

heading: 5 Related Work
text: Our work is closely related to Whitehouse et al. that studies generations from various open-source and commercial LLMs for cross-lingual performance over reasoning tasks. However, they rely on instances from the target sets as few-shots for generations and do regular finetuning over labels derived from the class-conditional prompts.
Furthermore, our work is grounded in ideas inspired by He et al., incorporating self-training on unlabeled synthetic text produced by Language Models. However, He et al. fine-tuned generators using target data, and their experiments were limited to GLUE tasks (in English). In contrast, our focus is on multilingual models, aiming for crosslingual transfer from source data in a high-resource language across arbitrary domains in different task languages. We achieve this by self-training on zeroshot generations from LLMs without utilizing any target data during generation or training.
We draw inspiration from (Swayamdipta et al., 2020) to design data selection techniques based on the principle of dataset cartography. (Liu et al., 2022) use dataset cartography on a large NLI dataset (MNLI) to choose instances with complex reasoning patterns, and instructs GPT-3 to generate new examples with similar patterns. Automatically generated examples undergo filtering, and ultimately, human crowdworkers review, revise, and label them. In a similar vein, Khanuja et al. present language-agnostic methods to pick specific data points to be labeled from a large, unlabelled multilingual dataset. These points are chosen either by considering their distance from the target set, the uncertainty of model predictions over them, or finding a balance between minimizing distance and maximizing model uncertainty. While WANLI (Liu et al., 2022) only explores English generation/evaluation, both depend not only on human-in-the loop annotation of unlabeled text, but also depend on existing target data for generator finetuning (Khanuja et al., 2023) or few-shot prompting (Liu et al., 2022).
De Raedt et al. propose in-place augmentation of data instances from high-resource languages for better out-of-distribution generalization by leveraging LLMs. However, these techniques are specifi-
cally applicable to single-text classification tasks. For tasks like NLI or QA, which involve multiple components in each instance, automatically making such edits or augmentations while preserving the intended relationships between components (and affecting labels) is not straightforward.
In contrast to (Li and Callison-Burch, 2023), and (Riabi et al., 2021), we make use of an open-source LLM to generate task and domain-specific synthetic data. We also explore the more realistic setting of having no access to source data that matches the target domain; this is not explored in the above two works. Similar to our work, synthetic data generation has been explored in (Agrawal et al., 2023; Gekhman et al., 2023). However, both works do not use any form of data selection for the synthetically generated data. (Gekhman et al., 2023) do not show evaluations on low-resource languages; their multilingual experiments are limited to high-resource languages such as English, French, Spanish and German. (Agrawal et al., 2023) show experiments in the few-shot setting, while we operate in the zero-shot setting.

heading: 6 Conclusion
text: In this work, we focus on the broader problem of boosting zero-shot cross-lingual transfer using LLM-based augmentations. We highlight the importance of using data selection strategies to select smaller subsets that result in more efficient training and improved performance on downstream target language tasks. We also compare and contrast the utility of pseudolabeling generations using labels from LLM prompts versus using a teacher model to label the generations. One of the main takeaways is that LLM generations, in conjunction with our data selection strategies, can help improve cross-lingual transfer regardless of whether task-specific source data matches the domain of the target tasks or not.

heading: Acknowledgements
text: The authors thank the anonymous reviewers for their constructive feedback and engagement during the rebuttal that helped improve the quality of the draft. The last author would like to gratefully acknowledge a faculty grant from Google Research India supporting her research on multilingual models.
Limitations
1. The generations from LLMs are sensitive to the prompts used. Although we share our custom prompts, the quality of the generated content is heavily reliant on the particular domain and task for which the data is generated creating some non-determinism.
2. Because of budget constraints, our investigations were constrained to an open-source LLM (LLAMA-2). It is possible that highercapacity commercial LLMs could yield better performance.
3. We explore many data selection techniques but a clear winner across all tasks/settings has not emerged. Although ambiguous selection gives best scores for XNLI, more target domains and languages should be included to study the most effective filtering techniques in general.
4. We have only experimented with generating data for classification tasks; generating data for more structured tasks like QA or commonsense reasoning tasks could pose challenges.
5. For the translate-train models, one assumes access to MT models for the target language which may not always be available.

heading: A Dataset Description
text: Table 10 shows the details of the source tasks. SST5 is used as a source task for all the sentiment tasks, and SNLI as a source task for the XNLI tasks. Table 11 shows the details of the different target tasks. We treat both the dev and test sets of these target tasks as test sets and do not use their train sets for model training. The language "Hinglish" refers to the Hindi-English code-mixed text, the data is present in a mixed script (both Romanized and Devanagari) form. Table 12 shows the domain information present in the different target tasks. The domain list is not exhaustive, we made use of the domains that were easier to represent in a prompt.

heading: B Zero-shot evaluations
text: Tables 13, 15, 16, 17 shows the zero-shot results of the baselines and our techniques. In table 13, except for a few languages, our techniques beat the
baselines by a reasonable amount. Table 14 shows the translate-train numbers of the GLUECos task. The evaluation set of GLUECos is in romanized Hindi-English (code-switched text), we suspect this to be the reason for the negative delta.

heading: C Code-mixed Text generation
text: To generate code-mixed text in mixed-script format for the GLUECos sentiment analysis task, we trained a mt0-xl (Muennighoff et al., 2023) model. The pre-trained model weights were downloaded from the HuggingFace repo7. The training is done using the huggingface’s Trainer API. We made use of Lora (Hu et al., 2021) PEFT technique (Mangrulkar et al., 2022) to train the model, with the hyperparameters: alpha=32, dropout=0.05, r=16, and Lora matrices being applied to query, key, and value attention matrices. The model is trained for 15 epochs, train batch size of 16, gradient accumulation steps of 4, learning rate of 2e-4, max grad norm of 0.3, and warmup ratio of 0.03. The best model checkpoint was selected using the evaluation loss. The data for training the model was obtained from (Mondal et al., 2022). Table 18 shows few examples. Before generating the translate-train code-mixed text, we first translate the SST5 English train set to Hindi, which we then feed into our model to get the Hinglish mixed-script outputs.

heading: D LLAMA-2 Generation Details
text: We use LLAMA-2 13b-chat-hf model for all our generations, as it is a recent state-of-the-art and open-source model. We use a 4-bit quantized version of the model owing to memory constraints. For quantization, we make use of the Bitsandbytes library (Dettmers et al., 2022). We use nucleus
7https://huggingface.co/bigscience/mt0-xl
sampling for all our generations with p=0.9, we avoid repeating bi-grams, and we keep a temperature between 1.5-2.5 depending on the response of the model to the input prompt. We generated
approximately 2lac sentences for each task, which came down to 1.5lac sentences post-processing and cleaning. We took a random subset of 1.3 lac sentences from the total generations, to which we then applied data selection techniques.

heading: E LLAMA-2 Prompt Details
text: Tables 19, 20, 21, 22, 23, 24, 25, 26, 27 show the prompts we used to generate data for different target tasks.8 Custom prompts are employed for each
8We also tried generating task-specific sentences without specifying any domain information. This resulted in noisier generations compared to when we added a domain description in the prompt. This might be an artefact specific to the LLM
target class. In the case of the Hindi product, where the goal is to generate product-specific reviews, numerous neutral (class-conditioned) generations still exhibited subtle indications of positive or negative sentiments. Consequently, we generated additional neutral sentences using a slightly modified prompt to achieve a balanced distribution of classes in the augmented dataset.
For the Marathi Sentiment target task, we devise specific prompts for various domains such as political tweets, generic tweets, subtitles, and movie reviews. To create premise-hypothesis pairs for XNLI, we initially generate the premises providing domain information in the prompts. The hypothesis is then generated using the respective premise as input along with the NLI label (entailment, neutral, and contradiction). Tables 28, 29 show the generations belonging to different domains for sentiment classification and natural language inference (NLI) task.

heading: F Computational Budget
text: Training a student model on an Nvidia A100 GPU with 80G of RAM took ~100 mins for 15 epochs. We utilized the same GPUs for text generation. Generating ~2 lac sentences of max-length 256 tokens using LLAMA-2 13B model with a batch size of 60 took ~20 hrs using a single GPU and ~50 GB of RAM.

heading: G Annotator Details
text: Both annotators had professional competence in English. The instructions given for the two tasks are listed below:
1. Sentiment Task: Given a sentence, identify the polarity of sentiment which could be one of the three types: positive, negative and neutral.
2. NLI Task: Given two pieces of text called premise and hypothesis, mark the pair as “entailment” if premise entails the hypothesis, “contradiction” if premise contradicts the hypothesis and “neutral” if there’s neither entailment nor contradiction i.e the factuality of two statements is independent from each other.
we used, and needs further investigation.

heading: Future Work
text: Determining the optimal augmentation size for each target task is left as future work. Augmenting with Target Train Data. To explore whether LLM generations boost performance even in the presence of source data that matches in domain to the target task (henceforth referred to as target training data), we train teacher models on a subset of 15K sentences from the XNLI train set. (Note that the numbers in Table 1 were obtained using SNLI as the source data. ) As expected, we see significant improvements in teacher accuracies in Table 6 when using target train data. Student models trained on generations pseudolabeled with this superior teacher further boost accuracies; the best results are obtained using a combination of
translate-train and div-k selection. Also, the accuracies obtained using SRC+GEN models are inferior to the best scores obtained using T models, again highlighting the importance of data selection. Generations Uniform across Classes. By default, we create class-balanced augmentations by sampling 2500 instances from each class based on teacher labels. To analyse the effect of class imbalance on downstream evaluation, we augment the sentiment source data with class-imbalanced augmented data sets for the Marathi SA task. The total size of the augmented data remains constant, while the class distribution is altered by eliminating neutral sentences from 2500 to 0, thereby transitioning towards sentiment-rich augmentations. We see in Table 7 that students trained on data that is uniformly distributed across classes along with the top-k selection strategy exhibits superior performance, compared to those trained on a subset of the generated data with imbalanced class proportions. This suggests that employing a class-balanced augmentation is an important consideration. Quality of prompt labels. To evaluate whether the LLM prompt labels are truly reflected in the filtered generated text or not, we ran a human evaluation on a set of 100 generations each for sentiment analysis and NLI. These instances were randomly
selected generations from among a set of pseudolabels predicted with high probability by the teacher model for the respective tasks. The average accuracy of label alignment between annotator-provided labels and prompt-derived labels was found to be 87. 88% and 71. 72%, with Cohen’s kappa coefficients of 0. 752 and 0. 749, respectively for the two tasks. This suggests that the prompt labels in the subset obtained after teacher-based filtering are of fairly high quality. Please refer to Appendix G for more details. Analyzing Diversity. We introduce a simple metric that we call a “diversity score" to capture the dissimilarity in text embeddings across sentences in a dataset. This is computed by encoding each instance using LABSE (Feng et al. , 2022), taking the average of the cosine distance of the LABSE embedding with every other instance in the data sample and finally taking an average of these mean distances across all data samples. To check if the data selected using the div-k selection technique is indeed diverse, we compute the diversity score for each task and data selection strategy. Figure 2 shows the trend of diversity scores. It is clear that the diversity of the 7500 sentences selected using div-k technique is greater than the diversity of the sentences selected via top-k and rand-k across all tasks. Cross-Domain Analysis. Recall that the prompts to the LLM also contained domain information of the target task. To evaluate the impact of domainspecificity of the generations on zero-shot performance, we create two cross-domain datasets in the medical and law domain (unrelated to the target task domains). Table 8 shows zero-shot results of
the student models trained on the in-domain and out-of-domain (law, medical) augmented datasets for the Marathi sentiment and XNLI Hi tasks. We observe that the student model trained on out-ofdomain datasets perform comparably to the student models trained on in-domain data. This shows that models can effectively learn task information from augmented data even if it comes from domains that differ from the target task. To further disentangle the roles of task and domain, we made an attempt to capture sentiment richness present in the in-domain vs out-of-domain data for the Marathi SA task. We use a sentiment lexicon6 consisting of a positive and negative sentiment score for each word along with its POS tag, word sense, etc. We simply add the corresponding scores for each word in the corpus found in the lexicon (along with matching the POS tag) and normalize the sums by the word count of the corpus. Table 9 shows that the computed sentiment scores for the generated out-of-domain datasets are much better than the in-domain dataset for Marathi SA. Here, overall score is calculated by summing both the positive and negative scores for each word. Hence, it is plausible that the sentiment richness in the generated out-of-domain datasets compensates for the domain mismatch, thus yielding comparable
6We used the SentiWordNet 3. 0 sentiment lexicon available at: https://github. com/aesuli/SentiWordNet? tab=readme-ov-file). or slightly better results in Table 8. (Tables 28, 29 in Appendix E show examples of generations from different domains. )