LLM-powered Data Augmentation for Enhanced Cross-lingual Performance
Authors: Chenxi Whitehouse
Monojit Choudhury
Alham Fikri Aji
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Data Augmentation
Multilingual Commonsense Reasoning

Abstract: This paper explores the potential of leveraging Large Language Models (LLMs) for data augmentation in multilingual commonsense reasoning datasets where the available training data is extremely limited.
To achieve this, we utilise several LLMs, namely Dolly-v2, StableVicuna, ChatGPT, and GPT-4, to augment three datasets: XCOPA, XWinograd, and XStoryCloze.
Subsequently, we evaluate the effectiveness of fine-tuning smaller multilingual models, mBERT and XLMR, using the synthesised data.
We compare the performance of training with data generated in English and target languages, as well as translated English-generated data, revealing the overall advantages of incorporating data generated by LLMs, e.g. a notable 13.4 accuracy score improvement for the best case.
Furthermore, we conduct a human evaluation by asking native speakers to assess the naturalness and logical coherence of the generated examples across different languages.
The results of the evaluation indicate that LLMs such as ChatGPT and GPT-4 excel at producing natural and coherent text in most languages, however, they struggle to generate meaningful text in certain languages like Tamil. We also observe that ChatGPT falls short in generating plausible alternatives compared to the original dataset, whereas examples from GPT-4 exhibit competitive logical consistency.

LLM-powered Data Augmentation for Enhanced Crosslingual Performance
Chenxi Whitehouse 1,3+
Monojit Choudhury?
Alham Fikri Aji²
Cityy University of London "Microsoft
MBBZAA
chenxi wwi whitehouse@city ac uk
monojitcemicrosoft..o com
alham. fikri@mbzuai ac.aa
Abstract
model is trained across data of diverse tasks and languages, exhibiting the capability to handle un- seen tasks, particularly in larger models (Artetxc and Schwenk, 2019: 9; Nooralahzadeh et al., 2020; Huang et al., 2021). However, when aiming for task-specific objectives, smaller, fine-tuned model dedicated to that particular task often outperforms larger perppss- zero-shot models. In addi- tion, smaller task-specific model is more practi- cal and cost-effective for training and deployment. Nevertheless, developing a powerful task-specific model becomes challenging in the absence of train- ing data (Lauscher et al., 2020).
This paper explones the potential of leveraging Large Language Models (LLMs) for data aug- mentation in multilingual commonsense rea- soning datasets where the available training data is extremely limited. To achieve this, utilise several LLMs, namely Dolly-v2, Sta- Vicuna, ChatGPT, and GPT-4, to augment three datasets: XCOPA, XWinograd, and XS- toryCloze. Subsequently, we evaluate the ef- fectiveness of fine-tuning smaller multilingual models, mBERT and XLMR, using the syn- thesised data. We compare the performance of training with data generated in English and target languages, as well as translated English- generated data, revealing the overall advantages of incorporating data generated by I.LMs, e.g. a notable 13.4 accuracy score improvement for the best case. Furthermore, we conduct a hu- man evaluation by asking native speakers to assess the naturalness and logical coherence of the generated examples across different lan- guages. The results of the evaluation indicate that LLMs such as ChatGPT and GPT-4 ex- cel at producing natural and coherent text in most languages, however, they struggle to gen- erate meaningful text in certain languages like Tamil. We also observe that ChatGPT falls short in generating plausible alternatives com- pared the original dataset, whereas exam- ples from GPT-4 exhibit competitive logical consistency. We release the generated data at https:///ithuu com/mbzuai ceptuuaiinnp/Geee -nlp/Gen-X
Conversely, recent powerful Large Language Models (LLMs) excel at handling general instruc- tions and have shown promise in data genera- tion tasks (Wang et al., 2023). In this work, we leverage LLMs to generate synthetic data for var- ious multilingual commonsense reasoning tasks, XCOPA (Ponti et al., 2020). XWinograd (Tikhonov and Ryabinin, 2021), and XStoryCloze (Lin et al., 2022), where the training data is limited even for Linglish (see Table 1). To augment the training data, we provide LLMs with instructions and examples from the original training data, prompting them to generate new and diverse examples. We explore the generation of synthetic data in English using different LLMs, including open sper-source models like Dolly-v2' and Stable Vicuna?, as well as ChatGPT and GPT-4. Although the weights and capabili- ties of the latter two models remain undisclosed, explore them they extend the capability of generating texts languages beyond English.
1 Introduction
The success of NLP models greatly depends on the availability and quality of training data. This poses a significant challenge for multilingual NLP. as data for languages other than English is typically limited (Ponti et al., 2019; Joshi et al., 2020; White- house et al., 2022). An approach to address the data scarcity challenge through zero-shot cross- lingual transfer or multitask training, in which
We develop asecifpec models by fine-tuning multilingual pre-trained language models, namely mBERT (Devlin al., 2019) and XLM-R (Con- neau et al., 2020), using the generated data. We then compare their performance against models trained on a limited set of human-created data in the
It tes:tps//// //gi thub. wwwpdattabrckkllaassoooooa Pht testtp////ti wwtp///aaallttyl///taaaaaccc
Work conducted while visiting MBZUAI.

---

Train
Validation
the naturalness and logical coherence of the data generated in various languages; Releasing the synthesised datasets for public use and reproducibility.
DATASET
XX
EN
XCOPA
400
0
100
100
500
500
XWinograd
1858
0
233
0
233
424
XStoryCloze
300
300
60
60
1511
1511
2
Related Work
Table Number of examples available XCOPA, XWinograd, and XStoryCloze. XX denotes the average number of non- non-Englin English examples per language. Since validation split not specified XStoryCloze, we take 60 random examples from the train split for validation. XWinograd has no train/validationntesl split, and we follow an 80/10/10 split for the experiments.
Multilingual and Low-Resource NLP
Recently, there has been increased attention on expanding NLP beyond English, including the de- velopment of multilingual models (Devlin et al., 2019; Conneau et al., 2020; Xue et al., 2021; Scao et al., 2022) well as the creation of benchmarks to address multilingual challenges (Conneau et al., 2018; 8; Artetxe et al., 2020; Adelani et al., 2021;
target language whenever available, and otherwise through zero-shot transfer learning from manually created English training data. Our experiments demonstrate that training the models with relatively large synthetically generated datasets yields better performance than training with limited manually- created datasets. This finding empirically confirms the utility of synthetic data generated by LLMs for improving downstream task-specific models.
Winata al., 2023). Among the prevailing chal- lenges faced across various languages, common theme the scarcity of available data.
Consequently, when data is lacking, one ap- proach is to employ zero-shot cross-lingual trans- fer. Studies conducted by Winata al. (2023) have demonstrated the effectiveness of zero-shot cross- lingual transfer for related languages. Additionally, Muennighoff et al. (2023) show that models fine- tuned only with English instruction data are capable of understanding multilingual instructions. In this work,
We expand the multilingual data synthesis using ChatGPT and GPT-4 on XCOPA and find that gen- erating multilingual datasets generally surpasses the effectiveness of the zero-shot cross-lingual transfer. We further assess the quality of the gener- ated dataset different languages by asking native speakers to evaluate the naturalness and logical soundness of the generated dataset compared to the numan-writted examples. The annotation results reveal that while ChatGPT and GPT-4 successfully generate natural text in most languages, they strug- gle with generating understandable text in certain languages such as Tamil. Moreover, a noticeable gap is observed in terms of commonsense coher- data to
tackling similar scenario where the availability of data is limited.
Multilingual Data Augmentation
Lauscher et al. (2020) show that few-shot can dras- tically increase the cross-lingual performance of small models, proving that multilingual data aug- mentation is effective strategy. A series of works try predict the cross-lingual accuracy of models through measurements and modelling (Xia et al., 2020), and study strategies for multilingual data augmentation, such as choosing the transfer lan- guages (Lin et al., 2019), and predicting multilin- gual few-shot accuracy leading for optimal data augmentation approaches (Srinivasan et al., 2022).
ence when comparing ChatGPT-geneenn human-construutted data. On the other hand, GPT-4 significantly narrows this difference.
To summarise, our work has the following key contributions:
Augmenting three low-resource, multilingual commonsense reasoning datasets by leverag- ing and prompting four LLMs;
Many works focus on synthetic data augmenta- tion for code-mixing including utilising linguistic theories (Lee et al., 2019; Pratapa et al., 2018), ma- chine translation models (Tarunesh et al., 2021), parallel corpus and Wikipedia (Winata et al., 2019; W Whitehouse et al., 2022), and employing ChatGPT (Dai et al., 2023). Our work explores data augmen- tation on multilingual commonsense datasets with powerful instruction-tuned LLMs.
Fine-tuning smaller models, mBERT and XLMR, using the synthesised data and show- casing the practical value of the LLM- generated data:
Performing an extensive analysis of the effects of various target languages in data generation and scaling, as well as a human evaluation of

---

XCOPA
XWINOGRAD
XSTORYCLO2E
We are collecting more examples the COPA dataset which will be used to test system's ability Commonsense Causal Judgments. The format of the data:
are collecting more examples the Winegrad Schema Challenge. Each example has short sentence that con- tains two phrases and one pronoun replaced by "_ and the challenge is to de- terming the referent of the promoun, which can only be inferred from the context. Here are n. examples of the data:
We are collecting more examples story cloze datasct. Each example consists of 4-seatence story. one correct eoding sentence which is a plausible continu- ation of the story, and wrong ending sentence which is logically inconsistent with the context.
A premise; statement of something that huppened, and two choices that could plau- sibly (occur as the result be the causse) of the premise. The correct thoice is the alternative that is more plausible than the wrong choice.
Here are examples of the data:
Example I: Sent-1: Tina is very tired every single morn- ing. Sent-2: She does not get enough sleep because of her two jobs. Scat-3; Tina decides to quit one of the jobs. Sent-4: now gets enough sleep to function everyday. Correct ending:
Example I: Seuteace: Harley hides from Dyna because is scary. Who/What is scary? Correct answer: Dyna. Wrong am- Hurley.... ... Exumple Based on the examples above, generate mn new examples. Both noun phrases in each example can be males, females, inan- immate ohjects, or groups of people or ob- jects. There should only be one "_ in the sentence. The correct should be one of the noun phrases men- tioned in the sentence.
Here are n examples {L
is well rested. Wrong ending: Tina more tired thun ever hefire. Example Based on the examples above, provide on new similar examples. Reguirements: 1) the story should read like a coherent story, with a specific boginning and ending, w where something huppens in hetween 2) both endiing sentences should be entirely reasonable, realistic and sensible when read in isolation, and 3) both ending sen- tences should follow up the story by sharing least one characters the story.
Exumple I: Premnise: The man wanted result?
money. What happened
Correct choice: He cut back on making frivolous purchases. Wrong choice: He
withdrew money from his savings account. Example n: ......
Based on the examples ahove, generate yn new examples in (language).
wrong answer
@ Premise: The politician made contro- versial statement. What happened
@ Sentence: Sam gave Andrew book hecause_. had already read it. Who/What had already read the book? Correct
2 Scat-1: 1: Jordan was a high school student who wanted become doctor. Sent 2: He spent all his free time studying biology and chemistry. Sent 3: One day, his school hosted a science fair competition. Sent-4; Jor- dan's 'sss project won first place. Correct eading: Jordan went on to study medicine in college. Wrong ending: Jordan gave up his dream of becoming doctor.
sult? Correct choice: The politician faced criticism from the media. Wrong choice:
swer; Sam, Wrong answer: Andrew.
The politician's 's approval ratings increased.
@ Sentence: The dog chused the cat, but was too fast. Who/What was too fast? Correct answer: the cat. Wrong answer:
Premise: f& T tI # Hil P1 1 T.. JL
T- What was the cause? Correct choice:
X O 22 H:
Wrong choice:
The dog.
# 7 FEEE)
Table 2: Examples of instructions and L.M--repponses (ChatGPT) for XCOPA, XWinograd, and XStoryCloze.
3 Dataset Augmentation
ing to evaluate the commonsense reasoning ability of machine. Given a statement with two noun phrases and pronoun, the challenge of WSC is to determine the referent of the pronoun, which can only be inferred from the context.
Our experiments use XCOPA, XWinograd, and XS- toryCloze, which are selected due to (1) the limited availability of training data and (2) commonsense reasoning datasets present greater challenges for data synthesis. Table 1 summarises the statistics of the three datasets.
KStoryCloze is collected by Lin et al. (2022), where the validation split of the original English StoryCloze dataset (Mostafazadch et al., 2016) is translated into 10 other typologically diverse lan- guages (RU, ZH, ES: Spanish, AR: Arabic, HI: Hindi, ID, TE: Telugu, SW, EU: Basque, and MY: Burmese). Each example consists of a four- sentence commonsense story, a correct ending, as well as a wrong ending.
XCOPA is a cross-lingual Choice of Plausible Alternatives dataset that translates and re-annotates the validation and test sets of English (EN) COPA (Roemmele et al., 2011) into 11 target languages (ET: Estonian, HT: Haitian Creole, ID: Indonesian, IT: Italian, QU: Quechua, SW: Swahili, TA: Tamil, TH: Thai, TR: Turkish, VI: Vietnamese, and ZH: Chinese). 3 Each instance consists of a premise, question (cuaselresulr), and two alternatives. The task is predict the more plausible alternative.
3.1 LLMs for Data Generation
Our preliminary experiments reveal that language models that are specifically fine-tuned on down- stream NLP tasks, such as BLOOMZ (Scao et al., 2022) and Flan-T5 (Chung et al., 2022), struggle to follow the complex instructions. Conversely, more recent LLMS such as Dolly-v2, StableVicuna ChatGPT, and GPT-4, which are designed to han- dle more intricate and general-purpon instructions, have demonstrated success in following our instruc- tions for data generation. ChatGPT and GPT-4 also
XWinograd expands the original English Wino- grad Schema Challenge (WSC) (Levesque al., 2012) to five other languages (FR: French, JA: Japanese, PT: Portuguese, RU: Russian, and ZH)." which consists of pronoun resolution problems aim-
>nttps://huggingfadd httpss//hhggnngnee.ooodasssss/st co/datasetss/xopp "https::/huggiingface http://gggnnngaccco./tttttteeeee co//attasett//eemmighffff
xwinograd

---

stand out with the capability of generating exam- ples in non-English languages.
Model
XCOPA
XWinograd XStory
DOLLY-V2
41.6%
22.4%
41.2%
We explore synthetic data generation with the four aforementioned LLMs, balancing be- tween open-access models and closed models (see $5.1). Specifically, we use dolly-v2-12b, which is derived from EleutherAI's Pythia- 12b (Bider- man et al., 2023) and fine-tuned structions generated by Databricks employees; and StableVicuna-11 an RLHF (reinforcement learning from human feedback) fine-tuned Vicuna model on various conversational and instructional datasets Vicuna is an open-source LLaMA model (Touvron et al., 2023a) fine-tuned on user-shared conversations collected from ShareGPT.
STABLEVICUNA
36.1%
33.8%
36.1%
CHATGPT GPT-4
86.4%
43.8%
77.6%
89.7%
85.0%
89.3%
Table 3: Generation Success Rate in English (valid examples obtained / total examples requested) with dif- ferent LLMs on the three datasets.
~15K in-
mats. We report the success rate for different LLMs on the three datasets in Table 3, which indicates that GPT-4 has the most robustness.
Among the datasets, LLMs have the lowest gen- eration success rate for XWinggrd,, which is more challenging. XWinograd requires both answers to be from the generated sentence, with only one pronoun being replaced. In addition, we observed pronoun inconsistency in the generated XWinograd data. Despite the requirement for interchangeable pronouns in the options, models frequently fail to comply. For example, "The dog bit the mailman because entered the yard." is generated by Chat- GPT with the options "The dog"" "the mailman", however, " in the sentence cannot be replaced by the same pronoun for the given two options, hence may make the task easier and the example is con- sidered suboptimal. We keep those instances in the dataset and discuss further in $6.1
3.2 Instructions and Responses
We utilise LLMs to generate synthetic examples for all datasets by prompting them. We construct instructions using the descriptions from the dataset papers reference and provide LLMs with some examples, randomly sampled from the train (+vali- dation) split of the original dataset, then ask LLMs to generate similar data points. We experiment with various instructions and evaluate the synthe- sised data on a smaller scale, update the instruc- tions based on the errors, and then choose the best instruction to generate the final datasets.
The final instructions and responses are in Ta- ble 2. Our data generation process comprises the following key steps: (1) We establish the desired total number of examples to generate. This quantity can be determined by various factors such as bud- get constraints, a fixed ratio concerning the original dataset, etc. (2) We proceed to generate examples through the following iterative process: (a) To en- sure diversity, we randomly sample set of examples from the training datasets. (b) We ap- pend these sampled examples to the instructions and prompt the model to generate an additional set of m new examples. (c) Afterwards, we perform post-processing and only add valid and unique ex- amples to the generated set. Typically, the values of n and m are set to 5 to 10.
4 Experimental Setups
We first generate synthetic English examples for XCOPA, XWinograd, and XStoryCloze, with Dolly-v2, Stable Vicuna, ChatGPT, and GPT-4. The size of the final filtered synthesised data for the three datasets is 3.7k, 2K, and 1.7K, respec- tively. We then fine-tune mBERT, XLMR--ase. and XLMR-large with the synthesised data and compare the zero-shot cross-lingual transfer perfor- mance across different languages, where the original validation set in target languages.
For XCOPA, we additionally experiment with generating data points directly in non-English lan- guages, by providing examples in the target lan- guage and specifying the language desired for the generated data (sec Table 2). However, since no examples for cause are included in TH and TR train/validation data (they do appear in the test split), we do not generate XCOPA for the two lan- guages. We use ChatGPT and GPT-4 for multilin- gual synthetic data generation, as both Dolly-v2
We focus on fixed-budget scenario and first generate total of 3-4K data points for each dataset with LLMs. LLMs tend generate fewer samples than requested or inconsistent output in invalid for-
Model details are included in Appendix "https: / ttw.//ggtttttt..c thub. con/lm-e sys/FastChat
"An analysis of the diversity of the generation as well as topic coverage is included in Appendix B.

---

Fine-tuned Model
LLM for Generation
XCOPA
XWINOGRAD
XSTORYCLOZE
ORI
GEN
O+Ga.
OR!.se
2EN
O+G3.8h
ORI 300
GEN1.7
O+Ga
DOLLY-V2 STABLEVICUNA
47.9
53.3 15.4
54.0 16.1
52.9
59.6 16.7
59.3 16.4
65.0
68.7 13.7
68.1 13.1
47.9
$2.9
54.7 f6.8
52.9
53.7
58.5
65.0
64.6 jo.
67.3 (2.3
mBERT
CHATGPT
47.9
55.0 p11
54.1 16.2
52.9
56.0 ra.1
58.3 154
65.0
64.3 10.7
68.3 13.3
GPT-4
47.9
56.4 18.5
57.2 19.3
52.9
54.9 12.0
57.5 14.6
65.0
68.0 13.0
69.8 (18
DOLLY-V2
54.8
58.1
13.3
58.1 13.3
53.5
56.5 r3.0
66.3 f12.8
73.0
75.8 12.8
76.5 r3.5
STABLEVICUNA
54.8
57.6 12.8
59.3 59.3 14.5
53.5
59.0 15.5
66.0 112.5
73.0
69.6 43.4
74.2 (1.2
XLMR-Base
CHAIGPT
54,8
58.2 13.4
59.4
53.5
62.7 19.2
65.9 1124
73.0
67. 15.6
74.5 (1.5
GPT-4
54,8
62.7
17.9
63.0 18.2
53.5
63.3 19.8
66.9 113.4
73.0
74,6 11.6
79.3 16.3
DOLLY-V2
63.0
58.6 14,4
65.0 12.0
80.1
76.9 13.2
83.1 13.0
85.0
84.8 jo.2
86.4 hL.
STABLEVICUNA
63.0
64.4 114
68.7
80.1
68.2 (11.9
82.0 119
85.0
74.6
84,8 1ni2
XLMg-Large GPT-4
CHATGPT
63.0
64.6 11.6
68.1 15.1
80.1
73.2 16.9
83.2 13.1
85.0
77.3 17.7
85.8 10.8
63.0
72.1 19.1
72.2
19.2
80.1
76.4 13.7
83.5 1344
85.0
86.0 11.0
88.4 13.4
Table 4: Comparison of Average Accuracy across all languages for mBERT, XLMR- Base, and XLMR-Large on XCOPA, XStoryCloze, and XWinograd. Training datasets include ORI (original EN data), GEN (LL.M-generated EN data), and O+G (both), with the number of examples used for training indicated by the subscripts. The best results obtained with the same amount of training data are highlighted in bold. Green and red subscripts denote improvement and decline in performance compared to the baseline (OR/). See per language results in Appendix D.
and Stable Vicuna exhibit limitations in effectively generating multilingual text. The size of the multi- lingual synthesised data is ~3.6K in each language.
GPT-4 achieves the highest accuracy in other set- tings. The most significant improvement is shown in X WWinograd with XLMR-Base, where the addi- tion of an extra 2k datapoints leads to an average accuracy enhancement of 12.8 compared to the baseline, across all four LLMs.
We fine-tune models on all datasets multiple- choice tasks by searching best learning rate from (5e-6, 10e-9) and batch size from {8, 16, 32). All the fine-tuning experiments are conducted on a single 40G A A100. For generating data with Dolly- v2 and StableVicuna we use 2x40G A100.
When using only LLM -LM-geerrated data, smaller models like mBERT and XLMR-Bas generally outperform the baseline. However, with XLMR- Large, which achieves stronger baselines. e.g. >80 in XWinograd and XStoryCloze, the accuracy re- mains similar or worse compared to using the original data. GPT-4-generated data demon- strates the best robustness but still experiences a decline in performance in XWinograd when the generated data size similar to the original data. This highlights the challenges of generating data at a human-level quality.
5 Results and Discussion
This section presents the main results of fine-tuned models on the three datasets and compares per- formance with generated data in different LLMs, languages, and scales.
5.1 General Result
Table 4 presents the average accuracy of fine-tuned mBERT, XLMR-Base, and XLMR-Large models across all languages on the three datasets. The models are trained using original data (ORI), dif- ferent LLM-ggnerated data (GEN), as well as a combination of both sources (O+G) in English.
5.2 Multilingual Data Generation
We investigate whether the synthetically generated multilingual dataset outperforms training solely in English. We choose the XCOPA dataset and explore two settings: synthetic multilingual data by asking LLMs to generate responses in the tar- get languages directly and translating the English- generated data to target languages with Google Translate API. We exclude Dolly-v2 and Stable- Vicuna due to their limited effectiveness in gener- ating non-English text. Although GPT-4 exhibits the most promising performance, it is significantly costlier compared to ChatGPT. Therefore, we also
Across different datasets, LLMs, and fine-tuned models, consistent improvements are observed when using both original and LLM-ggeneated data. Among the models, Dolly-v2 performs the best on Xingorad when fine-tuned mBERT, while
*In our preliminary experiments, find that formulating XWinograd as binary text classification results poorly, in line with the observation from Liu et al. (2020) that the task formula- tion is essential to the performance of Winograd.

---

Fine-tuned
LLM
Training data
AVG
EN
ET
HT
ID
IT
SW
TA
VI
7.H
BASELINE
ORI
47.2
53.8
44.2
48.6
47.2
46.2
45.4
48.4
43.6
47,4
GEN - ORd
54.6
59.6
56.4
53.6
53.8
51.4
51.6
50.4
55.0
59.2
CHATGPT
GENxX + ORI
56.8
59.6
58.8
54.6
56.2
61.2
54.6
53.6
52.0
60.2
GENER ORI
58.7
59.6
59.8
58.2
62.8
61.0
52.6
56.8
58.2
59.4
mBERT
GEN - ORI
59.3
72.6
58.8
53.0
62.0
61.0
50.0
54.0
57.6
64.6
GPT-4
GENxx OR!
61.8
72.6
61.2
58.2
62.2
66.4
57.4
53.4
63.0
61.8
GENER" +OR
62.6
72.6
58.6
55.2
6..66
65.4
53.8
62.6
64.6
65.4
BASELINE ORI
55.6
57.6
54.6
50.6
59.6
54.8
55.0
53.44
54.8
59.6
GENsw - ORI
59.8
63.8
51.6
62.6
59.8
51.6
60.4
64.8
CHATGPT
GENxx + ORU
59.9
63.8
60.6
55.0
64.6
59.6
54.6
56.4
59.6
64.8
GENER mM ORI
61.1
63.8
60.0
58.0
6..00
60.8
53.8
60.2
62.6
66.0
XLMR-BaR
GRN - OR
63.6
69.6
63.8
51.2
67.2
62.4
58.4
63.8
66.8
69.4
GPT-4
GENxx + ORT
64.0
69.6
62.2
56.2
68.6
63.8
57.8
61.2
66.8
70.0
G ORT
63.9
69.6
61.6
56.6
68.4
65.2
58.2
60.2
66.0
69.6
BASELINE
ORI
64.4
71.4
62.8
51.4
69.0
65.8
60.6
62.0
69.4
66.8
GRNEN ORT
69.5
76.4
69.8
48.2
76.0
72.8
63.4
67.8
73.4
77.8
CHATGPT
GENxx + ORT
65.2
76.4
62.4
55.2
75.0
62.2
58.2
55.4
66.2
76.2
G ORI
67.0
76.4
60.0
59.6
66.2
66.6
59.0
64.8
74.8
75.6
XLMR-Lurg
GEN - ORI
73.7
84.6
70.4
50.0
80.8
80.2
65.8
72.8
78.4
GPT-4
GENxX + ORI
74.6
84.6
77.0
56.0
82.2
77.0
65.0
73.8
76.2
80.0
G ORI
74.1
84.6
74.2
57.2
82.0
77.4
62.2
75.0
74.4
79.6
Table 5: Accuracy XCOPA. ORT corresponds the original data, GENEN and GENxx represents data generated in English and target languages. Trans denotes tramslations of the English-generatee data. We show languages that are available settings. Improvement and decline performance are represented with green and red shadows.
consider using ChatGPT as contrasting experi- ment under resource-constrainet conditions.
performs worse in those languages than XLMR- Large (Ahuja et al., 2023).
Table shows the results for the languages that are available for all settings, excluding TR and TH (unavailable for LLM-generation, refer to $4), and QU (not supported by the Google Translate API). We can see the impact of the generated data varies across different fine-tuned models and lan- guages, aligning with the findings of Kumar et al. (2022). Training on GPT-4 synthesised data dis- plays consistent improvement across scenarios and languages, except the zero-shot cross-lingual result on HT with ML-a-ree
Translating the English dataset generally shows overall better results than training on the data gener- ated directly the target languages, with the excep- tion of XLMR- Large with GPT-4. For SW, XLMR models fined-tuned with CharGPT-generated data exhibit performance decline in most cases, even when the English-generated data benefits all other languages. This observation suggests that XLMR struggles with SW. In $6. select TA, SW, and the two best languages, ID and ZH, along with EN, for human evaluation.
More fluctuating results can be observed with ChatGPT-generated data. comparison between GENEN ORI and GENxx + ORI indicates that utilising data generated in target languages gener- ally leads to improved performance with GPT-4 generated data, as well as in base models with ChatGPT-generated data. However, for XLMR- Large, employing ChatGPT-gemerated data in tar- get languages mostly yields negative outcomes. In languages such as TA and VI, training on gener- ated data in the target languages results in more performance degradation compared to zero-shot cross-lingual transfer. This suggests that ChatGPT
Additionally, we conduct experiments adding Target Languages in Validation (TLV). This only results in minor variations in the performance, con- sistent with the findings of Ponti et al. (2020), We include the full results in Table 11 in Appendix D.
5.3 Dataset Scaling Up
We now investigate the impact of training larger scale of generated data on model perfor- mance. We focus on the XCOPA dataset and expand the generated data with ChatGPT (more budget-efficient to 28.6k examples in English. We also compare the results of zero-shot lingual

---

GENEx + ORIEN
GENES
ORIEN
sion in performance with ChaGGPT-generated data, namely TA and SW (see Table 5). In addition the original examples and the generated examples in the target languages, we include 50 examples that are translated from the same English-generated examples (that were selected for annotation).
Model
3.7k
28.6K
3.7K
28.6K
mBERT 54.3
56.0
58.0
60.1
XLMR-Base 60.1
61.8
61.2
61.7
XLMR-Large 69.7
72.4
67.2
71.4
Table 6: Accuracy on XCOPA when scaling up the generated data to over 28K with ChatGPT. We report average results on all XCOPA languages excl. QU, since it not available with the Google Translate API.
To ensure impartiality, all the examples are shuf- fled, and the annotators are not provided with in- formation regarding the source of the examples (human-created, LLM-generated, or translated).
6.1 Text Naturalness
transfer with translating the English-generatel data target languages.
Figure presents the annotation results for XCOPA, averaged from two annotators for each language. For Text Naturalness, we can see that in EN, ID, ZH, and SW, both ChatGPT and GPT-4 achieved higher naturalness than the original dataset. This is particularly prominent in ID, revealing the fluency issue in the original ID data in XCOPA, which is also confirmed by native speaker.
The results in Table 6 demonstrate the positive impact of scaling up the generated data on model performance. Particularly, XLMR- exhibits the most significant improvement.
Furthermore, we conduct experiments on gener- ating data with a fixed ratio of the original datasets and the results are included Appendix C.
Issue with Tamil
6 Human Evaluation
In contrast, the performance of the TA dataset is surprisingly low, with a majority of examples clas- sified as "not understandable." Upon consulting language experts, we have identified several main issues in Tamil, including (1) the insertion of redun- dant words with the same meaning, such as "I will retry to try it again" (2) verb agreement errors, and (3) the presence of uncommon and out-of-context words.
To better evaluate the quality of the generated datasets and compare them with the human-created data, we ask native speakers annotate the multi- lingual data generated by ChatGPT and GPT-4. For each dataset, we first select 50 generated examples in English, and then request two anno- tators to evaluate the examples in two categories: (I) Text Naturalness. The annotators are asked to choose one of the following options for each ex- ample: "the text sounds natural", "the text sounds awkward but understandable", or "the text is not understandable". and (2) Logic Soundness. This category focuses on the commonsense aspect of the examples. The annotators are required to select the most appropriate description from: "the correct op- tion is (clearly) more plausible", "both options are equally plausible", "both options are implausible", "the wrong option is actually more plausible". We only ask the annotators to evaluate the logic if the text is at least understandable.
It is worth noting that generating Tamil us- ing GPT-4 both slow and costly. We suspect that the tokenizer for Tamil, as well as similar languages like Telugu and Kannada, are poorly trained, resulting in unusable generation in those languages. While the low quality of the generated data could explain the significant decline in the per- formance of the XLMRR-Lrre model when trained on ChatGPT-generatee data Tamil, intriguingly. models trained on Tamil data generated by GPT-4 show improvement over the baselines.
To further investigate this issue, we conduct an experiment where we fine-tune the models using only five examples from the TA examples generated by GPT-4 that are identified as natural and sound by the annotators. The improvement mBERT under this setting is 50% of the total improvement seen with the entire 3.6K TA examples. For XLMR-base and XLMR-larree 15% and 3% of the total improve- ment can be observed, respectively. Considering that the estimated number of correct samples in
For X XWinograd, we introduce an additional eval- uation criterion. Annotators are asked to determine whether the two noun phrases in the examples can be replaced by the same pronoun (refer to $3.2). For XCOPA, we extend the annotations to non- English languages, where we choose the two lan- guages that demonstrate the most notable improve- ment, namely ZH and ID, as well as the two lan- guages that exhibit the least improvement or regres-

---

Number Examples
50
Original
ChatGPT-Genz
ChalGPT-Genn
Both options equally plausible options implausible Wrong plausible
EN
ID
ZH
SW
EN
ID
SW
TA
EN
SW Logic Issues (ChatGPT)
ID
ZH
TA
Text Naturainess
Logic Soundness
Number Examples
50
Original
GPT-4-Genxx
GPT-4-Gen
Beth options equally plausible Both options implausible Wrong option more plausible
40
30
20
10-
EN
ID
ZH
SW
EN
ID
SW
EN
SW
Text Naturalness
Logic Soundness
Logic Issues (GPT-4)
Figure I: Human evaluation of 50 random examples from the original XCOPA, ChatGPT (top) and GPT-4 (bottom) generated data in target languages, and translation of English generated data. Examples are annotated by two native speakers each language. The subplots the column show logic issues XCOPA data, where the three bars for each language represent Oringal, Cenxx: and Gon EN (from left to right).
the 3.6k dataset around 360, is plausible that training solely on those examples could raise the accuracy level, or even surpass, what we observe for the entire dataset." An intriguing question that remains to be investigated in future research is why the remaining 3.2k incorrect or unnatural examples do not negatively impact the model's performance.
tends to generate options with more clear-cut dif- ferences in plausibility, mirroring the original data. We note that despite the description/instruciinn that both alternatives could happen, both the original dataset and the data synthesised by GPT-4 tend to present one plausible and one implausible option. For English XWinograd and XstoryCloze, the majority of the examples both original and gener- ated examples are evaluated as natural and logically sound. For X XWinograd, although more than 47 ex- amples are evaluated to exhibit high text quality and follow commonsense logic, only 23 ChatGPT- generated examples fulfil the requirement that both noun phrases should be interchangeable with the same pronoun. GPT-4 examples demonstrate better consistency, with following this rule, whereas all original examples are found satisfactory.
The translated text is typically less natural than the original and generated data (apart from ID due to issues in the original data). This result affirms that LLMs generally excel in generating fluent text for the languages it supports.
6.2 Logic Soundness
terms of logic soundness, ChatGPT falls short compared to the original dataset. We further illus- trate the categorised issues in the last column of the plots in Figure 1. We can see that for ChatGPT, the majority of the examples are labelled as "both options are equally plausible", only SW has more problematic examples with "the wrong option is actually more plausible"" We suspect that this issue arises from the instruction provided (taken from the description of the original COPA dataset), which states that "both options could be plausible, but one is more plausible." In some cases, ChatGPT generates two choices that are excessively similar in terms of plausibility. On the other hand, GPT-4
Conclusions
This paper explores the effectiveness of utilis- ing LLMs for data augmentation in cross-lingual datasets with limited training data. We specifically focus on commonsense reasoning tasks that are challenging for data synthesis. Our experiments including four LLMs for data generation threc datasets, showcase enhanced cross-lingual zero- shot transfer on smaller fine-tuned task-specific lan- guage models. However, the impact varies across different datasets and languages. Notably, larger models such as XLMR XLMRR-arge, which have higher
We could not conduct this experiment as the entire dataset was not manually lahelled.

---

baselines, demonstrate more difficulty in achieving performance improvements with LLM-ggnerated data, Among the four LLMs, GPT-4-generated data exhibits mostly consistent superior performance.
study demonstrate the potential benefits of leverag- ing LLMs for multilingual dataset augmentation.
Ethical Consideration
Expanding data generation directly in target lan- guages also shows general improvements com- pared to cross-lingual zero-shot with the English- generated data. Human evaluation of the synthe- sised multilingual dataset shows that the ChatGPT and GPT-4 generated data demonstrate high natu- ralness in most languages, even surpassing the orig- inal data. However, in certain languages like TA, both models fail to generate natural text. Addition- ally, when assessing the logical soundness of the dataset, examples synthesised by ChatGPT reveal notable inconsistencies regarding more plausible options compared to the original human-created data. In contrast, GPT-4 exhibits a performance on par with numan-written data.
Synthetic data generation with LLMs, especially multilingual data, should be approached with sensi- tivity and respect, as it reflects the linguistic, social, and cultural identity of a multilingual community. Since LLMs are trained web data, they may encode biases perpetuating stereotypes, discrimi- nation, or marginalisation of specific languages or communities. Therefore, collaboration with lin- guists language experts, and community represen- tatives is necessary to avoid the unintentional per- petuation of stereotypes and cultural insensitivity.
References
David Ifcoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D' D'suzza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijh- wani, Sebastian Ruder. et al. 2021. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Computational Linguistics, 9:1116-1131.
In conclusion, leveraging LLMs for data aug- mentation shows promise. However, the choice of LLM used for data generation significantly influ- ences the quality of the resulting data, as well as its applicability to the language under consideration. circumstances where a more advanced model such as GPT-4 cannot be accessed, other models can be utilised, though this might result in perfor- mance difficulties in certain non-English languages challenge that also exists for GPT-4 and con- cerns regarding logical coherence. A compelling direction for future rescarch could involve explor- ing the efficacy of more recent instruction-tuned or aligned open-source LLMs, such as LLaMA (Touvron et al., 2023b) or TÜLU (Wu et al., 2023), in enhancing data augmentation.
Kabir Abuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, al. 2023. Mega: Multilingual evaluation of generative al. arXiv preprint arXiv:2303. 12522.
Mikel Artetxe, Sehastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 4623-4637, Online. Association for Computational Linguistics.
Mas- Mikel Artetxe and Holger Schwenk. 2019.
sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Trunsactions of the Association for Computational Linguistics, 7:597-610.
Limitations
We have identified the following limitations in this work: (1)While LLMs, especially GPT-4, exhibit promising results in the context of multilingual commonsense data augmentation, they may en- counter challenges when applied to extremely low- resource languages. (2) In order to achieve opti- mal performance, few-shot examples in the target language are still necessary for generating new ex- amples. However, acquiring such examples may not always be feasible for all languages of inter- est. (3) The usage of closed models like GPT-4 is limited by licensing restrictions, and the results ob- tained from these models may not be reproducible. Nonetheless, the experiments conducted in this
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference volume 202 of Proceedings of Machine Learning Research, pages 2397-2430. PMLR.
Machine Learning.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210. 1446..

---

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Grandee Lee, Xianghu Yue, and Haizhou Li. 2019. Lin- guistically Motivated Parallel Data Augmentation for Code-Switch Language Modeling. In Proc. Inter- speech 2019, pages 3730-3734.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams. Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen. Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettle- moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019-9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang. Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, et al. 2023. Chataug: Lever- aging chatgpt for text data augmentation. arXiv preprint 2rXiv:2302. 12.302113000 13007.
Yu-Hsiang Lin, Chian-Yu Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junx- ian He, Zhisong Zhang. Xuezhe Ma, Antonios Anas- tasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learn- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125-3135 313,, Florence, Italy. Association for Compu- tational Linguistics.
Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Haokun Liu, William Huang. Dhara Mungra, and Samuel Bowman. 2020. Precise task formaliza- tion matters in Winograd schema evaluations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8275-8228 8280, Online. Association for Computa- tional Linguisties.
Kuan-Hao Huang. Wasi Ahmad, Nanyun Peng, and Kai- Wei Chang. 2021. Improving zero-shot cross-lingual transfer learning via robust training. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 1684-1697, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 239-849, San Diego, California. Association for Computational Linguis- tics.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282 6293, Online. Association for Computational Linguistics.
Shanu Kumar, Sandipan Dandapat, and Monojit Choud- hury. 2022. "diversity and uncertainty in moderation" are the key to data selection for multilingual few-shot transfer. In Findings of the Asxociation for Computa- tional Linguisties: NAACL 2022. pages 1042-1055, Seattle, United States. Association for Computational Linguistics.
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai- ley Schoelkopf, Xiangru Tang, Dragomir Radev. Alham Fikri Aji. Khalid Almubarak, Samuel Al- banie, Zaid Alyafcai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generaliza- tion through multitask finetuning. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15991-16111, Toronto, Canada, Association for Computational Linguistics.
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero- shro-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1483-4499 On- line. Association for Computational Linguistics.

---

Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020. Zero-shot cross-lingual transfer with meta learning. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4547-4562, Online. Association for Computational Linguisties.
Cross-Linguse Transfer in Commonsense Reasoning. In Findings of the Association for Computational Linguisties: ACI-IJCNLLP 2021, pages 3534-3546, Online. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and effi- cient foundation language models. arXiv preprint UrXVv:2302.13771.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulié, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal common- sense reasoning. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2222222 2376, Online. As- sociation for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 0rXiv:2307.099 09288.
Edoardo Maria Ponti, Helen O' Horan, Yevgeni Berzak, Ivan Vulié, Roi Reichart, Thierry Poibeau, Ekate- rina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on ty- pological linguistics for natural language processing. Computational Linguisties, (5(3):55-660)
Yizhong Wang. Yeganeh Kordi. Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. 2023. Self-instruct: Aligning language models with self- self-generated instructions. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). pages 33484-13508 Toronto, Canada. Association for Computational Linguistics.
Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018 Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1543-1553, Melbourne, Aus- tralia. Association for Computational Linguistics.
Chenxi Whitehouse, Fenia Christopoulou, and Igna- cio lacobacci. 2022. EntityCS: Improving zero-shot cross- gross-lingual transfer with entity- entity-centric code switch- ing. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 6698-677 14, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawi- jaya, Rahmad Mahendra, Fajri Koto, Ade Romad- hony, Kemal Kurniawan, David Moeljadi, Radi- tyo Eko Prasojo, Pascale Fung, Timothy Baldwin. Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023. NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages. In Proceedings of the 17th Conference of the European Chopter of the A Association for Computational Linguistics, pages 315-834, Dubrovnik, Croatia. Association for Com- putational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, Ei- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 2rXhv:2211.051000
Anirudh Srinivasan, Gauri Kholkar, Rahul Kejriwal, Tanuja Ganu, Sandipan Dandapat, Sunayana Sitaram, Balakrishnan Santhanam, Somak Aditya, Kalika Bali, and Monojit Choudhury. 2022. Litmus predictor: An ai assistant for building reliable, high-performing and fair multilingual nip systems. Proceedings of the AAAI Conference on Artificial Intelligence. 36(11):13227-1322
Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2019. Code-switched language models using neural based synthetic data from par- allel sentences. In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 271-280, Hong Kong, China. Asso- ciation for Computational Linguistics.
Ishan Tarunesh, Syamantak Kumar, and Preethi Jyothi. 2021. From machine translation to code-switching: Generating high-quality code-switched text. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 11th Inter- national Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 3154-3169 Online. Association for Computational Linguistics.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannanch Hajishirzi. 2023. Fine-grained human feedback gives better rewards for language model training. arXiv preprint 2rXiv:2306.01693.
Alexey Tikhonov and Max Ryabinin. 2021. It's All in the Heads: Using Attention Heads as a Baseline for
Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020. Predicting

---

performance for natural language processing tasks. Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 8625- 8646, Online. Association for Computational Lin- guistics.
Model
Ratio
XCOPA
XWingrad
XStoryCloze
1x
64.0
50.2
74,6
2x
64.8
51.9
76.8
MBERT
5x
68.0
57.1
80.6
10x
69.8
65.7
80.3
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 202/ Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-496 On- line. Association for Computational Linguistics.
1x
58.0
45.9
70.7
2
59.0
53.7
79.7
XLMR-BAL -Bse
5
63.0
67.8
81.9
10x
65.8
71.2
84.1
1
56.0
78.1
81.1
2
61.2
79.8
90.9
XLMR-Large
5x
81.4
82.0
89.9
10x
85.2
82.8
91.9
A Model Details
Table 7: Performance English test examples training on GPT-4-generated English data and the original data. Original data points selected from the three datasets set to 200. 1x corresponds to using only the original data, 2x means using 200 original data and 200 gener- ated data.
The open-source models used in the experiments
are as follows:
mBERT:
https://hugginngaace. co/ bert-baseemmtliinnggaa-nncsss
XLMR-base: https://huggingfacc.co xlm-roberta-base
generate longer sentences, especially for the end- ing sentences, whereas in the original dataset, they tend to be the shortest among all sentences.
https://huggingfacc.cmm co/ xlm-roberta-large
XLMR-large:
C Fixed Ratio Data Augmentation
Dolly-v2: https://huggingface. co/ databbiccks/dolly--21111
We experiment with generating data with fixed ratio of the original datasets. Specifically, we com- pare training with the original English data (200 randomly selected examples) and augment it with different quantities of English examples generated by GPT-4, where we include original training in- stances in all cases.
Vtauua https://hugginggaace co/ CarperAi/staallleiicnne--1---e
Sentences and Event Diversity of ChatGPT-generatee StoryCloze Data
The results in Table 7 showcase the performance on English test examples when fine-tuning mBERT and XLMR models with training data sizes that are Ix, 2x, 5x, and 10x the size of the orig- inal dataset. We can see that performance con- sistently improves as we increase the amount of generated data except XStoryCloze, which has the highest baselines, echoing the previous findings. The relative performance gain is generally more pronounced when increasing the data from 2x 5x for the other two datasets.
As the StoryCloze dataset contains more sentences and has richer content, we follow the analysis of the ROC story and further compare the stylistic features in terms of sentence length, and the most frequent events 10 generated by ChatGPT with the original data. This helps us to determine whether ChatGPT-generatte data can capture the corpus dis- tribution by randomly sampling examples from the dataset in the instructions.
In Figure 2, we present the results of compar- ing the generated data points with the original 300 train set used as few-shot examples in the gener- ation instructions. We can see that 23 of the 30 most frequent events in the original dataset can also be found in the 30 most frequent events of the ChatGPT-gennratee data. Regarding the sen- tence length, we observe that ChatGPT tends to
D Additional Results
This section includes the following additional re- sults: Table 8, Table 9, and Table 10 show gen- erated data in English with different LLMs on XCOPA, XWinograd, and XStoryCloze. Table 11 and Table 12 show the full result on XCOPA with ChatGPT and GPT-4
Here we follow Mostafazadeh et al. (2016) where an event counted as any hyponym of "event" or "process" in WordNet.

---

Original
ChatGPT
walk
crive
take
0.11
[ill
decide
take touch
design
think
0.56
fa
come
0.32
beget
0.
rca lize
Number of Words
work
09
find
determine
0.46
2.24
want
15
try
find
suffer
start
make
start
sp
give
interpret
went
count
buy
do
work
forget
interpret
detemine
play
gloe play
sent3 correctwrong Sentences
distinguish
Original StaryClaze
ChatGPT Generated StoryClaze
Figure 2: Comparison between the 30 most frequent events and the lengths of the sentences the original and the ChatGPT-generaate English StoryCloze dataset.
Fine-tuned
Train Data LLM
AVG
EN
HT
IT
QU
SW
TR
ZH
Dolly-v2
54.0
63.4
52.0
52.2
54.0
53.8
47.6
48.6
53.4
53.4
52.8
50.4
58.2
StableVicuna
53.5
62.4
51.6
49.2
55.8
55.8
50.0
50.2
50.2
52.6
51.0
50.4
56.0
GEN
CharGPT
56.0
64.8
54.8
52.6
57.4
49.8
48.4
55.6
52.8
53.2
53.0
59.0
GPT-4
58.2
69.2
59.2
54.0
60.6
59.2
50.8
48.2
55.0
48.2
53.8
57.6
61.0
MBERT
Dolly-v2
54.4
53.2
56.4
52.4
50.4
49.8
58.8
StubleVicuna
55.6
65.2
53.4
50.4
59.0
60.0
51.6
50.4
49.4
52.0
52.4
54.0
58.2
GEN-ORT
ChaiGPT
54.6 59.6
56.4
53.6
53.8
51.4
51.4
51.6
50.4
52.6
54.0
55.0
59.2
GPT-4
59.3
72.6
53.0
61.0
53.0
54.0
52.0
64.6
Dolly-v2
59,0 64.4
58.8
52.8
60.8
61.0
50.8
55.6
60.4
58.0
57.2
58.6
59.0
StableVicuna
58.5
60.4
59.4
53.6
60.8
56.8
56.0
61.2
60.4
54.8
59.6
58.6
GEN
CharGPT
58.8
62.4
56.4
52.4
61.4
58.6
52.2
52.0
63.4
61.2
56.4
62.8
GPT-4
63.6
67.0
62.4
52.0
68.6
62.6
51.8
58.6
65.4
64.8
63.2
66.6
69.6
XLMR- Bass
Dolly-v2
52.2
58.4
$2.4
57.4
58.4
59.8
StubleVicuna
61.1
65.0
62.4
49.4
64.2
62.4
46.2
60.4
59.6
58.0
58.0
63.0
63.4
GEN-ORI
ChatGPT
59.8
63.8
61.6
51.6
62.6
59.8
51.2
51.6
60.4
61.6
61.8
64.8
62.0
GPT-4
63.6
69.6
63.8
51.2
67.2
62.4
52.6
58.4
63.8
66.0
64.2
66.8
69.4
Dolly-v2
59.6 62.4
58.6
49.6
64.8
59.2
50.6
56.8
60.8
58.8
57.0
61.0
63.0
StableVicuna
65.7
71.4
66.2
50.4
71.4
70.2
50.0
60.0
64.0
63.6
68.0
68.2
69.8
GEN
ChaiGPT
71.2
51.6
66.6
58.8
66.0
69.0
68.8
GPT-4
73.6
83.2
71.2
52.0
81.2
78.2
$1.0
62.2
76.6
77.4
75.0
78.4
79.0
XI. XL.MR-I. -argg
Dolly-v2
66.4
62.8
53.0
70.4
61.6
65.6
66.2
69.6
67.6
70.6
StahleVicuna
69.9
76.0
69.8
51.2
75.0
74.2
51.2
64.4
70.2
71.6
72.2
72.6
75.4
GEN+ORI
ChatGPT
69.5
76.4
69.8
48.2
76.0
72.8
50.8
63.4
67.8
70.8
70.2
73.4
77.8
GPT-4
73.7 84.6
70.4
50.0
80.8
80.2
51.8
65.8
72.8
76.0
74.8
78.4
80.4
Table Accuracy XCOPA with English generated data from different LLMss.

---

Fine-tuned
Training data
LLM
AVG
EN
FR
JA
PT
RU
Z.H
Dolly-v2
56,47
71.24
53.01
$2.45
53.23
54.92
53.97
StableVieuna
53.73
54.94
56.63
50.26
50.57
52.06
57.94
GEN
ChatGPT
56,00
54.94
54.22
54.01
52.09
55.87
64.88
GPT-4
54.90
56.22
56.63
52.55
51.71
52.38
59.92
MBERT
Dolly-v2
59.32
71.24
57.83
53.81
56.65
59.05
57.34
StableVicuna
58.46
57.94
63.86
53.81
57.41
58.41
59.33
GEN+ORI
ChatGPT
58.26
56.65
53.60
56.27
60.00
56.75
GPT-4
57.48
53.65
62.65
54.43
55.89
57.14
61.11
Dolly-v2
59.63
71.24
57.83
55.79
$7.03
57.78
58.13
StableVicuna
58.95
60.09
55.42
57.35
52.47
58.73
69.64
GEN
ChatGPT
62.69
69.10
60.24
61.42
57.03
61.27
67.06
GPT-4
63.32
69.10
61.45
61.52
56.65
60.95
70.24
XLMR-Base
Dolly-v2
66.33
75.54
63.86
65.80
64.26
62.86
65.67
StableVieuma
65.97
64.38
66.27
67.15
63.88
65.71
68.45
GEN+ORI
ChatGPT
65.94
65.24
60.24
68.93
70.72
62.86
67.66
GPT-4
66.88
68.24
67.47
66.94
63.88
63.49
71.23
Dolly-v2
76.86
87.55
81.02
76.43
74.29
74.40
StableVicuna
68.22
74.25
63.86
68.20
66.16
63.81
73.02
GEN
ChatGPT
73.20
81.97
66.27
73.10
66.92
72.38
78.57
GPT-4
76.37
81,55
74.70
75.91
71.86
75.24
78.97
XLMR-Large
Dolly-v2
83.10
90.56
79.52
85.19
84.03
80.95
78.37
VtableVicuna
82.02
83.26
80.72
83.84
86.31
82.22
75.79
GEN+ORI
ChatGPT
83.22
85.84
80.72
87.38
85.93
80.95
78.50
GPT-4
83.52
85.41
81.93
85.92
86.69
80.63
80.56
Table Accuracy on XWinograd with Fnnlishh generated data from different L.L.Ms.
Fine-tuned
Training data
LLM
AVG
AR
TE
SW
EU
MY
Dolly-v2
68.7
78.8
71.3
73.6
74,2
67.4
66.9
69.0
65.0
60.9
66.8
62.0
StahleVicuna
64.6
71.4
66.8
68.8
68.1
64.3
63.6
66.1
61.2
58.6
63.6
58.4
GEN
ChatGPT
66.4
68.1
68.0
64.6
64.5
62.3
58.4
GPT-4
75.5
70.8
73.3
70.4
67.6
68.2
69.6
63.1
62.3
65.4
62.2
MBERT
Dolly-v2
68.1
75.7
71.2
72.4
73.2
66.4
67.1
68.9
64.5
61.4
67.1
61.0
StableVicuna
67.3
77.0
71.0
70.2
71.4
67.2
66.5
68.4
62.4
60.5
64.3
61.4
GRN+ORT
ChatGPT
76.4
68.5
72.9
73.0
66.3
68.6
71.1
67.4
63.4
GPT4
69.8
79.5
73.1
75.3
73.4
68.1
69.8
71.9
64.1
62.0
68.9
61.6
Dolly-v2
75.8
81.4
79.2
80.3
78.0
73.6
74.7
80.7
73.0
68.8
72.2
71.7
StableVicuna
72.3
71.1
71.5
70.4
68.3
70.4
68.0
67.7
GEN
ChaiGPT
69.7
68.9
68.5
68.7
68.2
68.7
66.6
GPT-4
74.6
78.2
78.0
78.1
77.0
73.5
75.7
77.6
71.7
68.4
73.6
69.2
MRM-RBase
Dolly-v2
76.5
81.5
80.0
80.5
79.4
75.1
75.0
79.6
71.5
72.3
72.6
StableVieuna
79.2
77.4
77.8
76.4
74.0
71.7
69.6
GEN+ORI
CharGPT
74.5
78.0
76.6
78.8
76.2
72.9
73.9
78.9
71.5
69.6
72.3
71.0
GPT4
79.3
85.4
83.2
82.6
83.0
78.0
79.9
82.7
75.9
72.9
74.9
74.3
Dolly-v2
84.8
87.4
87.3
87.8
86.6
83.0
84.4
87.1
84.1
81.0
82.9
81.4
StableVicuna
74.6
76.7
75.9
77.4
76.2
72.9
74.5
76.2
74.3
70.8
73.5
72.5
GEN
ChalGPT
77.3
78.6
79.9
78.0
77.9
75.8
77.4
78.0
76.4
73.5
77.1
77.7
GPT4
86.0
88.5
88.2
88.2
88.0
84,9
85.7
87.8
83.7
81.3
85.6
84.3
XI.MReL
Dolly-v2
86.4
89.2
87.2
89.5
85.2
85.0
83.0
85.7
83.8
StableVicuna
84.8
88.4
87.6
87.8
86.6
82.9
83.3
87.4
83.7
81.3
83.7
80.0
GEN+ORI
ChatGPT
85.8
88.5
88.0
88.3
87.3
83.7
85.9
87.2
83.7
81.6
85.4
83.8
GPT-4
88.4
92.3
91.5
91.5
90.5
86.4
88.4
91.1
84.8
83.1
87.4
85.2
Table 10: Accuracy XStoryCloze with English generated data from different LLMs.

---

Model
Training Data
IDatal
AVG
EN
QU
OR (BASBLINE)
400
47.2
53.8
44.2
48.6
47.2
46.2
50.6
45.4
48.4
49.8
49.8
43.6
47.4
GENEN
3.7k
56.0
64.8
54.8
52.6
58.0
57.4
49.8
48.4
55.6
52.8
53.2
53.0
59.0
GENEN +OR
4.1k
54.6
59.6
56.4
53.6
53.8
51.4
51.4
51.6
50.4
52.6
54.0
55.0
59.2
GENEN ORU (TLV)
4.1k
57.6
68.0
55.4
54.0
61.2
59.8
51.8
51.2
55.8
54.4
52.2
53.4
59.2
GENEN
28.6k
57.2
66.2
55.8
50.8
58.6
58.2
53.2
51.2
57.2
53.2
52.0
56.0
61.0
GENEN
29k
57.0
66.6
51.4
52.4
53.6
53.2
54.8
62.8
GENEs (TL)
29k
57.0
66.6
51.4
59.2
52.4
53.6
53.2
54.8
GRNxX
3.6k/lang
54.6
GENxX ORI
56.8
59.6
54.6
56.2
61.2
53.6
53.6
52.0
60.2
GENER" + OR!
4k
58.7
59.6
59.8
59.8
62.8
61.0
52.6
56.8
53.4
56.2
58.2
59.4
GENER + ORI
29k/lang
60.6
66.6
61.8
57.8
60.8
62.2
53.2
58.4
53.2
63.0
60.6
63.8
OR (BASBLINE)
400
55.6
57.6
54.6
50.6
59.6
54.8
46,0
55.0
53.4
56.2
55.2
54.8
59.6
GENES
3.7k
58.8
62.4
56.4
52.4
61.4
58.6
52.2
52.0
63.4
61.2
56.4
S9.6
62.8
GENEN (TL))
4.1k
59.8
63.8
61.6
51.6
62.6
59.8
51.2
51.6
60.4
61.6
61.8
64.8
62.0
4.1k
60.7
63.2
51.4
64.8
61.2
51.2
53.6
62.6
63.0
58.2
61.0
66.6
GENeN
56.0
53.0
60.0
XLMR-BASE
GENEN +ORE
29k
62.1
64.6
50.6
66.8
63.6
48.0
65.8
63.6
63.2
66.8
GENEN OR (TL)
29k
60.9
66.4
61.8
49.8
66.2
59.8
54.6
53.4
62.4
58.2
62.8
GENxX
3.6k/lang
58.8
62.4
57.0
55.6
61.4
59.0
55.6
54.4
56.8
60.6
62.0
GENxX ORR
4k
59.9
63.8
60.6
55.0
64.6
59.6
52.6
54.6
56.4
59.6
64.8
GENÉR" ORI
4k
61.1
63.8
60.0
58.0
65.0
60.8
53.8
60.2
66.2
56.6
62.6
66.0
GENTE + ORI
29k/lang
62.2
64.6
63.2
57.2
64.8
61.2
$5.0
61.2
59.2
59.5
64.2
68.4
(BASELINE)
400
64.4
71.4
51.4
69.0
65.8
$2.0
60.6
62.0
64.0
61.2
69.4
66.8
3.7k
65.2
71.2
51.6
70.8
$1.0
66.0
68.2
68.8
68.8
GENew +ORE
76.4
GENeN ORI (TL)
4.1k
71.9
80.6
50.8
78.6
77.2
51.8
69.2
71.2
72.8
77.2
GENEN
28.6k
71.8
80.6
74.4
51.0
78.4
75.2
51.2
63.4
69.8
70.6
69.8
75.6
77.4
XIMR-LARGE GENeN ORI
29k
72.4
81.0
73.8
54.4
80.2
75.2
48.8
61.4
70.4
73.8
70.4
75.6
79.8
GENEs + ORU (TLV)
29k
72.4
81.0
73.8
54.4
80.2
75.2
48.8
61.0
70.4
73.8
70.4
75.6
79.8
GENxx
3.6k/lang
63.4
71.2
62.6
54.2
71.0
65.8
49.4
53.8
56.4
64.0
71.6
GENxx ORI
4k
65.2
76.4
62.4
55.2
75.0
62.2
$4.0
58.2
55.4
66.2
76.2
GENÉS" ORI
4k
67.0
76.4
60.0
59.6
66.2
66.6
59.0
64.8
71.2
65.2
74.8
75.6
GENER" ORI
29k/lang
71.5
81.0
71.8
57.2
79.8
74.4
54.8
71.4
72.6
70.0
77.2
75.6
Table 11: Full results XCOPA (with ChatGPT-genern
data). +TLV corresponds to including the original validation set in all Target Languages the Validation set. Rows are sorted by the number of instances used in training, AVG shows average results for languages that available in all settings (excl. QU, TH, TR).

---

Model
Training Data
AVG
EN
ET
HT
QU
SW
TA
ORI
47.2
53.8
44.2
48.6
47.2
46.2
50.6
45.4
48.4
49.8
49.8
43.6
47.4
GENEN
58.2
59.2
54.0
60.6
50.8
48.2
55.0
48.2
GEN + ORI
59.3
72.6
58.8
53.0
62.0
61.0
53.0
50.0
54.0
48.2
52.0
57.6
64.6
mBERT
GENxX
60.2
69.2
59.4
56.2
60.2
63.8
54.4
55.2
54.0
61.2
62.2
GENxx +
61.8
72.6
61.2
58.2
62.2
66.4
54.4
57.4
53.4
63.0
61.8
GENEN
61.4
69.2
59.2
56.8
65.4
65.2
53.4
56.8
52.6
59.6
61.8
GENEN" ORI
62.6
72.6
58.6
55.2
65.6
65.4
53.8
62.6
53.2
58.8
64.6
65.4
ORI
55.6
57.6
54.6
50.6
59.6
54.8
46.0
55.0
53.4
56.2
55.2
59.6
GENEN
63.6
67.0
62.4
52.0
68.6
62.6
$1.8
58.6
65.4
64.8
63.2
66.6
69.6
GENEN ORT
63.6
69.6
63.8
51.2
67.2
62.4
52.6
58.4
63.8
66.0
64.2
66.8
69.4
XLMR-Bas
GENxX
63.2
60.2
GENxx + ORI
64.0
69.6
62,2
56.2
68.6
63.8
56.8
57.8
61.2
66.8
70.0
GENTN
62.5
67.0
60.0
55.6
66.0
62.4
58.0
60.4
64.4
64.6
64.0
68.8
GENT
63.9
69.6
61.6
56.6
68.4
65.2
58.2
60.2
68.0
62.6
69.6
ORI
64.4
71.4
62.8
51.4
69.0
65.8
52.0
60.6
62.0
64.0
61.2
69.4
66.8
GENEN
73.6
83.2
71.2
52.0
81.2
78.2
$1.0
62.2
76.6
77.4
75.0
78.4
79.0
GENeN ORI
73.7
84.6
70.4
50.0
80.8
80.2
51.8
65.8
72.8
76.0
74.8
78.4
80.4
XLMR-Large GENxx
72.8
83.2
75.2
55.2
78.4
76.0
52.4
63.0
68.2
77.8
78.6
GENxx ORI
74.6
84.6
77.0
56.0
82.2
77.0
56.0
65.0
73.8
76.2
80.0
GENEX
71.0
72.4
55.6
79.4
60.6
67.8
GENE + ORI
74.1
84.6
74.2
57.2
82.0
77.4
62.2
75.0
75.2
72.8
74.4
79.6
Table 12: Accuracy on XCOPA. GENEN and GENxx represents 3.7K and 3.6K data in English and target languages generated by GPT-4. AVG shows average results for languages that are available all settings (excl. QU, TH, TR).


Reviews:
[{"review_id": "yecuuLWASQ", "reviewer": "Reviewer_4SYH", "summary": "", "questions": "In table 4, the number of generated examples is independent of the number of original examples. Did you consider running experiments with a fixed ratio of synthetic examples (e.g. exactly the same size as original examples, 2x, 3x) or with a fixed \"total\" number of examples (e.g. 1000, 2000...)?\n\nWhen analyzing results on TA, you find that only few examples can help improve the performance of the model to a degree. Did you run similar examples with other languages using a few (e.g., 5, 10, 50) examples?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging feedback, questions and suggestions. We would like to address the questions as follows:\n\nWe initially focus on fixed-budget experiments (e.g., generating 3K examples). We acknowledge the valuable suggestion of comparing experiments with a fixed data ratio. To this end, we've conducted experiments on GPT-4 generated English XCOPA data, inclusive of the original 400 train data. Our plan is to extend these experiments to cover additional datasets and expand findings in the camera-ready version.\n\n| XCOPA-EN |   1X   |  2X   |  5X   | 10X   |\n|-----------|------|-------|------|-------|\n| Data Size  |   400    | 800| 2000| 4000|\n| mBERT     |   62.6   |  68.4  |  69.2 |  70.4 |\n| XLMR-base  |   57.6   |  59.4  |  66.4 |  69.8 |\n\nThe observation that only a few examples in TA contribute to the performance is derived from the human evaluation quality of TA. We have not run similar experiments on other languages, but it is a great suggestion to study and add to the final version.\n\nWe will also add the quantitative improvement results in the abstract as suggested to strengthen the claim.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- well written paper, with well defined methodology and strong evaluation\n- promising experimental results\n- resources and findings that are useful to the community", "reasons_to_reject": "None, it's a very solid paper.", "questions_to_authors": "In table 4, the number of generated examples is independent of the number of original examples. Did you consider running experiments with a fixed ratio of synthetic examples (e.g. exactly the same size as original examples, 2x, 3x) or with a fixed \"total\" number of examples (e.g. 1000, 2000...)?\n\nWhen analyzing results on TA, you find that only few examples can help improve the performance of the model to a degree. Did you run similar examples with other languages using a few (e.g., 5, 10, 50) examples?", "paper_topic_and_main_contributions": "The paper explores the use of LLMS to generate additional training data for common sense reasoning in cross-lingual setting. The authors compare different LLMs, tasks, and languages. They experiment both with generating data in English and in the target language. They also present a human evaluation of the quality of the target-language generated data.\n\nThe main contributions of the paper are:\n\n- empirical analysis of the impact that synthetic data has on the performance of downstream classifiers\n- a systematic comparison between different models, datasets, and languages\n- the resources created as part of the research", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I suggest that you make stronger claims in the abstract/introduction and bring some quantitative results (e.g. improvement by X % in best case)"}, "extra_text": {}}, {"review_id": "0D3s3Do2fs", "reviewer": "Reviewer_AwiM", "summary": "", "questions": "Question A: What is the value of the \"m\" parameter mentioned in Table 2?\n\nQuestion B: What value of the \"temperature\" hyper-parameter was used in the experiments?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comprehensive feedback and suggestions from the reviewer. We would like to first address the main concern related to the diversity and the data augmentation approach. Our *clarification* on the data generation process is as follows:\n\nOur **data generation process** involves the following steps:\n1. We set a total number of examples we want to generate (usually related to budget), in our case it is around 3K (but it can be a fixed number of data or a fixed ratio w.r.t the original dataset)\n2. Generate examples as follows until we have enough valid examples:\n\n    a. To ensure *diversity*, we **randomly** sample **n** examples (*typically set to 5*) each time from the train sets;\n\n    b. Append the sampled examples to the instruction, and prompt the model to generate **m** (*set to 5-10*) new examples;\n\n    c. Post processing and add the *valid and unique* examples to the generated set.\n\nThe values of *m* and *n* depend on example length and model constraints (input/output length). In our case, we set them to 5-10 depending on the specifics of each dataset.\n\nWe want to clarify that we employ random samples in the prompt each time, in line with the reviewer's suggestion. \nWe acknowledge the need to explicitly state that examples are not generated all at once and to *specify m and n values*. This clarification will be incorporated in the camera-ready version.\n\nBeyond addressing the data augmentation approach, we address additional concerns:\n1. Regarding diversity, a study in the appendix (lines 860-862) compares topic coverage to evaluate diversity through \"randomly sampling n examples from the dataset in the instructions.\" While this isn't emphasised presently, we intend to enhance this description for the camera-ready version.the current version and will improve the description in the camera-ready version.\n2. Number of generated examples. We initially focus on fixed-budget experiments (e.g., generating 3K examples). We acknowledge the valuable suggestion of comparing experiments with a fixed data ratio. To this end, we've conducted experiments on GPT-4 generated English XCOPA data, inclusive of the original 400 train data. Our plan is to extend these experiments to cover additional datasets and expand findings in the camera-ready version.\n\n| XCOPA-EN |   1X   |  2X   |  5X   | 10X   |\n|-----------|------|-------|------|-------|\n| Data Size  |   400    | 800| 2000| 4000|\n| mBERT     |   62.6   |  68.4  |  69.2 |  70.4 |\n| XLMR-base  |   57.6   |  59.4  |  66.4 |  69.8 |\n\n3. Regarding other data augmentation approaches, methods that focus on the selection and filtering of the automatic generation data  can also be applied after our augmentation pipeline; our work aims to explore the potential of data augmentation from LLMs, driving exploration in LLMs and open-source models like LLAMA2. We will expand the comparison and the limitation section accordingly.\n\nWe would also like to answer the **questions**:\n1. We set m and n to 5-10 in our experiments\n2. We use the default temperature (1.0)\n\nWe kindly request the reviewer's consideration to revise the score in light of our provided clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-motivated and studies an important problem of performing data generation in multilingual, low-resource settings for a challenging task of commonsense reasoning. Although the general-purpose LLMs exhibit impressive zero-shot learning capabilities, smaller task-specific models can often be more accurate, cost-effective, and practical, given that enough training data is available. Therefore, improving the accuracy of task-specific models by generating synthetic data for fine-tuning is still an important research topic.", "reasons_to_reject": "One of the weaknesses of the presented approach is that it does not provide a mechanism to control the diversity of the synthesized examples. Although the authors claim that their method generates diverse examples (lines 61-62), it has neither been supported empirically nor discussed in the paper. Notably, the results presented in Section 5.3 suggest that the benefits of generating larger amounts of data are rather limited, possibly due to the low diversity of the synthesized examples.\n\nIn addition, the clarity of the paper needs to be improved. Specifically, the authors generated 2-4k new data points to fine-tune the models for each data set. It is unclear whether all instances were synthesized in one inference step or the models were prompted multiple times for each data set. Moreover, given that the number of valid examples obtained with different models differs substantially (see the success rates in Table 3), it needs to be clarified why an equal number of synthesized data points is reported for each LLM used for generation. Presumably, the models were prompted multiple times until the required number of examples was reached, but the paper needs to explain this. Generating small batches of examples by prompting the model multiple times, each time using a different set of seed examples, might also improve the diversity and, in turn, the quality of the synthesized data.\n\nAnother point is that the choice of the number of synthesized samples per data set should be discussed in the paper. The number of examples is neither fixed nor proportional to the size of the corresponding original data set. For XWinograd, which has the largest training set out of the examined benchmarks, additional 2k examples are generated, which doubles its size. In contrast, the size of XCOPA and XStoryCloze increases 10 and  6.7 times, respectively.\n\nFinally, the authors need to discuss their method's effectiveness in comparison with other data augmentation approaches. Note that after deduplication and discarding invalid instances, only up to 42% of examples are retained (in the case of open-access models). The usage of closed generative LLMs might also not be cost-effective. The reader would benefit from a more in-depth discussion of the advantages and disadvantages of the proposed method. The limitations section needs to be updated accordingly.", "questions_to_authors": "Question A: What is the value of the \"m\" parameter mentioned in Table 2?\n\nQuestion B: What value of the \"temperature\" hyper-parameter was used in the experiments?", "paper_topic_and_main_contributions": "The authors of this paper study the capabilities of large language models (LLMs) in performing data synthesis in cross-lingual, commonsense reasoning data sets. To this end, they leverage the in-context learning abilities of LLMs. Specifically, they construct data set-specific prompts that instruct the model to generate new data points based on a few given examples from the original data set. New data points are generated directly in the target language or English. In the latter case, they are translated to the target language using Google Translate API.\n\nThe authors employ four LLMs (Dolly-v2, StableVicuna, ChatGPT, and GPT-4) to augment three multilingual data sets (XCOPA, XWinograd, and XStoryCloze) and use the synthesized data to fine-tune two pre-trained multilingual transformer models (mBERT and XLMR). As a result, the models trained on augmented data sets exhibit improved accuracy. Overall, the best results are achieved with the data generated by the GPT-4 model. Moreover, training on an English data set translated to the target language performs generally better than using data generated directly in the target language. The authors also perform qualitative evaluation by asking native speakers to assess the generated examples in terms of naturalness and soundness.\n\nIn sum, this work makes the following contributions: 1) It proposes a straightforward LLM-based data synthesis method; 2) It presents the results of an experimental evaluation of the accuracy of two pre-trained multilingual transformer models fine-tuned with the synthesized data in a cross-lingual, commonsense reasoning task and 3) It performs a qualitative human evaluation of the examples generated by the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mGkOBEKrQ4", "reviewer": "Reviewer_7m4W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' detailed feedback and suggestions. We would like to address the concerns as follows:\n\n1. One of the motivations and the main focus of this study is to evaluate the effectiveness of augmentation with synthetic data generated from large/better models on smaller models (line 10, 43). We believe the impact will be less profound in larger models as we can observe in XLMR-large, owing to their superior capacity and task baselines. \n2. We acknowledge it is a great suggestion to report the performance of the LLMs used for data augmentation on the tasks, which will also provide an indication of the dataset quality. We will add this in the camera-ready version.\n3. We appreciate the reviewer’s suggestion on future exploring directions and agree that studying the most recent open-source LLMs such as LLAMA 2 would provide great insights on data augmentation. \n4. Our current work focuses on showcasing the potential of data augmentation from different LLMs, however, we agree that studying the mix of multiple models for data augmentation is a great direction to explore.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n1. The paper focuses on an important research problem in NLP: multilingual commonsense reasoning. In the last few years, large language modeling research has primarily focused on English. It's important to see how the benefits from English large language models can be used in other lower resource languages, and this paper does a good job studying this research question on commonsense reasoning tasks.\n\n2. This paper has a comprehensive experimental setup, and experiments with 3 commonsense reasoning datasets, three multilingual models, and experiments with data generated by 3 large language models. The paper has experiments generating English examples (zero-shot cross lingual generalization), translated examples, as well as examples in the language itself. The paper also has some good analysis with human evaluation experiments on the quality of the generated examples.\n\n3. This paper is very well written, and easy to follow.\n\n4. The authors extensively discuss and are upfront about the limitations of their work.\n", "reasons_to_reject": "\n1. I am wondering what is the effect of these data augmentation methods when the base model is made larger / more powerful. The current experiments focus on fairly small multilingual models (mBERT / XLM), which are <500M parameters. It would be interesting to see how these results generalize to larger models like mT5 or BLOOM. This is important because it's possible that these larger models need lesser examples to generalize, and data augmentation will not as useful then.\n\n2. A related point to weakness #1, the paper is missing baseline experiments which use the large language models themselves to perform the task, through few-shot learning or fine-tuning on original data using the GPT3 fine-tuning API (please me know in case I missed these experiments while reading the paper!). Since models like ChatGPT / GPT-4 are effectively used to create training data for downstream fine-tuning, I'm suspecting they can already perform the task pretty well?\n\n3. This has been acknowledged by the authors in their limitations section, but there are several commerical restrictions on using ChatGPT/GPT4 outputs, [1] says \"one may not use output from the Services to develop models that compete with OpenAI\". This may restrict the applicability of the proposed approach for downstream applications. In future versions of the paper, it will be helpful to see how stronger instruction-tuned / aligned open LLMs like LLAMA 2 [2] or TULU [3] can help with the data augmentation. However, note that both these models came out in the last two months (during / after EMNLP submission period), so this is not a valid weakness for the current version of the paper.\n\n4. Finally, in the long term I am not very convinced by using one language model \n\n[1] - https://openai.com/policies/terms-of-use  \n[2] - https://huggingface.co/blog/llama2  \n[3] - https://arxiv.org/abs/2306.04751", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores data augmentation methods for multilingual commonsense reasoning tasks. The key idea in this work is to leverage few-shot prompting methods on large language models (like ChatGPT / GPT4) to generate a lot of task-specific English data. Smaller multilingual models like mBERT are then fine-tuned and evaluated zero-shot on other languages (\"zero-shot crosslingual generalization\").\n\nAcross three datasets, three multilingual models, and four data generating language models, the paper shows that the proposed data augmentation method is quite effective at improving zero-shot crosslingual generalization. The paper also conducts extensive analysis on the quality of the generated data (via human studies), and explores data augmentation using multilingual data as well.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the used of several LLMs (like StableVicuna and GPT-4) like  to generate synthetic data to train a smaller multilingual language models like mBERT and XLM-R. They performed evaluation on several languages and on three tasks: XCOPA, XWinograd and XStoryCloze. 

All the reviewers found this to be interesting and a strong evaluation especially for low-resource settings and the efficiency of smaller models. It's also great the authors compared with machine translated texts which had been done in past works. There a few comments by the reviewers about the diversity of the generated sentences, the use of bigger models, use of more recent LLMs for generation, and some unclear part of the write-up, which the authors have adequately addressed. I believe this paper is a good contribution of LLMs for non-English languages especially in terms of building smaller models with similar or better accuracy than LLMs.
Decision: Accept-Main