Venue: NAACL 2025

# M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in

Large Language Models

Rishabh Maheshwary\({}^{\lx@sectionsign}\), Vikas Yadav\({}^{\lx@sectionsign}\), Hoang Nguyen\({}^{1}\)

**Khyati Mahajan\({}^{\lx@sectionsign}\), Sathwik Tejaswi Madhusudhan\({}^{\lx@sectionsign}\)**

ServiceNow

\({}^{\dagger}\) University of Illinois at Chicago

{rishabh.maheshwary, vikas.jadv} @ servicenow.com

Work done during internship at ServiceNow

###### Abstract

Collecting instruction fine-tuning (IFT) data is a resource and time intensive task, especially in multilingual settings where finding proficient native speakers is challenging. Moreover, traditional data collection is prone to privacy risks, toxicity and lacks scalability. While fully synthetic datasets are a promising alternative, research on their use in multilingual domain is limited as existing approaches still rely on machine translation to improve multilingual performance. To bridge this gap we introduce _M2Lingual_, the first _fully synthetic_, _multi-turn_ multilingual dataset having \(175K\) conversations across \(70\) languages with a balanced mix of high, low and mid-resourced languages. _M2Lingual_ is constructed using a cost-efficient and scalable method that uses our _novel two-step Evol_ prompt taxonomy to transform a small set of human written instructions to complex and challenging conversations. Results across _three_ model families, _six_ baseline datasets and evaluation spanning 31 languages demonstrates the effectiveness of _M2Lingual_ over other datasets.

## 1 Introduction

The recent success of large language models (LLMs) Achiam et al. (2023); Jiang et al. (2024); Touvron et al. (2023); Team et al. (2023) can be largely attributed to the availability of large, diverse, and high quality instruction fine-tuning (IFT) datasets Taori et al. (2023); Chiang et al. (2023); Xu et al. (2023). However, the majority of IFT datasets are in English with very limited coverage for other languages Zhang et al. (2023).

Existing multilingual IFT datasets can be divided into those that require human involvement and those that rely on machine translation (Table 1). The development of human-involved datasets is resource-heavy, often requiring native speakers, which introduces potential for annotator errors, uneven data distribution, and privacy and toxicity concerns Abay et al. (2019); Zhao et al. (2024). These challenges lead to low-complexity conversations Xu et al. (2023) as well. Machine-translated datasets offer less resource-intensive methods to create the data, but suffer from translation artifacts known as _translationese_Bizzoni et al. (2020); Vanmassenhove et al. (2021) that fail to capture linguistic nuances Wang et al. (2023). In conjunction with limited language coverage, overly simple instructions, and unbalanced NLP task representation, most multilingual datasets are not multi-turn, limiting the ability of models to engage beyond single utterances Wei et al. (2023).

Fully synthetic datasets offer a promising solution to address the above concerns. Not only do synthetic datasets address the high cost of data collection, toxicity and privacy concerns, english synthetic datasets like WizardLM, Vicuna, Ultrachat, etc have been proven to significantly enhance the performance of LLMs in English Xu et al. (2023); Chiang et al. (2023); Ding et al. (2023). However, there is a lack of research on synthetic datasets in the multilingual domain that encompass a wide range of languages, NLP tasks, and multi-turn conversations. To address this gap, we present the following contributions:

1. We introduce _M2Lingual_, the first _fully synthetic_, _multi-turn_, and _diverse_ multilingual dataset, containing \(175K\) complex and challenging conversations across \(70+\) languages and \(19\) NLP tasks built with the _Evol_ taxonomy.
2. We construct a _novel_, _two-step Evol_ taxonomy (Figure 2), covering \(19\) NLP tasks, each with \(9\) distinct methods to transform seed instructions to make them more complex and challenging. Additionally, to synthesize multi-turn conversations, we develop \(21\)_Evol_ prompts to increase engagement. This controlled setup _ensures a balances representation of different languages_,

especially low resource languages (Figure 4) which is challenging to achieve in real-world scenarios Przystupa and Abdul-Mageed (2019). The _Evol_ taxonomy enables a fully-synthetic, scalable, and cost-efficient method for constructing enriched multi-turn multilingual conversational IFT dataset which is extendable to any task and language.
3. We provide detailed analyses highlighting the impact of seed instructions, each step of the data enrichment and synthesis process. Additional analysis on low resource languages, content moderation, conversation length, and language distribution, demonstrate the superiority of _M2Lingual_ over other datasets.

## 2 Related Work

Multilingual Instruction Finetuning.Due to the widespread availability of high-resource language pretraining corpora multilingual instruction finetuning has proven to be a cost effective solution for improving performance Ranaldi et al. (2023); Chen et al. (2024); Ustun et al. (2024). Several approaches have been adopted to expand access to multilingual IFT corpora. Notable among these are datasets derived from NLP tasks (e.g., FlanT5, Supernatural Instructions) Chung et al. (2024); Sanh et al. (2021); Wang et al. (2022)

_Human-generated datasets_ such as Aya Singh et al. (2024) and OpenAssistant Kopf et al. (2024) involve humans creating conversation topics, writing questions, and crafting responses. While these datasets are typically high quality, their creation is extremely resource and time intensive. Moreover, finding native speakers for diverse languages is challenging, with potential annotator errors and uneven data distribution. Singh et al. (2024); Gilardi et al. (2023), making it difficult to scale these datasets. _Human-AI generated datasets_ such as LM-Sys Zheng et al. (2023), WildChat Zhao et al. (2024) and ShareGPT RyokoAI (2023) are less resource-intensive than purely human-generated ones, as they involve humans interacting with LLMs to generate conversations. However, they still present challenges, as humans must write instructions and create diverse questions in native languages, a process that remains time-consuming. Additionally, this approach can raise privacy concerns Abay et al. (2019), introduce toxic data Zhao et al. (2024), and result in low-complexity conversations Xu et al. (2023). Finally, _machine-translated datasets_ such as BactrainX Li et al. (2023) offer a more resource efficient method. However, such datasets often suffer from translation artifacts known as _translationese_Bizzoni et al. (2020); Vanmassenhove et al. (2021) that fail to capture linguistic nuances Wang et al. (2023). On the contrary to these, our presented _M2Lingual_ dataset utilizes IFT seeds from native speakers across various languages (section 3.1) and applies task specific mutation in each language (section 3.2), thus maintaining linguistic nuances in respective individual language. _M2Lingual_'s generation pipeline is also completely synthetic (table 1), making it a scalable and an affordable method for multilingual IFT data generation.

Synthetic Datasets.Fully synthetic datasets have emerged as a promising alternative towards addressing constraints with existing data generation methods. Popular English synthetic datasets, such as Alpaca Taori et al. (2023), WizardLM Xu et al. (2023), and Vicuna Chiang et al. (2023), generate new instructions from a small initial set using methods like Self-instruct Wang et al. (2023) or _Evol_Instruct Xu et al. (2023), and have shown strong performance. However, there is limited research on leveraging synthetic d

\begin{table}
\begin{tabular}{l c c c c c c c c c} Dataset & Size & Multi turn? & Langs & Resource Level & Task specific? & General & Translated & Fully \\  & & & Low & High & specific? & instructions? & dataset? & synthetic? \\ \hline Aya Dataset Singh et al. (2024) & 200K IR pairs & ✗ & 70 & 37 (1) & 32 & ✗ & ✓ & ✗ & ✗ \\ MultiAlpaca Wei et al. (2023) & 132K IR pairs & ✗ & 11 & 0 & 11 & ✗ & ✓ & ✗ & ✓ \\ M-Alpaac Chen et al. (2024) & 52K IR pairs & ✗ & 12 & 0 & 12 & ✗ & ✗ & ✓ & ✓ \\ Bactrain-X Li et al. (2023) & 3.4M IR pairs & ✗ & 52 & 15(1) & 36 & ✗ & ✓ & ✓ & ✓ \\ \hline OpenAssistant Köpf et al. (2024) & 10K convs & ✓ & 35 & 3 & 32 & ✗ & ✗ & ✗ & ✗ \\ ShareGIPT RyokoAI (2023) & 94K convs & ✓ & 45 & 4 (2) & 39 & ✓ & ✗ & ✗ & ✗ \\ WildChat Zhao et al. (2024) & 1.04MN convs & ✓ & 74 & 21 (3) & 50 & ✗ & ✗ & ✗ & ✗ \\ \hline _M2Lingual_ & 182K convs & ✓ & 70 & 37 (1) & 32 & ✓ & ✓ & ✗ & ✓ \\ \end{tabular}
\end{table}
Table 1: Comparison of multilingual IFT datasets with _M2Lingual_. The top 4 rows are task based multilingual focused IFT datasets and the bottom 3 rows are datasets collected in the wild. Resource level classification taken from NLLB Costa-jussá et al. (2022). Languages not found in the NLLB table are counted as low, in parentheses.

 

lingual capabilities, with the exception of MultiAlpaca Wei et al. (2023), which uses Self-instruct. This approach has been shown to be susceptible to repetitive and noisy outputs Chen et al. (2023); Ghosh et al. (2024), and suffers from low performance (Tables 12 & 13).

## 3 Methodology

_M2Lingual_ has three main synthesis steps. _Step 1: Seed Selection_ involves the selection of diverse multilingual seeds. _Step 2: Guided Evol_ uses the _Evol_ prompt taxonomy to generate complex instruction and response (IR) pairs and _Step 3: Multiturn Evol_ uses the multi-turn portion of the taxonomy to extend IR pairs to multilingual conversations. Figure 1 captures an overview of each step in _M2Lingual_ synthesis and Figure 2 presents the categories of _Evol_ prompts*.

Footnote †: [https://github.com/lcw99/evolve-instruct/blob/main/evolve.py](https://github.com/lcw99/evolve-instruct/blob/main/evolve.py)

### Seed Selection

To ensure that we select diverse seeds capturing language nuances and covering a variety of NLP tasks, we select seed examples from the Aya dataset and collection as both have a high average approval ratio by human annotators Singh et al. (2024).

**Aya dataset seeds.** Aya dataset has general IR pairs written by native speakers that captures region specific language nuances and cultural contexts. We randomly select \(100\) IR pairs for each of the \(70\) languages, resulting in \(7000\) seed IR pairs.

**Aya collection seeds.** Aya collection covers \(19\) NLP tasks where each task has parallel examples in \(113\) languages. To ensure a proper balance of the number of examples across all languages, we only focus on \(70\) languages and exclude two NLP tasks -- 1) text simplification, as it is already supported by our _Evol_ prompts, and 2) multilingual event entity task, as Aya do not have a consistent format for this task. Finally, for each task in the collection, we randomly sample \(6\) examples per language, resulting in \(6\times 70\times 17=7140\) IR seeds. We select \(6\) random samples per task per language to ensure balanced amount of seed samples from Aya collection when compared to the seeds from Aya dataset. Thus, our final seeds contain \(7000+7140=14140\) IR samples.

### Guided _Evol_

The seed instructions span a variety of NLP tasks but are generally straightforward and overly simplistic. To enhance LLMs' instruction following capabilities, particularly for complex tasks, we apply _Evol-Instruct_Xu et al. (2023) to our selected seed instructions. The _Evol-Instruct_ method uses Evol prompts to transform simple instructions into more intricate ones. However, the generic Evol conditions from the original work+ provide very limited guidance for generating new IR pairs, especially for the \(19\) diverse NLP tasks for which we aim to generate training data. To address this, we

Figure 1: Walk-through for data synthesis of _M2Lingual_. Step 1 is seed selection. In Step 2 for each instruction corresponding task specific _Evol_ prompt taxonomy is used for generating complex evoked instruction. Finally, in Step 3, multi-turn instruction are generated on Step 2 evoled instructions using multi-turn _Evol_ prompt taxonomy.

 

develop a novel _Evol_ prompt taxonomy covering both general instructions and specific NLP tasks, as illustrated in Figure 2.

_Evol_**Prompt Taxonomy.** For general instructions from the Aya dataset, we design \(6\)_Evol_ prompt conditions that enhance multilingual features. We create \(9\) task-specific _Evol_ prompts for each NLP task in the Aya collection to ensure that we tailor _Evol_ conditions for individual tasks. We use GPT-4 [1] to transform the seeds using our _Evol_ prompt taxonomy. These are applied to specific seeds as described below:

* _Aya dataset seeds._ As Aya dataset has general IR pairs, we apply the \(6\)_generic evol prompts_ to each seed example. This results in \(7K\times 6=42K\) instructions which are complex, challenging, and captures all nuances and complexities of languages.
* _Aya collection seeds._ For each seed instruction from one of the \(17\) tasks (top block of Figure 2), we apply its corresponding \(9\)_Evol_s resulting in a total of \(7140\times 9=64260\) instructions from Aya collection.

### Multi-turn _Evol_

The final step involves generating multiple user-assistant turns based on the task- _Evol_ed instructions from the previous phase. Conversations between a user and an AI assistant generally fall into four broad categories: _Follow-up_, _Refinement_, _Expansion_, and _Recollection_[35]. We propose a multi-turn _Evol_ prompt taxonomy with \(21\) distinct dialogue variations (final block labeled as part 2 in Figure 2) that build upon the original generic four categories. Additionally, we ensure that all subsequent instructions are generated in the same language as the initial instruction by explicitly prompting GPT-4. We select all the _Evolved_ instructions from the Aya dataset, and pick a balanced subset of size \(35\)K from Aya collection and generate turns as follows:

1. [leftmargin=*]
2. _User turns._ We use the prompt specified in Appendix A.11 to generate multiple user turns. Specifically, we use the task-_Evolved_ instruction from the previous step (i.e., Step 2), with its language and one of the \(21\) dialogue variations to generate the next user instruction.
3. _Assistant turns._ For all the generated user turns, we generate subsequent responses via GPT-4 using the entire conversation history. To mitigate the impact of topic drift from the long conversations [8], we restrict the total number of turns to \(<=6\).

Post-Hoc Filtering.Upon manual inspection, we find that some IR pairs generated using GPT-4 have repetitive long sequences and n-grams. To mitigate this, we apply a filtering step following [19, 10] to remove IR pairs with frequent n-grams. This filtering is performed after steps \(2\) and \(3\). The final dataset consists of \(75\)K multi-turn conversations with \(100\)K single turn

Figure 2: Taxonomy of _Evol_ prompt conditions applied towards creating _MZLingual_. Part 1 includes _Evol_ prompts for Aya seeds and Part 2 has multi-turn _Evol_ prompts applied for creating conversation.

 

conversations.

## 4 Experiments

We conduct experiments across _three_ model families & _five_ model sizes -- Mistral-7B Jiang et al. (2023), LLaMA-3-8B Touvron et al. (2023) and QWEN-4B Bai et al. (2023). Furthermore, to demonstrate the effectiveness across different model scales, we fine-tune a larger model, LLaMA-2-13B Touvron et al. (2023), and a smaller model, QWEN-1.8B Bai et al. (2023). To evaluate how well the datasets work with instruction-tuned models, we also experiment with Mistral-Instruct-7B.

**Baselines** -- We use \(3\) multilingual instruction finetuning (IFT) datasets _MultiAlpaca_, _Bactrian-X_, and _Aya_ for main evaluation. Furthermore, to highlight the importance of each step in our synthesis, we consider several ablations. Specifically, we train models using \(1)\) only **Seed** samples, \(2)\) seed samples with the generated _Evol_ s (**Seed + Evol**) and \(3)\) seeds, _Evol_ s and the multi-turn conversations (**Seed + Evol + MT**).

### Evaluation

**Multilingual benchmarks.** We utilize the EleutherAI evaluation Gao et al. (2021) for consistent comparisons on the following tasks:

* _Question Answering_: We focus on \(3\) QA datasets 1) XQUAD Artetxe et al. (2019), TyDiQA Clark et al. (2020) and MLQA Lewis et al. (2019). We use \(3\) in-context examples and in the interest of time, we keep the number of examples per language to \(100\) for XQUAD and MLQA, and \(1000\) for TyDiQA. We use the validation set for XQUAD and test set for TyDiQA & MLQA with F1-score as the metric.
* _Summarization_: We use the XLSUM Hasan et al. (2021) dataset and focus on \(6\) languages -- Arabic, English, Spanish, French, Japanese and Russian with \(100\) examples per language and use GPT-4 as a judge to rate the generated summaries on a scale of 1 to 5. We also report ROUGEL Lin (2004) & BLEU Papineni et al. (2002) for comparison.
* _Multilingual math word problems_: We use MGSM Shi et al. (2022) that translates GSM8K Cobbe et al. (2021) to \(10\) languages. We use \(3\) in-context examples and compute exact match (EM) with ground truth answer.
* _Classification_: We focus on XNLI Conneau et al. (2018) and XCOPA Ponti et al. (2020) with \(15\) and \(11\) languages respectively in a zero-shot setting and report the results in Appendix. We compute the accuracy (Acc) by looking at the log-likelihood assigned to the ground truth answer on the validation set.

**Multilingual MT-Bench.** We evaluate conversational complex instruction following ability using MT-Bench Zheng et al. (2023). It has \(80\) multurn questions across \(8\) domains. The models are required to respond to an initial and a follow-up question and GPT-4 assesses the responses on a scale of \(1\) to \(10\), with the overall score being the mean over two turns. We translate it into \(8\) different languages French, Canadian French, German, Italian, Spanish Japanese, Dutch, Portuguese) with professional linguists to ensure high quality evaluation. We modify the judge prompt to include the language of the question, and instruct GPT-4 to make sure the responses are in the same language. We report the average scores across \(80\) examples for each language and the average MT-Bench score across all languages.

**Low-resource Evaluation.** To demonstrate the wide coverage of low resource languages in _M2Lingual_, we further evaluate models by on by translating MT-Bench to \(6\) low-resource languages namely Hindi, Urdu, Thai, Tamil, Bengali and Gujarati using GPT-4. Finally, we also perform low-resource evaluation across 10 languages from Flores200 Costa-jussa et al. (2022); Goyal et al. (2022). We present BLEU Papineni et al. (2002) scores for translating each language into every other language. The final score for a language is the average BLEU score across all its translations to the remaining languages. The languages we used are Arabic (arb), Assamese (asm), Awadhi (awa), Belarusian (bel), Haitian Creole (hat), Kirghiz (kir), Burmese (mya), Nepali (np), Somali (som), and Yoruba (yor). This selection covers a wide range of geographic regions South Asia, Africa, Eastern Europe, and the Middle East) and includes languages with different writing systems: Latin, Cyrillic, Arabic, and Devanagari scripts.

## 5 Results

**Multilingual MT-Bench** -- Table 2 shows the average scores on MT-Bench across 9 languages. On average, _M2Lingual_ outperforms other baseline datasets by \(1.01\) MT-bench score with Mistral-7B, and \(1.2\) with LLama-3-8B. The significant improvements across most baselines highlight the 

strengths of _M2Lingual_. Detailed results across all 9 languages of MT-bench with different base models are shown in Appendix in Table 12 and 15.

**Multilingual NLP results -- _M2Lingual_** leads in performance on multilingual NLP tasks across Mistral-7B and LLama-3-8B. Specifically in Table 2, on average _M2Lingual_ always outperforms on QA task by \(~{}6-8\%\) and on MGSM by \(8\%\) across both models. On summarization across both models our dataset outperforms by \(0.5\) GPT-4 score on an average. We observed GPT-4 score to be a more reliable metric than BleU or ROUGEL score for evaluating summarization quality, as those metrics tend to un-fairly penalize long form LLM answers (more in Section 6). However for completeness we report results on BleU and ROUGEL in Table 13. Finally, Table 13 and 14 in Appendix has results on XNLI, XCOPA and other base models Appendix A.8.

**Importance of _Evol_ -- The Seed + _Evol_ rows in Table 2 show the performance of seeds and synthetic _Evol_ IR pairs. In comparison to **Seed** data across both models, the average MT-Bench score increases by at least 1.75 points, with gains in every language (see Table 12 and 15), especially Japanese. Similarly, _M2Lingual_ leads to improvements of around \(5\) and \(10\) points on math word problem across Mistral-7B and LLama-3-8B respectively. On summarization and QA, the results are mixed (either increase, slight drop or same) as shown in Table 2. This is due to the increased verbosity of LLMs trained on _M2Lingual_, which contains detailed synthetic _Evol_s. (more in Section 6). Finally, to demonstrate that the importance lies in the use of synthetic _Evol_s rather than simply increasing the amount of seed-like data, we sampled an additional \(86K\) IR pairs from the Aya dataset and collection and replace it with synthetic generated _Evol_s and observed that the performance decreases across all benchmarks, especially in MT-Bench and MGSM by \(1.1\) and \(3.38\) points respectively (results are shown in Appendix A.5).

**Importance of multi-turn _Evol_.** The **Seed + _Evol_ + MT** rows in Table 2 shows performance after adding synthetically generated turns using multi-turn _Evol_. This boosts performance in MT-Bench evaluations substantially by \(0.8\) points with the most significant gain of \(1.31\) and \(1.0\) points on French and Japanese for Mistral-7B model (Table 12). Adding multi-turn data also helps in multilingual benchmarks as the results consistently improve by \(3-4\%\) across all evaluations. Additional results are shown in Appendix, Table 12, 13, 14, and, 15.

## 6 Additional Analysis

Comparison with Human-AI generated data in the wildFor completeness, we also include performance comparisons with Human-AI generated datasets collected from voluntary participation, such as OpenAssistant, ShareGPT, and WildChat in Table 3, where _M2Lingual_ shows strong performance results with both Mistral-7B and LLama-3-8B. Concretely, _M2Lingual_ outperforms OpenAssistant and ShareGPT by \(0.8\) and \(0.6\) on multilingual MT-Bench, \(8\%\) and \(12\%\) on QA, \(0.7\) and \(1.0\) on summarization and \(8\%\) and \(5\%\)

\begin{table}
\begin{tabular}{c c r r r r r r} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{1}{l}{**Dataset**} & \multicolumn{1}{l}{**MT-AVG**} & \multicolumn{1}{l}{**XQUAD**} & \multicolumn{1}{l}{**TyDiQA**} & \multicolumn{1}{l}{**MLQA**} & \multicolumn{1}{l}{**XLSUM**} & \multicolumn{1}{l}{**MGSM**} \\  & & & F1 & F1 & F1 & GPT-4 score & EM \\ \hline \multirow{4}{*}{**Embed**} & MultiAlpaca & 4.77 & 67.99 & 64.44 & 55.69 & 3.06 & 11.71 \\  & Bactrian-X & 5.25 & 71.91 & 66.63 & 60.27 & 2.59 & 17.14 \\  & Aya & 5.18 & 70.46 & 66.95 & 57.47 & 3.13 & 13.86 \\ \cline{2-9}  & Seed & 5.04 & 72.52 & 65.89 & 59.33 & 2.94 & 13.65 \\  & Seed + Evol & 5.56 & 71.01 & 65.04 & 57.47 & 3.41 & 18.38 \\  & Seed + Evol + MT (_M2Lingual_) & **6.54** & **74.53** & **67.57** & **62.40** & **3.55** & **22.00** \\ \hline \multirow{4}{*}{**Embed**} & MultiAlpaca & 4.41 & 75.08 & 64.49 & 59.01 & 3.17 & 10.68 \\  & Bactrian-X & 5.66 & 69.57 & 56.45 & 58.51 & 2.79 & 22.86 \\  & Aya & 4.95 & 75.14 & 59.60 & 53.14 & 3.29 & 22.09 \\ \cline{2-9}  & Seed & 3.54 & **77.27** & 68.57 & 60.01 & 2.84 & 11.71 \\  & Seed + Evol & 6.12 & 76.17 & **69.89** & 63.09 & 3.52 & **28.00** \\  & Seed + Evol + MT (_M2Lingual_) & **6.74** & 75.91 & 67.84 & **63.5** & **3.68** & 27.36 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison of Mistral-7B and LLaMA-3-8B. MT-Avg is average MT bench results across 9 languages (French, Canadian French, German, Italian, Spanish, Japanese, Dutch, Portuguese). _Seeds_ are \(15.1\)K seeds; _Seed + Evol_ is additional _Evol_ IR pairs. _Seed + Evol + MT_ has additional multi-turn data.

 

on math word problem solving. _M2Lingual_ performs comparable to WildChat on multilingual MT-Bench however strongly outperforms by \(2-3\%\) on QA, \(0.6\) on summarization and \(2\%\) on math word problem solving. We report per language MT bench scores, results on other metrics and classification benchmarks in Appendix (Tables 12, 13 14 and 15). Finally we also compare the performance of _M2Lingual_ on low resource languages (see below) and observed that _M2Lingual_ notably outperforms these Human-AI generated datasets as shown in table 4 and table 5. However, we would like to point out that these datasets are not focused specifically towards improving multilingual abilities, even though their creation methods lend inadvertently to multilingual data. For holistic comparison, we also include details with the next best performing dataset WildChat in further analysis.

Low-resource languages.Table 4 shows the results on \(6\) languages. _M2Lingual_ performs better than all the baselines across both Mistral-7B and LLama-3-8B. Specifically, _M2Lingual_ improves the performance by \(1.3\) and \(0.2\) on average for both models respectively. We also evaluate cross-lingual machine translation performance on extremely low resource languages as shown in Table 5. Table 5 demonstrates that _M2Lingual_ outperforms existing baseline datasets by noticeable margin of \(0.2\) BLEU score improvements with Mistral-7B and \(0.5\) with LLama-3-8B. The M2Lingual models outperform all baselines in translating between different low-resource languages, except on Somhali, Awadhi (LLama-3-8B) and Yoruba (Mistral-7B) where their performance is a close second. This highlights a better coverage of low-resourced languages in _M2Lingual_ (Figure 4).

Effect of IFT datasets on different sized LLMs.We also study the impact of _M2Lingual_ on a smaller scale model (QWEN-1.8B) and a larger model (LLaMA-2-13B). As shown in Table 6, QWEN-1.8B on an average, _M2Lingual_ leads to \(1.96\), \(10\) and \(4.5\) points improvements across MTbench, QA and MGSM respectively. Similarly for the LLaMA-2-13B we get \(0.8\), \(3.75\) and \(3.0\) points increase. The MT-Bench results across different languages are shown in Table 16 in Appendix.

Scaling data synthesis with seed size.Finally, to show how performance changes as we scale synthetic data generation on more seed examples only, we ran \(2\) ablations where we 1) use only 25% of seed examples and use its synthesized data and 2) use 50% of the seeds. Figure 3 shows that as

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Dataset**} & **MT-AVG** & **XQUAD** & **TyDiQA** & **MLQA** & **XLSUM** & **MGSM** \\  & & F1 & F1 & F1 & GPT-4 score & EM \\ \hline \multirow{4}{*}{**Mistral-7B**} & Open Assistant & 5.66 & 67.99 & 54.22 & 53.64 & 2.60 & 16.05 \\  & SharegPT & 5.80 & 66.33 & 56.97 & 50.78 & 2.41 & 11.32 \\  & WildChat & 6.53 & 72.55 & 64.27 & 59.53 & 2.93 & 18.41 \\  & _M2Lingual_ & **6.54** & **74.53** & **67.57** & **67.57** & **3.55** & **22.00** \\ \hline \multirow{4}{*}{**LLaMA-3-8B**} & Open Assistant & 5.12 & 64.38 & 52.65 & 47.08 & 2.92 & 17.36 \\  & SharegPT & 6.10 & 56.98 & 58.48 & 43.43 & 2.54 & 25.32 \\ \cline{1-1}  & MultChat & **6.75** & 63.15 & 59.88 & 63.16 & 2.90 & 26.36 \\ \cline{1-1}  & _M2Lingual_ & 6.74 & **75.91** & **67.84** & **63.50** & **3.68** & **27.36** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance comparison of Mistral-7B and LLaMA-3-8B with OpenAssitant, SharegPT & WildChat.

 

we scale data synthesis by selecting more seeds the performance increases across all benchmarks. Specifically, on an average we see \(0.5\) improvement in multilingual MT-bench, \(5\%\) in QA and MGSM and \(2.50\) in summarization. Additional analysis and tables for Figure 3 are shown in Appendix A.6.

Distribution of Languages.Figure 4 shows a balanced representation of languages in our dataset compared to WildChat and Aya, which have uneven or very skewed distribution. This highlights a broader coverage of mid to low resource languages and explains the consistent performance improvements across high-mid resource languages in Tables 2, 3, 12 and 13 and low resource languages in Table 4 and 5.

Token Lengths per Utterance.Table 7 shows that _M2Lingual_ has one of the highest user and assistant turn tokens and the highest total number of tokens (computed via LLama tokenizer). This explains on complex benchmarks such as MT-Bench and MGSM, which require reasoning using a chain-of-thought or processing long contexts; and an occasional slight drop in performance on QA or summarization tasks, as the F1-score does not fully capture long, detailed answers.

Content moderation.To ensure low toxicity in _M2Lingual_'s content as well as evaluate the _Evol_ synthesis method's sensitivity in data generation, we conduct moderation testing with OpenAI Moderation API [9] following [8]. Table 8 shows that less than \(0.2\%\)

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Dataset**} & \multicolumn{1}{l}{**MT-AVG**} & **XQUAD** & **TyDiQA** & **MIQA** & **XLSUM** & **MGSM** \\  & & & F1 & F1 & F1 & GPT-4 score & EM \\ \hline \multirow{3}{*}{Owen-1.8B} & MultiAlpaca & 2.03 & 31.60 & 33.38 & 19.30 & 2.03 & 7.45 \\  & WildChat & 2.59 & 45.12 & 42.42 & 29.39 & 1.90 & 8.00 \\  & _M2Lingual_ & **4.27** & **52.12** & **47.66** & **38.24** & **2.83** & **12.23** \\ \hline \multirow{3}{*}{LLaMA-2-13B} & MultiAlpaca & 4.46 & 55.07 & 59.46 & 48.74 & 2.48 & 7.80 \\  & WildChat & 6.00 & 67.64 & 60.14 & 53.69 & 2.55 & 9.95 \\ \cline{1-1}  & _M2Lingual_ & **6.08** & **69.38** & **64.66** & **54.64** & **3.03** & **11.95** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Evaluations of QWEN-1.8B and LLaMa-2-13B for highlighting impact on different sized LLMs.

Figure 3: Performance vs seed size in data synthesis



of _M2Lingual_ is flagged by the Moderation API. We remove the flagged utterances before making the dataset public. It is worth noting that Human-AI generated datasets like WildChat have substantially more sensitive content.

## 7 Conclusion

We introduce _**M2Lingual**_ - the _first fully synthetic, multi-turn multilingual dataset_ - containing \(175K\) complex conversations across \(70+\) languages and \(19\) NLP tasks. We propose a scalable, cost-efficient and fully synthetic method for creating conversations using a two-step enrichment process based on the _Evol_ prompt taxonomy, which can be adapted to any task or monolingual data. Exhaustive experiments across _three_ model families and _five_ model sizes with evaluations spanning \(31\) languages demonstrate the advantages of _**M2Lingual**_ over other datasets. Furthermore, our ablations and analysis on low-resource language support, content moderation, conversation length and language distribution demonstrate the quality of _**M2Lingual**_ over other datasets.

## 8 Limitations and Ethical Considerations

_**M2Lingual**_ covers over 70 languages in total, with dialects added in with _Evol_ as well - a significant number of languages, more than all relevant datasets (Table 1). However, there are many more languages in the real-world, and we cannot cover them all. We hope that our contribution helps expand access to languages, and future work can further build better access for all. Moreover, the performance of LLMs improves on low resource data with finetuning on _**M2Lingual**_, showcasing the importance of including multiple languages and turns.

Some major limitations of _**M2Lingual**_ include the limited conversation length, possible presence of toxic data, and dependence on GPT-4 translated MT-Bench for low-resource language evaluation. While potentially longer conversations could be built with _Evol_, it would take significantly more resources to extend each conversation beyond the current limit. For toxicity, our seed dataset Aya does not contain specific flags for toxic, harmful, or offensive speech Singh et al. (2024), and Aya authors report that they believe there is a low risk for these in Aya data. However, to mitigate risk, we conduct moderation analysis of the generated _Evol_ IR pairs for _**M2Lingual**_, and find that less than \(0.2\%\) of the generated data was flagged, which we filter out before making the data public. Lastly, we conduct limited manual evaluation of the GPT-4 generated low-resource multilingual MT-Bench data generated by GPT4, and find that it performs satisfactorily well. However, improving evaluation on low-resource data remains an area of future work.

Figure 4: Comparison between Aya, WildChat and _**M2Lingual**_ language distribution.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Dataset** & **User\%** & **Chatbot\%** & **Avg\%** \\ \hline Alpaca & 0.01 & 0.02 & 0.01 \\ Aya & 0.10 & 0.24 & 0.22 \\ Open Assistant & 0.53 & 0.45 & 0.49 \\ Share GPT & 0.16 & 0.28 & 0.22 \\ WildChat & 6.05 & 5.18 & 5.61 \\ _**M2Lingual**_ & 0.21 & 0.13 & 0.17 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Content moderation analysis reported from respective dataset papers (BactrainX does not perform toxicity analysis) 



## References

* A. Abay, Y. Zhou, M. Kantarcioglu, B. Thuraisingham, and L. Sweeney (2019)Privacy preserving synthetic data release using deep learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part I 18, pp. 510-526. Cited by: SS3.
* O. J. Ashiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. Leoni Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadet-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A. Brockman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, E. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, Z. Chen, B. Chess, C. Cho, C. Chu, H. Won Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dolan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Elouandu, D. Farhi, L. Fedus, N. Felix, S. On Posada Fishman, J. Forte, I. Folford, L. Gao, E. Georges, M. Gibson, W. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Graftein, S. Gray, R. Greene, J. Gross, S. Shane Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Hearon, J. Heideecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschelle, B. Houghton, Kenny Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, L. Kaiser, A. Kamali, I. Kanitscheider, N. Shirish Keskar, Tabarak Khan, Logan Kilpatrick, J. W. Kim, C. Kim, Y. Kim, H. Kirchner, J. Ryan Kiros, M. Knight, D. Kokotajlo, L. Kondraciuk, A. Kondrciuk, A. Kondrich, A. Konstantinis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. Ming Li, R. Lim, M. Lin, S. Liu, M. Litwin, T. Lopez, R. Lowe, P. Luc, A. Adeola Makanu, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. Mayer McKKinney, C. McLeavey, P. McMillan, J. McNNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. P. Mossing, T. Mu, M. Murati, O. Murak, D. M'ely, A. Nair, Reitichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O'Keefe, Jakub W. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, Joel Parish, Emy Paparparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, A. Perelman, F. de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, M. Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Poststed, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin D. Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer on Uribe, Andrea Vallone, Arun Vijayeriya, Chelsea Voss, Carroll L. Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willmer, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciceh Zemmba, Rowan Zelters, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. (2023)Gpt-4 technical report. External Links: Link Cited by: SS3.
* M. Arttetxe, S. Ruder, and D. Yogatama (2019)On the cross-lingual transferability of monolingual representations. arXiv preprint arXiv:1910.11856. Cited by: SS3.
* J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, Rui Meng, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Su, J. Wu, B. Xu, J. Cao, H. Yang, J. Yang, J. Yao, J. Yu, Y. Bowen, H. Yi, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu (2023)Qwen technical report. arXiv:abs/2309.16609. External Links: Link Cited by: SS3.
* Y. Bizzoni, T. S. Juzek, C. Espana-Bonet, K. Dutta Chowdhury, J. van Genabith, and E. Teich (2020)How human is machine translationese? comparing human and machine translations of text and speech. In Proceedings of the 17th International conference on spoken language translation, pp. 280-290. Cited by: SS3.
* L. Chen, S. Li, J. Yan, H. Wang, K.



Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2023. Alpagasus: Training a better alpaca with fewer data. _arXiv preprint arXiv:2307.08701_.
* Chen et al. (2024) Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Andrey Kutuzov, Barry Haddow, and Kenneth Heafield. 2024. Monolingual or multilingual instruction tuning: Which makes a better alpaca. In _The 18th Conference of the European Chapter of the Association for Computational Linguistics_, pages 1-10. Association for Computational Linguistics.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Liamin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impression gpt-4 with 90%* chatgtp quality. _See https://vicuna. lmsys. org (accessed 14 April 2023)_.
* Choquette et al. (2021) Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. Nvidia a100 tensor core gpu: Performance and innovation. _IEEE Micro_, 41(02):29-35.
* Chung et al. (2024) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. _Journal of Machine Learning Research_, 25(70):1-53.
* Clark et al. (2020) Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garreette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in ty topologically di verse languages. _Transactions of the Association for Computational Linguistics_, 8:454-470.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, et al. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Conneau et al. (2018) Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating crosslingual sentence representations. _arXiv preprint arXiv:1809.05053_.
* Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's first truly open instruction-tuned llm. _Company Blog of Databricks_.
* Costa-jussa et al. (2022) Marta R Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Hefferman, Elae Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. _arXiv preprint arXiv:2207.04672_.
* Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yuja Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. _arXiv preprint arXiv:2305.14233_.
* Elmadary et al. (2023) AbdelRahim Elmadary, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Octopus: A multitask model and toolkit for arabic natural language generation. _arXiv preprint arXiv:2310.16127_.
* Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.
* Ghosh et al. (2024) Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha, et al. 2024. A closer look at the limitations of instruction tuning. _arXiv preprint arXiv:2402.05119_.
* Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Mael Kubli. 2023. Chatgptt outperforms crowd workers for text-annotation tasks. _Proceedings of the National Academy of Sciences_, 120(30):e2305016120.
* Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhry, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Runzato, Francisco Guzman, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. _Transactions of the Association for Computational Linguistics_, 10:522-538.
* Guo et al. (2021) Han Guo, Bowen Tan, Zhengzhong Liu, Eric P Xing, and Zhiting Hu. 2021. Efficient (soft) q-learning for text generation with limited good data. _arXiv preprint arXiv:2106.07704_.
* Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharejee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahrivar. 2021. XI-sum: Large-scale multilingual abstractive summarization for 44 languages. _arXiv preprint arXiv:2106.13822_.
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Menesch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral ?b. _arXiv preprint arXiv:2310.06825_.
* Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. _arXiv preprint arXiv:2401.04088_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Kopf et al. (2024) Andreas Kopf, Yannic Kilcher, Dimitri von Rutte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, et al. 2024. Openassistant conversations-democratizing large language model 

alignment. _Advances in Neural Information Processing Systems_, 36.
* Kwan et al. (2024) Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. Mt-eval: A multi-turn capabilities evaluation benchmark for large language models. _arXiv preprint arXiv:2401.16745_.
* Lewis et al. (2019) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Mlqa: Evaluating cross-lingual extractive question answering. _arXiv preprint arXiv:1910.07475_.
* Li et al. (2023) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-x: A multilingual replicable instruction-following model with low-rank adaptation. _arXiv preprint arXiv:2305.15011_.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* OpenAI (2024) OpenAI. 2024. Opensal moderation api.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vulic, and Anna Korhonen. 2020. Xcopa: A multilingual dataset for causal commonsense reasoning. _arXiv preprint arXiv:2005.00333_.
* Przystupa and Abdul-Mageed (2019) Michael Przystupa and Muhammad Abdul-Mageed. 2019. Neural machine translation of low-resource and similar languages with backtranslation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_, pages 224-235.
* Ranaldi et al. (2023) Leonardo Ranaldi, Giulia Pucci, and Andre Freitas. 2023. Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations. _arXiv preprint arXiv:2308.14186_.
* RyokoAI (2023) RyokoAI. 2023. Sharegpt.
* Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. 2021. Multitask prompted training enables zero-shot task generalization. In _International Conference on Learning Representations_.
* Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Sorousn Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. _arXiv preprint arXiv:2210.03057_.
* Singh et al. (2024) Shivalika Singh, Freddie Vargus, Daniel Dsouza, Borje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deviidas Maticunas, Laura OMahony, et al. 2024. Ava dataset: An open-access collection for multilingual instruction tuning. _arXiv preprint arXiv:2402.06619_.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. _arXiv preprint arXiv:2312.11805_.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lacdaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Ustun et al. (2024) Ahmet Ustun, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'sourza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. _arXiv preprint arXiv:2402.07827_.
* Vanmassenhove et al. (2021) Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. 2021. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. _arXiv preprint arXiv:2102.00287_.
* Wang et al. (2023a) Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Ai Ti Aw, and Nancy F Chen. 2023a. Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. _arXiv preprint arXiv:2309.04766_.
* Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishizzi. 2022sb. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508.
* Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Aliopolomolabshi, Yeganeh Kordi, Amirreza Mirzeia, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Angana Arunkumar, David Stap, et al. 2022. Supernaturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In _Proceedings of 

the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5085-5109.
* Wei et al. (2023) Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023. PolyIM: An open source polyplot large language model. _arXiv preprint arXiv:2307.06018_.
* Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2023. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In _The Twelfth International Conference on Learning Representations_.
* Zhang et al. (2023) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Lu, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. _arXiv preprint arXiv:2308.10792_.
* Zhang and Zhao (2021) Zhuosheng Zhang and Hai Zhao. 2021. Advances in multi-turn dialogue comprehension: A survey. _arXiv preprint arXiv:2103.03125_.
* Zhao et al. (2024) Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. _arXiv preprint arXiv:2405.01470_.
* Zheng et al. (2023a) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhnao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. 2023a. Lmsyschait-1m: A large-scale real-world llm conversation dataset. _arXiv preprint arXiv:2309.11998_.
* Zheng et al. (2023b) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. _Preprint_, arXiv:2306.05685 

Appendix

### Experiment Details

We conduct experiments across _three_ model families & _five_ model sizes -- Mistral-7B Jiang et al. (2023), LLaMA-3-8B Touvron et al. (2023) and QWEN-4B Bai et al. (2023). Furthermore, to demonstrate the effectiveness of our dataset across different model scales, we fine-tune both a larger model, LLaMA-2-13B Touvron et al. (2023), and a smaller model, QWEN-1.8B Bai et al. (2023). To evaluate how well the datasets work with instruction-tuned models, we also experiment with Mistral-Instruct-7B.

### Baseline Datasets

We use _six_ different multilingual datasets as baselines for comparison: 1) the top ranked conversation trees from **Open Assistant**Kopf et al. (2024), 2) **Aya**Singh et al. (2024), 3) self-instruct dataset **MultiAlpaca**Wei et al. (2023), 4) machine translated **Bactrian-X**Li et al. (2023) derived from Alpaca-52k Taori et al. (2023) and Dolly-15k Conover et al. (2023), 5) the **ShareGPT**++ collection, and 6) **WildChat**Zhao et al. (2024).

Footnote ‡: [https://sharegpt.com/](https://sharegpt.com/)

For a fair comparison with WildChat, we use \(200\text{K}\) non-English conversations, ensuring the same language proportions, and downsampled \(60\text{K}\) English conversations, resulting in a total of \(260\text{K}\) conversations. Similarly for Bactrian-X, we sample \(1\text{M}\) IR pairs ensuring the same language proportions as in the original dataset.

**Additional Baselines** To highlight the importance of each step in our data curation process, we consider several ablations as baselines. Specifically we conduct experiments by training models using 1) only **Seed** samples, 2) seed samples with the generated evols (**Seed + Evol**) and 3) seeds, evols and the generated multi-turn conversations (**Seed + Evol + MT**). Finally, to see whether adding parallel data (PD) helps in improving the over model's performance, we collect \(60\text{K}\) from the Aya collection and train a baseline by augmenting the PD with our full dataset (**Seed + Evol + MT + PD**).

### Training

All training is performed on \(8\) A-\(100\)\(80\text{GB}\) NViDIA GPUs Choquette et al. (2021), with the AxolotlSS framework. We used Mistral tags Jiang et al. (2023) for finetuning all models. We use a batch size of \(64\), a maximum sequence length of \(8192\), a learning rate of \(5\times 10^{-6}\), the Adam optimizer Kingma and Ba (2014) with a cosine scheduler, and \(10\) warmup steps. We reserve a \(5\%\) validation split, and train all the models until validation loss convergence. We compute the loss only on the targets using fp\(16\) training.

Footnote §: [https://github.com/OpenAccess-AI-Collective/axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)

### Evaluation

#### Multilingual benchmarks.

We utilize the EleutherAI evaluation framework Gao et al. (2021) for consistent comparisons. We evaluate the performance of different multilingual datasets on the following tasks:

* _Question Answering (QA)_: We focus on \(3\) multilingual QA datasets 1) XQUAD Artetxe et al. (2019) with QA across \(11\) languages, 2) TyDiQA Clark et al. (2020) which has human generated QA in \(11\) languages and 3) MLQA Lewis et al. (2019) with QA in \(7\) languages. While QA data requires short answer phrases, conversational IR pairs might lead to longer answer span generation. Hence, we use \(3\) in-context examples to get the right output format for LLMs. In the interest of time, we keep the number of examples per language to \(100\) for XQUAD and MLQA, and \(1000\) for TyDiQA. We use the validation set for XQUAD and test set for TyDiQA & MLQA, and compute the standard F1-score.
* Arabic, English, Spanish, French, Japanese and Russian. We restrict the total number of examples to \(100\) and prompt the model to generate a summary in the same language as the context. We look at the ROUGEL Lin (2004) & BLEU Papinieni et al. (2002) scores for comparison.
* _Classification_: We focus on XNLI Conneau et al. (2018) and XCOPA Ponti et al. (2020) with \(15\) and \(11\) languages respectively in a zero-shot setting. We compute the accuracy (Acc) by looking at the log-likelihood assigned to the ground truth answer on the validation set.
* _Multilingual math word problems_: We use MGSM Shi et al. (2022), a grade-school math benchmark that translates GSM8K Cobbe et al. (2021) to \(10\) different languages. Similar to QA tasks, we use \(3\) in-context examples and compute the exact match (EM) with the ground truth answer.

**Translated MT-Bench.** To evaluate the conversation and instruction following ability of multi 

lingual models across a wide array of tasks and languages, we translate MT-Bench [25]. MT-Bench comprises of \(80\) multi-turn questions across \(8\) domains. The models are required to respond to an initial and a follow-up question and GPT-4 assesses the model's responses on a scale of \(1\) to \(10\) (\(10\) being the best), with the overall score being the mean over the two turns. We translate it into \(9\) different languages with professional linguists to ensure high quality evaluation. We modify the judge prompt to include the language of the question asked at each turn, and additionally instruct GPT-4 to make sure the responses are in the same language as the question asked. We report the average scores across all \(80\) examples for each language and also report the average MT-Bench score across all languages.

### Importance of synthetic _Evol_s

To assess whether the importance lies in the use of synthetic _Evol_s rather than simply increasing the amount of seed-like data, we sampled an additional \(94.9K\) IR pairs from the Aya dataset and collection and replace it with synthetic generated _Evol_s. Results in 9 show that without synthetic _Evol_s the performance decreases, whereas having the same number of _Evol_ IR pairs leads to higher performance especially in MT-Bench and MGSM by \(1.1\) and \(3.38\) points respectively.

### _M2Lingual_ performance without seed examples

Table 10 demonstrates the performance of our dataset \((1)\) without seeds and \((2)\) with 25% seed examples. Results show strong performance with multilingual MT bench without any seeds. It improves slightly compared to the last column that has all seed examples. The performance on other benchmark drops slightly but it still outperforms the evaluated baseline datasets in the paper.

### Results with variable seed size

Finally, to show how performance changes as we scale synthetic data generation on more seed examples only, we ran \(2\) ablations where we \((1)\) use only 25% of seed examples and use its synthesized data and \((2)\) use 50% of the seeds. Figure 3 and Table 11 demonstrate that as we scale data synthesize by selecting more seed examples the performance increases across all benchmarks. Specifically, on an average we see \(0.5\) improvement in multilingual MT-bench, \(5\%\) in QA and MGSM and \(2.50\) in summarization.

### Complete Results

Tables 12, 13, 14 and 15 shows the complete results comparing _M2Lingual_ against all the baseline datasets, \(4\) base models across multilingual MT-Bench, question answering, summarization and classification tasks. Table 16 compares _M2Lingual_ against top performing baseline on a smaller (Qwen1.8B) and a larger (LLama-2-13B) model.

 

of multilingual IFT datasets on pre-instruction fine-tuned models. _M2Lingual_ leads Mistral-Instruct-7B to achieve best performance in \(5\) of \(8\) MT-Bench language evaluations and \(5\) of the \(7\) multilingual evaluation benchmarks as shown in Tables 14 and 15 respectively. Interestingly, the improvements from _M2Lingual_ in Mistral-Instruct-7B over baseline datasets is consistently higher when compared to Mistral-7B-base (Table 13) in all of the multilingual QA tasks, MGSM, and XCOPA. We also evaluate QWEN-4B model to showcase results from smaller LLM from different model family. We observe similar findings as QWEN-4B finetuned with _M2Lingual_ achieves competitive results in both MT-Bench and multilingual evaluation datasets. Another interesting observation is that improvements seem relatively higher for QWEN-4B model using _M2Lingual_ when compared to Mistral-7B and LLaMA-3-8B models, highlighting the usefulness of our proposed data on moderate sized LLMs.

### Examples of Generated Evols and conversations

#### Prompt Taxonomy for Evol-instruct

For Dolly, HotpotQA and MLQA we use evols from generic, OpenQA and Mitaka respectively.

 



\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Model** & **Dataset** & **XQUAD** & **TyDiQA** & **MLQA** & **XLSUM** & **MGSM** & **XXLI** & **XCOPA** \\  & & F1 & F1 & F1 & ROUGE\({}_{\text{L}}\) & BLEU & EM & Acc & Acc & Acc \\ \hline \multirow{8}{*}{**WordNet**} & Open Assistant & 67.99 & 54.22 & 53.64 & 10.86 & 0.85 & 16.05 & 42.74 & 56.73 \\  & MultiAlpaca & 67.99 & 64.44 & 55.69 & 10.9 & 1.59 & 10.41 & 42.18 & 58.91 \\  & Backrian-X & 71.91 & 66.63 & 60.27 & 3.30 & 0.20 & 17.14 & **43.91** & 58.64 \\  & Share GPT & 66.33 & 56.97 & 50.78 & 3.31 & 0.288 & 11.32 & 41.13 & 56.09 \\  & WildChat & 72.55 & 64.27 & 59.53 & 3.91 & 0.41 & 18.41 & 43.11 & 58.00 \\  & Aya & 70.46 & 66.95 & 57.47 & **12.5** & **2.01** & 13.86 & 41.78 & 59.00 \\ \cline{2-11}  & Seed & 72.52 & 65.89 & 59.33 & 11.53 & 1.72 & 13.65 & 42.28 & 57.64 \\  & Seed + Evol & 71.01 & 65.04 & 57.47 & 9.8 & 1.37 & 18.38 & 43.00 & 57.55 \\  & Seed + Evol + MT (**M2Lingual**) & **74.53** & **67.57** & **62.40** & 10.42 & 1.38 & **22.00** & 42.12 & **59.55** \\ \hline \multirow{8}{*}{**WordNet**} & Open Assistant & 64.38 & 52.65 & 47.08 & 9.38 & 1.21 & 17.36 & 46.17 & **63.82** \\  & MultiAlpaca & 75.08 & 64.49 & 59.01 & **10.98** & **1.45** & 10.68 & **46.93** & 63.55 \\  & Bactrian-X & 69.57 & 56.45 & 58.51 & 8.39 & 1.28 & 22.86 & 46.90 & 62.18 \\  & ShareGPT & 56.98 & 58.48 & 43.43 & 3.53 & 0.40 & 25.32 & 45.93 & 63.00 \\  & WildChat & 63.15 & 59.88 & 63.16 & 5.52 & 0.76 & 26.36 & 46.88 & 62.27 \\  & Aya & 75.14 & 59.60 & 53.14 & 10.38 & 1.39 & 22.09 & 45.64 & 63.55 \\ \cline{2-11}  & Seed & **77.27** & 68.57 & 60.01 & 9.92 & **1.45** & 17.18 & 46.02 & 62.82 \\  & Seed + Evol & 76.17 & **69.89** & 63.09 & 8.96 & 1.23 & **28.00** & 46.38 & 61.36 \\ \cline{2-11}  & Seed + Evol + MT (**M2Lingual**) & 75.91 & 67.84 & **63.50** & 8.87 & 1.25 & 27.36 & 46.18 & 62.55 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Evaluations of LLaMA-3-8B-base & Mistral-7B-base in different tasks. Same notations as in Table 12



\begin{table}
\begin{tabular}{l l c r r r r r r r r r r} \hline \hline Model Name & Dataset & MT-EN & MT-FR & MT-TT & MT-JP & MT-ES & MT-DE & MT-NL & MT-PT & MT-Avg & MGSM & MLQA & TyDQA \\ \hline  & WideChat & 4.99 & 2.75 (2.74) & 2.08 & 1.72 & 2.88 & 1.92 & 1.54 & 2.68 & 2.59 & 8.00 & 29.39 & 42.42 \\ Qwen-1.8B & MultiAlpaca & 3.97 & 1.93 (1.99) & 1.74 & 1.44 & 1.87 & 1.86 & 1.52 & 1.91 & 2.03 & 7.45 & 19.30 & 33.38 \\  & _MZingual_ & **6.20** & **4.55 (4.25)** & **3.85** & **3.27** & **4.40** & **4.11** & **3.32** & **4.51** & **4.27** & **12.23** & **38.24** & **47.66** \\ \hline  & WildChat & **6.64** & 6.25 (5.89) & 5.98 & 5.10 & **6.20** & **6.10** & 5.82 & 5.99 & 6.00 & 9.95 & 53.69 & 60.14 \\ LLaMA-2-13B & MultiAlpaca & 5.09 & 4.35 (4.55) & 4.35 & 3.52 & 4.47 & 4.54 & 4.69 & 4.62 & 4.46 & 7.80 & 48.74 & 59.46 \\  & _MZingal_ & 6.47 & **6.40 (6.20)** & **6.13** & **5.35** & 6.18 & 5.94 & **5.87** & **6.17** & **6.08** & **11.95** & **54.64** & **64.66** \\ \hline \hline \end{tabular}
\end{table}
Table 16: Evaluations of QWEN-1.8B and LLaMa-2-13B for highlighting impact on different sized LLMs.

\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Dataset**} & **XQUAD** & **TyDiQA** & **MLQA** & **XLSUM** & **MGSM** & **XXLI** & **XCOPA** & **MT-Avg** \\  & & F1 & F1 & F1 & ROUGEL & BLEU & EM & Acc & Acc & **MT-Avg** \\ \hline \multirow{8}{*}{**LMaMA-2-13B**} & Open Assistant & 53.63 & 45.30 & 46.34 & 4.15 & 0.29 & 17.50 & 38.52 & 58.45 & 3.47 \\  & MultiAlpaca & 51.81 & 53.51 & 40.26 & 8.9 & 1.0 & 12.1 & 38.3 & 58.40 & 2.93 \\  & Bactrian-X & 46.70 & 42.79 & 42.2 & 7.1 & 0.8 & 18.6 & 38.3 & 57.70 & 3.80 \\  & SharedGPT & 41.86 & 28.20 & 36.03 & 4.58 & 0.43 & 16.95 & 37.83 & **58.55** & 3.80 \\  & WildChat & 53.18 & 49.18 & 42.81 & 5.23 & 0.56 & 19.27 & **38.74** & 58.18 & **4.29** \\  & Aya & 54.00 & 52.14 & 48.28 & **10.91** & **1.31** & 16.50 & 37.59 & 57.73 & 3.43 \\ \cline{2-11}  & Seed & **66.55** & **58.09** & 48.25 & 10.65 & 0.65 & 15.36 & 37.59 & 58.00 & 2.47 \\  & Seed + Evol & 52.24* & 52.50 & 49.87 & 8.50 & 1.12 & 20.77 & 38.36 & 57.91 & 3.79 \\  & Seed + Evol + MT (_MZlingual_) & 49.12* & 47.53 & **50.36** & 8.30 & 1.02 & **21.36** & 38.37 & 58.36 & 4.23 \\ \hline \multirow{8}{*}{**LMaMA-2-13B**} & Open Assistant & 61.33 & 59.28 & 53.27 & 9.62 & 1.43 & 19.00 & 43.91 & 58.09 & 5.58 \\  & MultiAlpaca & 63.76 & 63.05 & 51.09 & 11.51 & 1.80 & 13.18 & **44.70** & 58.18 & 4.74 \\  & Bactian-X & 70.5 & 64.8 & 50.60 & 9.14 & 1.35 & 17.91 & 42.23 & 57.25 & 5.98 \\  & ShareGPT & 44.53 & 49.5 & 40.45 & 3.31 & 0.38 & 17.36 & 42.13 & 56.73 & 6.11 \\  & WildChat & 61.53 & 53.1 & 52.60 & 6.31 & 0.56 & 21.00 & 41.86 & 57.75 & 6.62 \\  & Aya & 69.9 & 66.43 & 57.27 & **12.58** & **2.05** & 16.36 & 42.84 & 58.60 & 5.20 \\ \cline{2-11}  & Seed & 68.78 & 61.54 & 56.11 & 12.45 & 2.04 & 18.27 & 43.23 & 58.45 & 3.92 \\ \cline{2-11}  & Seed + Evol & **72.87** & 68.43 & 55.43 & 12.51 & 1.33 & 22.00 & 42.51 & 58.09 & 6.48 \\ \cline{2-11}  & Seed & 4.80 & **47.21** & 69.44 & 58.33 & 9.57 & 1.51 & 19.82 & 42.37 & **59.45** & **6.64** \\ \hline \hline \end{tabular}
\end{table}
Table 14: Evaluations of QWEN-4B & MISTral-Instruct-7B in different tasks and MT-Bench score averaged across languages. Please see table 15 in appendix for MT-Bench score in each language. \(\star\) in XQUAD, TyDiQA scores for QWEN-4B show exception cases where outputs had repeated noisy patterns in multiple runs resulting in low scores.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline Model Name & Dataset & MT-EN & MT-FR & MT-TT & MT-JP & MT-ES & MT-DE & MT-NL & MT-PT & MT-Avg & MGSM & MLQA & TyDQA \\ \hline  & WordChat & 4.99 & 2.75 (2.74) & 2.08 & 1.72 & 2.88 & 1.92 & 1.54 & 2.68 & 2.59 & 8.00 & 29.39 & 42.42 \\ Qwen-1.8B & MultiAlpaca & 3.97 & 1.93 (1.99) & 1.74 & 1.44 & 1.87 & 1.86 & 1.52 & 1.91 & 2.03 & 7.45 & 19.30 & 33.38 \\  & _MZlingual_ & **6.20** & **4.55 (4.25)** & **3.85** & **3.27** & **4.40** & **4.11** & **3.32** & **4.51** & **4.27** & **12.23** & **38.24** & **47.66** \\ \hline  & WildChat & **6.64** & 6.25 (5.89) & 5.98 & 5.10 & **6.20** & **6.10** & 5.82 & 5.99 & 6.00 & 9.95 & 53.69 & 60.14 \\ LLaMA-2-13B & MultiAlpaca & 5.09 & 4.35 (4.55) & 4.35 & 3.52 & 4.47 & 4.54 & 4.69 & 4.62 & 4.46 & 7.80 & 48.74 & 59.46 \\  & _MZlingual_ & 6.47 & **6.40 (6.20)** & **6.13** & **5.35** & 6.18 & 5.94 & **5.87** & **6.17** & **6.08** & **11.95** & **54.64** & **64.66** \\ \hline \hline \end{tabular}
\end{table}
Table 15: MT-Bench evaluations in different languages for QWEN-4B and MISTral-Instruct-7B.





\begin{table}
\begin{tabular}{c l c} \hline
**Step** & **Content** & **Evol** \\ \hline Seed Sample & What is the longest-running Broadway show ever? \\ \hline evolved (Turn 1) Instr. & Which Broadway show holds the record for the longest run? & Concise \\  & Please answer briefly. \\ \hline GPT-4 Response & The Phantom of the Opera holds the record for the longest \\  & run on Broadway. \\ \hline Turn 2 & Now, break down the following multi-part question and & Expanding Queries \\  & answer each part: Who wrote this show? What are some & Change Context \\  & of its most popular songs? When did it first premiere on & \\  & Broadway? \\ \hline GPT-4 Response & 1. Who wrote this show? & \\  & The Phantom of the Opera was composed by Andrew & Lloyd Webber. The lyrics were written by Charles Hart and & Richard Stilgoe. \\  & 2. What are some of its most popular songs? & \\  & Some of its most popular songs include ”The Phantom of & \\  & the Opera,” “Music of the Night,” ”All I Ask of You,” and "Masquerade.” & \\  & 3. When did it first premiere on Broadway? & \\  & The Phantom of the Opera first premiered on Broadway on & \\  & January 26, 1988. \\ \hline \end{tabular}
\end{table}
Table 17: Conversation example from _M2Lingual_



\begin{table}
\begin{tabular}{c l c} \hline \hline
**Step** & **Content** & **Evol** \\ \hline Seed Sample & Formulate an answer to this elaborate question: Where is the regional airline based that Jim French is the former chairman and CEO of? & \\ \hline evolved (Turn 1) Instr. & Formulate an answer to this elaborate question: Identify the base location of the regional airline for which Jim French served as the former chairman and CEO, and explain the significance of this location in the context of the airline’s regional operations. & Combine Facts \\ \hline GPT-4 Response & Jim French served as the former chairman and CEO of Flybe, a British regional airline. The base location of Flybe was Exeter International Airport in Devon, England. & Yimbine Facts \\  & The significance of Exeter International Airport as the base location for Flybe lies in its strategic positioning within the United Kingdom. Exeter is situated in the southwest region of England, which allowed Flybe to efficiently connect various regional destinations across the UK and Europe. This location provided a central hub for operations, facilitating shorter flight routes and quick turnaround times, which are essential for regional airlines aiming to offer frequent and reliable services. Additionally, having a base in Exeter helped Flybe tap into a market with fewer competing airlines, allowing it to establish a strong regional presence and foster economic growth within the southwest of England by improving connectivity and accessibility. & Adjust Output Format \\ \hline Turn 2 & Please adjust the output to provide the answer in a concise text-only format. & Adjust Output Format \\ \hline GPT-4 Response & Jim French served as the former chairman and CEO of Flybe, a British regional airline based at Exeter International Airport in Devon, England. The significance of this location lies in its strategic positioning in the southwest of England, enabling efficient regional connectivity across the UK and Europe and fostering economic growth in the region. & \\ \hline Turn 3 & Now, rephrase your response as if you were an enthusiastic travel blogger sharing fascinating details with your readers. & Persona Rewriting Follow Up \\ \hline GPT-4 Response & Hey travel enthusiasts! Did you know that Jim French, the former chairman and CEO of Flybe, once led this fantastic British regional airline based at Exeter International Airport in beautiful Devon, England? It’s such a cool spot! Nestled in the southwest of England, Exeter’s prime location made it perfect for Flybe to connect countless regional destinations across the UK and Europe. This strategic hub wasn’t just about convenience; it played a crucial role in boosting regional connectivity and economic growth. So, next time you’re flying through Exeter, remember the incredible role it played in the world of regional air travel! & \\ \hline \hline \end{tabular}
\end{table}
Table 18: Conversation from _M2Lingual_



\begin{table}
\begin{tabular}{c l c} \hline \hline
**Step** & **Content** & **Evol** \\ \hline Seed Sample & Given the following statement, predict the tags for each word: Daniel Smith daniel.smith2@durham.ac.uk Durham University, United Kingdom & \\ \hline evolved (Turn 1) Instr. & Identify the appropriate tags for each element in this data: Daniel Smith daniel.smith2@durham.ac.uk Durham University, United Kingdom. Please provide a brief response. & Concise \\ \hline GPT-4 Response & Name: Daniel Smith Email: daniel.smith2@durham.ac.uk Institution: Durham University & \\  & Country: United Kingdom & \\ \hline Turn 2 & Now, please reformat the output to display only the tags and associated & Adjust Output Format elements in a structured JSON format. \\ \hline GPT-4 Response & \{ ”Name”: “Daniel Smith”, ”Email”: ”daniel.smith2@durham.ac.uk”, ”Institution”: ”Durham University”, ”Country”: ”United Kingdom” } \\ \hline \end{tabular}
\end{table}
Table 19: Conversation from _M2Lingual_



\begin{table}
\begin{tabular}{c l c} \hline \hline
* Use a belt sander to remove the spare plastic & Random Follow Up \\  &
- Use a coffee grinder to remove the spare plastic & Random Follow Up \\  & Answer: & Random Follow Up \\ \hline evolved (Turn 1) Instr. & Complete the sentence with the most appropriate choice: To prevent the documents from flying around in a windy environment, you can & Object Properties \\  & Choices: & Random Follow Up \\  &
- Use a heavy paperweight to keep them in place & Random Follow Up \\  &
- Use a feather to keep them in place & Random Follow Up \\  & Answer: & Random Follow Up \\ \hline GPT-4 Response & Use a heavy paperweight to keep them in place & Random Follow Up \\ \hline Turn 2 & Now, generate a paragraph explaining the pros and cons of using a heavy paperweight to keep documents in place in windy conditions. & Challenging Follow Up \\ \hline GPT-4 Response & Using a heavy paperweight to secure documents in a windy environment has several advantages. First, the weight of the paperweight can effectively prevent papers from being blown away, ensuring they stay in the designated area. It also allows for easy accessibility and organization of the documents as they can be quickly retrieved without the hassle of chasing loose papers. In addition, paperweights are available in various designs and materials, providing aesthetic appeal to your workspace. & However, there are also some potential drawbacks. The primary disadvantage is the risk of damage to the documents. If the paperweight is too heavy or it is dropped accidentally, it could tear or crumple the papers. Furthermore, paperweights take up desk space and can be inconvenient if you’re working with a limited area. Lastly, if the wind is extremely strong, even a heavy paperweight might not be enough to keep the documents in place. & Random Follow Up \\ \hline Turn 3 & Now, transform your previous response into a haiku format. The haiku should have a 5-7-5 syllable structure. Reconstruct your answer if necessary to fit this format. & Random Follow Up \\ \hline GPT-4 Response & Paperweight holds firm, & Random Follow Up \\  & Yet may damage precious sheets, & Random Follow Up \\  & In strong gusts, it fails. & Random Follow Up \\ \hline \hline \end{tabular}
\end{table}
Table 20: Conversation from _M2Lingual_




\begin{table}
\begin{tabular}{c l p{284.5pt}} \hline
**Step** & **Content** \\ \hline Seed Sample & Quelle pourrait ê la principale contribution de Dante, Boccacio et Petrach à la langue italienne? Etant donné la question precedente, écrivez un contexte qui contient la réponse. Cela peut être de 1 a 20 phrases. Le contexte: \\ \hline Evolved (Turn 1) Instr. & Comment les eeuvres de Dante, Boccace et Pétrarque ont-elles influucès non seulement la longue latienne, mais aussil la philosophie et la litterature européennes ? Erant dontôné cette question, croirze un contexte que ionient la rêponse. Cela peut êre de 1 a 20 phrases. Le contexte: \\ \hline GPT-4 Response & Les euvres de Dante, Boccace et Pétrarque ont profondement marquée la langue italienne et ont eu une influence durable sur la philosophie et la litterature européennes. Dante Alighieri, avec sa “Divine Comédie”, a non seulement consolidé l’usage du tossan comme language literaire, mais a gélament introduite des thmes philosopiqueses of thetophiques d’une grande profondeau, infuemoat la penasse mediavel et rennaissance. Giovanni Bocacca, a travers son ”Décameron", a enrichi la prose tailienne et a offernt un aperex critique et souvent satirique de la société de son temps, ouvarant la vovie a une litterature plus realistete at humaniste. Francesco Pértarque, avec ses somnets et poessies en langue vernacularaine, a non seulement perfectionen la forme du sooner, mais aussit contribuble h ’ideal de l'humanisme, valoraiant l’etude des textes classiques el taquet de la connaisance personnelle. Ensemble, es antevenus ont jetées les bases de la Rennaissance talienne ent not influucèdes des ecrivains et des penseurs a travers toute l’Europe, de Chaucer à Shakespeare, en passant par Montaigne et Cervantés, renforcant l’ide que la litterature peut ê un moyen puissant d’exploration philosophyique et d’expression culturel. \\ \hline Turn 2 & Maintenant, en vous basant sur le contexte fourni, powez-vous détailler comment les oeuves de cecs trois auteurs ont spécifiquement infinluence la Rennaissance italianene ? \\ \hline GPT-4 Response & Les euvres de Dante, Boccace et Pétrarque ont joule un role central dans le developpement de la Renaussance latialenne en redefinissamt les normes litteraires, linguisiques et philosophiques de l’epaque. Dante Alighieri, avec sa “Divine Comédie”, a non seulement demontre que le toscan pouvari ê une langue litériare de premier ordorie, mais a gélagement explore des themes universels comme la justice, la redemption et l’amour divin, qui sont devenus des preoccupations centales de la pense renaissance. Son usage du toscana a contribue à l’unification linguisique de l’Italie, jiant les bases d’une langue nationale. Givoxanni Boccace, par le biais de son ”Decameron", a offernt une vision plus terrestre et realistic de la soccite, abordant des themes teles que l’amour, la fortune ct l’intelligence humaine. Ses histosires, souvent Innovatiques et critiques, on encourage une approche plus humaniste et emprujque de la viie, en reaction aux dogmeg rigtides du Moyen Age. Boccace a gélament influencé la posse narrative, rendant les histotirees accessibles à un public plus large et diversité. Francescoe Pétrarapuate, quant à liu, est souvent considère comme le pierre de l'humansime. Ses euvres, nostamment von ”Canzoniere”, ont popularise le somnet et ont mis l’accent sur l’expression individiuelle et la quote de la connaisances personnelle. Pétrarque a redecouverf et valoriséles les textes classiques, inspirant une redecouverte des evurves de l'Antiquitie grocro-romaine, ce qui est une pierre angulaire de la RenasiANCE. Son admiration pour la culmez classique a neconomage les intellectats is rajevateur leurs propress traditions et a chercher à les harmoniser avec les ideaux analogues. Ensmemble, ces autevents ont transformé la culture italianae et ont ouvert la vovie à un movvement intellectualet et artistique qui allair revolutionner I’Europe, one metant l’accent sur la digitie de l'homme, l’importance de la raison, et la beauche de la langue et de la litterature. Leurs courves ont servi de modeles et de sources d’insertion pour les artistes, écrixaines et penseurs de la Renasiissance, consolidation amisileur place commune pliers de ce mouvement cuthrel et intellectel. \\ \hline \end{tabular}
\end{table}
Table 21: Conversation from _M2Lingual_ in Dutch



\begin{table}
\begin{tabular}{c l c} \hline \hline
**Step** & **Content** & **Evol Type** \\ \hline Seed Sample & Continuate a scrivere il seguente teso: Oliver ha colpito Bishop con tanta forza che il vecchio si e’stravolto all’indietro e’caduto sul lato del trono; poi Oliver si e'spostato sul late di Amelie. “no!” Oliver ha fatto un salto mentre Sam si aggrappava del palo per tiarulo fuori. “lei eVecchia." & C’era una volta un villaggio chiamato Lanciano, famoso per le sue feste. Un giorno, il pi�� coraggioso tra gli abitanti, un uomo chiamato Oliver, decise di sidare il vecchio e autoritario leader, Bishop. Cosi, durante la festa pi



\begin{table}
\begin{tabular}{c l c} \hline
* 3:21 & 2. “Oh My Darling Donf Cry”
- 3:24 & 3. “Blockobuster Night, Part 1”
- 2:32 & 4. “Close Your Eyes (And Count to Fuck)” con Zack de la Rocha
- 3:54 & 5. “All My Life”
- 3:08 & 6. “Lie, Cheat, Steal”
- 3:29 & 7. “Early” con BOOTS
- 3:44 & 8. “All Due Respect” con Travis Barker
- 3:40 & 9. “Love Again (Akinyle Back)“ con Gangsta Boo
- 3:02 & 10. “Crown” con Diane Coffee
- 3:48 & 11. “Angel Duster”
- 5:12 & \\ \end{tabular}
\end{table}
Table 23: Conversation from _M2Lingual_ in Spanish.


* **Step**: **Content**
* Bei Menschenk kann ein Mekmal durch cin Pan der vide Vriee Parer von was bestimmt werden? Wählen Sie ans den folgenden Optienn und begintende Sihe Hare Antwort deutillier. IGenz 2. Mikchsen 3. Zeldnun 4. Ozyine
* **Dei Menschkunn kan ein Mekmal durch cin Pan der vide Vriee Paire van waes bestimant werden? Walsh Sie ans den folgenden Organisation und begintende Sihe Hare Antwort deutillier. Jaffeen 5. Ozyine
* Chremosungen, die in Patane vorokommen, wobe et Chemonson von der Mutter and eines Vanvee versteub widl Veke Merinkante wertu von mehreren Greats (polyvere Viewthrough) beeinfunket, die in Kamplever Weise mittelander tunzigerren.
* **Mikroben++: Mikroben siad Mikroganismen, wie Baknetzen, Viem und Flaze. Obsohl Mixtroben cine Relfe der Gesenhelt und Kernisch von Messich norglich kornsen, beltem ne keirigten Oberstand Mechanie eines Monschen. Sei kornsen jedoch alts Immunigen beeinfunkent und zur Fuzzasibne tremerzhang betrigen.**Zelaus nur Zellaus nur Zellaus nur Zellaus nur Zellaus nur Zellaus nur Zellaus er 



\begin{table}
\begin{tabular}{c l p{142.3pt}} \hline \multicolumn{1}{l}{**Step**} & **Content** \\ \hline  & Repondez à cette question: On mesure un tapis neuf pour une piece de 5,5 m sur 4,6 m. Sur un ruban à mesurer, quelle est la plus petite unité necessaire pour mesurer la pièce avec preéision pour le tapis? Selectionenz parmi: (1) en millières. (2) centimetres. (3) a des metres. (4) kilometres. (5) milliêtres. (6) decamétrers. Réponse: \\ \hline Evolved (Turn 1) Instr. & Repondez à cette question: On mesure un tapis neuf pour une pièce de 5,5 m sur 4,6 m. Sur un ruban à mesurer, quelle est la plus petite unité necessitaire miliformes. (2) centimetres. (3) a des metres. (4) kilometres. (5) millières. RÉponse: \\ \hline GPT-4 Response & Port mesurer une pièce de 5,5 m sur 4,6 m avec préciision pour un tapis, la plus petite unité necessaire seranti le *millimetre**. Les milliêtres permettent une mesur treès precise aui esuvent necesaire pour des Ajustements fins dans des projets comme l’installation de tapis. \\  & Ainsí, la réponse est : **(5) milliêtres**. \\ \hline Turn 2 & Maintenant, reformulez votre réponse sous forme de listé à puces. & Format Follow-Up \\ \hline GPT-4 Response & - Pour mesurer une pièce de 5,5 m sur 4,6 m avec préciision pour un tapis, la plus petite unité necessaire serait le *milliêtres**. - Les milliêtres permettent une mesure tès precise qui est souvent necesaire pour des Ajustements fins dans des projetes comme l’installation de tapis. \\  & Ainsí, la réponse est : **(5) milliêtres**. \\ \hline Turn 3 & Quavez-vous expliquer pourquoi vousa avez choisi cette unité de mesure en particulier pour mesurer la pièce avec précision pour le tapis ? Clarification \\ \hline  & Bien stir, voici une explication détailée : \\  & - **Péécision nécessaire pour le tapis** : Lors de l’installation d’un tapis, il est crucial de mesurer la pièce avec une grande précision pour s’assurer que le tapis ’adapte parfaitemed sams laisser de videos ni nécessiter de grandes découpes supplementaires. \\  & - **Milliêtres versus auttes** : - - ***Milliêtres** : Ilst offrent la plus grande précision parmi les options donnees. Choaque milliêtente éunt un millière de mettre, il permet de capturer les dimensions exactises pour anjustern partifat. \\  & - **Centientiretées** : Bién que précis, les centimettes (1 cm = 10 mm) ne sont passauss! récis que les milliêtres et pourraient omtetre des details fins nécessaires à une coupe parfaite. - *Attérés** : Mesurer en metres (1 m = 1000 nm) serait trop grossier pour des dimensions assui presicuses ct fines. - *Decameutres** : et **kilométrees** : Cse unites sont beaucoup trop grandes pour une tâche de cette energeurg et ne sont pas pratiques pour des mesures de precision dans un contexete domestique. - *Multiiêmes** : Si



\begin{table}
\begin{tabular}{l l p{142.3pt}} \hline
**Evol Type** & **GPT-4 Prompt** \\ \hline \multirow{2}{*}{Adding Distractors} & Given a prompt \#given_prompt@ that represents an article to be summarized, based upon the \#given_prompt@ create a \#new_prompt@ in the same language that adds unrelated or distracting information in the article which is not relevant to the main topic: \#given_prompt@ <prompt> \\ \hline Technical Jargon & Given a prompt \#given_prompt@ that represents an article to be summarized, based upon the \#given_prompt@ create a \#new_prompt@ in the same language by adding technical jargon or industry specific terms that makes it difficult to summarize. \#given_prompt@ <prompt> \\ \hline Inconsistencies In Information & Given a prompt \#given_prompt@ that represents an article to be summarized, based upon the \#given_prompt@ create a \#new_prompt@ in the same language to include contradictions of inconsistencies within the article thus forcing the summarizer to discern which piece of information is accurate and relevant. \#given_prompt@ <prompt> \\ \hline Multiple Topics & Given a prompt \#given_prompt@ that represents an article to be summarized, based upon the \#given_prompt@ create a \#new_prompt@ in the same language that covers multiple topics or subtopics thus make summarization more complicated. \#given_prompt@ <prompt> \\ \hline Metaphors Idiom & Given a prompt \#given_prompt@ that represents an article to be summarized, based upon the \#given_prompt@ create a \#new_prompt@ simple metaphors, idioms and cultural references thus making summarization more challenging. \#given_prompt@ <prompt> \\ \hline Long Distance & Given a prompt \#given_prompt@ that represents an article to be summarized, based upon the \#given_prompt@ create a \#new_prompt@ by increasing the distance between related pieces of information in the text as this requires understanding the deeper structure of the text. \#given_prompt@ <prompt> \\ \hline Multiple Languages & Given a prompt \#given_prompt@ that represents an article to be summarized, based upon the \#given_prompt@ create a \#new_prompt@language+) using the task challenging as it requires understanding and proficiency in more than one language. \#given_prompt@ <prompt> \\ \hline Unstructured Data & Given a prompt \#given_prompt@ that represents an article to be summarized, based upon the \#given_prompt@ create a \#new_prompt@ by presenting the article in a non-linear or non-chronological format thus increasing the complexity as it becomes challenging to pick out the main points and summarize them accurately. \#given_prompt@ <prompt> \\ \hline Personal Opinion & Given a prompt \#given_prompt@ that represents an article to be summarized, based upon the \#given_prompt@ create a \#new_prompt@ incorporating bias or personal opinion as it greatly complicates the summarization process as the summarizer needs to remain neutral and objective. \#given_prompt@ <prompt> \\ \hline  & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 26: Generic

\begin{table}
\begin{tabular}{l c} \hline
**Eval Type** & **GPT-4 Prompt** \\ \hline Transliteration & Rewrite the \#given_prompt@ as an \textless{}translit_language\textgreater{} translation, and create \#new_prompt@. Additionally, conclude with a request to respond in \#translit_language\textgreater{} translation. \#given_prompt@ <prompt> \\ \hline Dialet & You are a brilliant \textless{}prompt_language\textgreater{} native speaker. Rewrite \#given_prompt@ by changing the dialect to \textless{}prompt_language\textgreater{} and create \#new_prompt@ Finally, ask to respond in the same \textless{}prompt_language\textgreater{} dialect. Write the \#new_prompt@ prompt prompt in \(\textless{}prompt_language\textgreater{}, given_prompt@; <prompt> \\ \hline Concise & Re-write the \#given_prompt@ concisely, and create \#new_prompt@ Additionally, conclude with a request to respond \#concibly. Write the \#new_prompt@ \textless{}prompt_language\textgreater{}, \#given_prompt@ <prompt> \\ \hline Deepen & Slightly increase the depth and breadth of \#given_prompt@, and create \#new_prompt@ Write the \#new_prompt@ in \textless{}prompt_language\textgreater{}, \#given_prompt@ <prompt> \\ \hline Concetize & Make \#given_prompt@ slightly more concrete, and create \#new_prompt@ Additionally, conclude with a request for an AI assistant to respond with the a detailed and concrete response. Write the \#new_prompt@ in \textless{}prompt_language\textgreater{}, \#given_prompt@ <prompt> \\ \hline Increase Reasoning & If \#given_prompt@ can be solved with just a few simple thinking processes, rewrite it to explicitly request multi-step reasoning, and create \#new_prompt@ Write the \#new_prompt@ in \textless{}prompt_language\textgreater{}, \#given_prompt@ <prompt> \\ \hline \hline \end{tabular}
\end{table}
Table 27: Abstract Summarization



\begin{table}
\begin{tabular}{l l p{142.3pt}} \hline
**Evol Type** & **GPT-4 Prompt** \\ \hline  & & Given a prompt \#given_prompt@ that asks an explanation of a joke, based upon the \#given_prompt@ create a _free_prompt_ in the same language by incorporating jaron to the jokes that are specific to a certain profession, field, or hobby thus requiring deeper knowledge of the field in order to explain the joke property. \#given_prompt@: <prompt> \\ \hline Slang & & Given a prompt \#given_prompt@ that asks an explanation of a joke, based upon the \#given_prompt@ create a _free_prompt_ in the same language that she slang or colloquial language, thus making it harder to understand and explain the punchline. \#given_prompt@; <prompt> \\ \hline Cultural References & & Given a prompt \#given_prompt@ that asks an explanation of a joke, based upon the \#given_prompt@ create a _free_prompt_ in the same language that increasingly use jokes which are culture-specific as will require cultural understanding tp provide explanations. \#given_prompt@: <prompt> \\ \hline Non Explicit Punchline & & Given a prompt \#given_prompt@ that asks an explanation of a joke, based upon the \#given_prompt@ create a _free_prompt_ in the same language such that jokes have a punchline isn’t explicitly stated, but rather implied. \#given_prompt@; <prompt> \\ \hline Time Bound & & Given a prompt \#given_prompt@ that represents an article and a question related to that article, based upon the \#given_prompt@ create a _free_prompt_ by making jokes that were relative to a certain time period or current event, thus making it harder to grasp. \#given_prompt@; <prompt> \\ \hline Compound Jokes & & Given a prompt \#given_prompt@ that asks an explanation of a joke, based upon the \#given_prompt@ create a _free_prompt_ in the same language such that it uses compound jokes which contain multiple punchlines within the same joke. This would make the explanation task difficult as one would need to explain multiple punchlines coherently, \#given_prompt@; <prompt> \\ \hline Language & & Given a prompt \#given_prompt@ that asks an explanation of a joke, based upon the \#given_prompt@ create a _free_prompt_ in \(\triangle\)language,\(\geq\) such that it challenge the language skills. This would make the explanation task difficult as one would need to explain multiple punchlines coherently. Finally ask to respond with explanation in \(\triangle\)language,\(\gtrsim\)1\% given_prompt@; <promptO* \\ \hline Long & & Given a prompt \#given_prompt@ that asks an explanation of a joke, based upon the \#given_prompt@ create a _free_prompt_ in the same language but make the jokes exceedingly long where the punchline isn’t delivered immediately and requires you to remember or understand preceding parts of the joke. \#given_prompt@: <prompt> \\ \hline Double Entendre & & Given a prompt \#given_prompt@ that asks an explanation of a joke, based upon the \#given_prompt@ create a _free_prompt_ in the same language but utilize jokes with double entendre, where there are two possible interpretations. \#given_prompt@: <prompt> \\ \hline \end{tabular}
\end{table}
Table 28: Joke Explain

\begin{table}
\begin{tabular}{p{79.5pt} p{106.5pt}} \hline
**Evol Type** & **GPT-4 Prompt** \\ \hline More Choices & Given a prompt \#given_prompt@ that represents a topic and multiple choice question related to that topic, based upon the \#given_prompt/ create a \#new_prompt by changing adding more choices to the question that are relevant to the topic but not correct. This will make it challenging to answer. \#given_prompt@: <prompt> \\ \hline More Complex & Given a prompt \#given_prompt@ that represents a topic and multiple choice question related to that topic, based upon the \#given_prompt@ create a \#new_prompt by making it more complex using technical or domain specific jargong This will make it challenging to understand the question thus making it difficult to answer. \#given_prompt@: <prompt> \\ \hline Negation & Given a prompt \#given_prompt@ that represents a topic and multiple choice question related to that topic, based upon the \#given_prompt@ create a \#new_prompt by asking negative questions that require the recognition of the negation included in the sentences. \#given_prompt@: <prompt> \\ \hline Multiple Correct Options & Given a prompt \#given_prompt@ that represents a topic and multiple choice question related to that topic, based upon the \#given_prompt@ create a \#new_prompt by changing adding more choices to the question that are correct and relevant to the topic. This will make it challenging as one will need to choose all correct options. \#given_prompt@: <prompt> \\ \hline Need For Context & Given a prompt \#given_prompt@ that represents a topic and multiple choice question related to that topic, based upon the \#given_prompt@ create a \#new_prompt by asking questions that require additional context other than the one provided in the topic.This will make it challenging as it will evaluate the knowledge someone has on the topic. \#given_prompt@: <prompt> \\ \hline Justification & Given a prompt \#given_prompt@ that represents a topic and multiple choice question related to that topic, based upon the \#given_prompt@ create a \#new_prompt@ is asking to respond with the correct answer and provide a detailed justification for the answer. \#given_prompt@: <prompt> \\ \hline Distracting Long & Given a prompt \#given_prompt@ that represents a topic and multiple choice question related to that topic, based upon the \#given_prompt@ create a \#new_prompt by asking questions that require additional context other than the one provided in the topic. This will make it challenging as it will evaluate the knowledge someone has on the topic. \#given_prompt@: <prompt> \\ \hline Close Choices & Given a prompt \#given_prompt@ that represents a topic and multiple choice question related to that topic, based upon the \#given_prompt@reading from predicting whether this process is able to predict. This will make the task more challenging. \#given_prompt@: <prompt> \\ \hline No Correct Option & Given a prompt \#given_prompt@ that represents a topic and multiple choice question related to that topic, based upon the \#given_prompt@ create a \#new_prompt by changing the choices such that no choice is the correct answer. \#given_prompt@: <prompt> \\ \hline \end{tabular}
\end{table}
Table 29: Flan Qa



**Evol Type** & **GPT-4 Prompt**

\begin{tabular}{l p{170.7pt}} \hline \hline
**Evol Type** & **GPT-4 Prompt** \\ \hline Ambiguity & Given a prompt \#given_prompt@ that represents an article and multiple questions related to that article, based upon the \#given_prompt@ create a "new_prompt@ by changing the questions only in the same language but making it much more vague and ambiguous than making it not so straightforward to answer. \#given_prompt@ <prompt> \\ \hline Long Form Question & Given a prompt \#given_prompt@ that represents an article and multiple questions related to that article, based upon the \#given_prompt@ create a "new_prompt@ by changing the question only in the same language by making it longer i.e. formulating the questions in long and complex sentences thus requiring the system to decipher the main question. \\ \hline Multilingual & Given a prompt \#given_prompt@ that represents an article and multiple questions related to that article, based upon the \#given_prompt@ create a "new_prompt@ by changing the questions in clanguage.2+ and article in clanguage.1-, having different linguistic structure. Finally, ask to answer the question in the clanguage.2+ \#given_prompt@: <prompt> \\ \hline Combine Facts & Given a prompt \#given_prompt@ that represents an article and multiple questions related to that article, based upon the \#given_prompt@ create a "new_prompt@ by changing the questions only in the same language by combining multiple facts this making the questions more complex and requiring combining multiple facts to answer correctly. \#given_prompt@ prompts@ promptly@ promptsp \\ \hline Implicit Question & Given a prompt \#given_prompt@ that represents an article and multiple questions related to that article, based upon the \#given_prompt@ create a "new_prompt@ by changing the questions only in the same language by asking implicit questions where the answer isn't explicit and requires understanding of the underlying implication. \#given_prompt@ <prompt> \\ \hline Negative Questions & Given a prompt \#given_prompt@ that represents an article and multiple questions related to that article, based upon the \#given_prompt@ create a "new_prompt@ by asking negative questions in the same language that require the recognition of the negation included in the sentences. \#given_prompt@ <prompts> \\ \hline Inference & Given a prompt \#given_prompt@ that represents an article and multiple questions related to that article, based upon the \#given_prompt@ create a "new_prompt@ by changing the question on the same language by asking questions that require a degree of inference or detection not directly provided. \#given_prompt@ <prompt> \\ \hline Multichoice Questions & Given a prompt \#given_prompt@ that represents an article and multiple questions related to that article, based upon the \#given_prompt@ create a "new_prompt@ by asking multiple choice questions in the same language based upon the given article, where the choices are from the article itself. Finally ask the model to respond with the correct choice and explain the decision. \#given_prompt@ <prompt> \\ \hline More Reasoning & Given a prompt \#given_prompt@ that represents an article and multiple questions related to that article, based upon the \#given_prompt@ create a "new_prompt@ by asking questions in the same language that require multistep reasoning processes, where participants need to follow a sequence of logical steps to arrive at the correct answer. \#given_prompt@ <prompt> \\ \hline \hline  & \\ \end{tabular} Table 31: Flan Coqa



\begin{table}
\begin{tabular}{l l p{142.3pt}}
**Evol Type** & **GPT-4 Prompt** \\ \hline \multirow{2}{*}{Cross Lingual} & Given a prompt \#given_prompt\# that asks to generate a question for a certain text snippet or a topic, based upon the \#given_prompt\#: ccreate a “new_prompt” in clanguage_2> and also asking to respond in clanguage_1>. \#given_prompt\#: c’promo\% \\ \hline Domain Language & Given a prompt \#given_prompt\# that asks to generate a question for a certain text snippet or a topic, based upon the \#given_prompt create a “new_prompt” in the same language by introducing domain specific language and related to the specialized field of topic described in the article; “given_prompt\#: cpompmp\textgreater{} \\ \hline Emotional Subtext & Given a prompt \#given_prompt\# that asks to generate a question for a certain text snippet or a topic, based upon the \#given_prompt\# create a “new_prompt\# in the same language by including sarcasm, euphemism, or other nuanced forms of communication thus make it harder to determine the possible question for the topic. \#given_prompt\#: cprompt\textgreater{} \\ \hline Need For Context & Given a prompt \#given_prompt\# that asks to generate a question for a certain text snippet or a topic, based upon the \#given_prompt\#: a“new_prompt\# in the same language that requires additional context or background to determine the relevant question for the text snippet, \#given_prompt\#: cprompt\textgreater{} \\ \hline Ambiguity In Wording & Given a prompt \#given_prompt\# that asks to generate a question for a certain text snippet or a topic, based upon the \#given_prompt\# create a “new_prompt\#: the same language by adding ambiguity to the text snippet thus making it more challenging to come up with a relevant question. \#given_prompt\textless{} cprompt\textgreater{} \\ \hline Long Text & Given a prompt \#given_prompt\# that asks to generate a rotation for a certain text snippet or a topic, based upon the \#given_prompt\# create a “new_prompt\# the same language by detailing and make the text snippet much longer thus making it more challenging to come up with a relevant question. \#given_prompt\#: cprompt\textgreater{} \\ \hline Idiom Slang & Given a prompt \#given_prompt\# that asks to generate a question for a certain text snippet or a topic, based upon the \#given_prompt\# create a “new_prompt\# by the use of atomic expressions or regional slang thus obscuring the meaning of text snippet and making it more challenging to come up with a question. \#given_prompt\#: cprompt\textgreater{} \\ \hline Adding Distractors & Given a prompt \#given_prompt\# that represents an article to be summarized, based upon the \#given_prompt\# create a \#new_prompt\# in the same language that adds unrelated or distracting information in the article which is not relevant to the main topic. \#given_prompt\#: cprompt\textgreater{} \\ \hline Multiple Questions & Given a prompt \#given_prompt that represents an article to be summarized, based upon the \#given_prompt\#: cprompt\textgreater{} \\  & Given a prompt \#given_prompt\#: that represents an article to be summarized, based upon the \#given_prompt\#: cprompt\textgreater{} \\ \end{tabular}
\end{table}
Table 33: Answer Ranking



\begin{table}
\begin{tabular}{l l}
**Evol Type** & **GPT-4 Prompt** \\ \hline Ambiguity & Given a prompt \#given_promppt that represents an article and a question related to that article, based upon the \#given_prompt* create a \#new_prompt by changing the question only in the same language but making it much more vague and ambiguous thus making it not so straightforward to answer. \#given_prompt*: cpmompc> \\ \hline Long Form Question & Given a prompt \#given_prompt* that represents an article and a question related to that article, based upon the \#given_prompt* create a \#new_prompt* by changing the question only in the same language by making it longer i.e. formulating the questions in long and complex sentences thus requiring the system to decipher the main question. \#given_prompt*: cpmcompp> \\ \hline Multilingual & Given a prompt \#given_prompt* that represents an article and a question related to that article, based upon the \#given_prompt* create a \#new_prompt by changing the question in clanguage_2> and article in clanguage_1>, having different linguistic structure. Finally, ask to answer the question in the clanguage_2> \#given_prompt*: cpmompc> \\ \hline Combine Facts & Given a prompt \#given_prompt* that represents an article and a question related to that article, based upon the \#given_prompt* create a \#new_prompt* by changing the question only in the same language by combining multiple facts thus making the question more complex and requiring combining multiple facts to answer correctly. \#given_prompt*: cpmompc> \\ \hline Implicit Question & Given a prompt \#given_prompt* that represents an article and a question related to that article, based upon the \#given_prompt* create a \#new_prompt* by changing the question only in the same language by asking implicit question where the answer isn’t explicit and requires understanding of the underlying implication. \#given_prompt*: cpmompc> \\ \hline Negative Questions & Given a prompt \#given_prompt* that represents an article and a question related to that article, based upon the \#given_prompt* create a \#new_prompt by changing the question only in the same language by asking negative questions that require the recognition of the negation included in the sentences. \#given_prompt*: cpmompc> \\ \hline Inference & Given a prompt \#given_prompt* that represents an article and a question related to that article, based upon the \#given_prompt* create a \#new_prompt* by changing the question only in the same language by asking questions that require a degree of inference or deduction not directly provided. \#given_prompt*: cpmompc> \\ \hline Multichoice Questions & Given a prompt \#given_prompt* that represents an article and a question related to that article, based upon the \#given_prompt* create a \#new_prompt by asking multiple choice question based upon the given article, where the choices are from the article itself. Finally ask the model to respond with the correct choice and explain the decision. \#given_prompt*: cpmompc> \\ \hline More Reasoning & Given a prompt \#given_prompt* that represents an article and a question related to that article, based upon the \#given_prompt* create a \#new_prompt* by asking that require multistep reasoning processes, where participants need to follow a sequence of logical steps to arrive at the correct answer. \#given_prompt*: cpmompc> \\ \hline \end{tabular}
\end{table}
Table 34: Mintaka

\begin{table}
\begin{tabular}{l p{195.1pt}}
**Evol Type** & **GPT-4 Prompt** \\ \hline Adding Distractors & Given a prompt \#given_prompt* that represents an article to be summarized, based upon the \#given_prompt* create a \#new_prompt* in the same language that adds unrelated or distracting information in the article which is not relevant to the main topic. \#given_prompt* <prompt* \\ \hline Technical Jargon & Given a prompt \#given_prompt* that represents an article to be summarized, based upon the \#given_prompt* create a \#new_prompt* in the same language by adding technical jargon or industry specifc terms that makes it difficult to summarize. \#given_prompt*: cpmcompp> \\ \hline Inconsistencies In Information & Given a prompt \#given_prompt* that represents an article to be summarized, based upon the \#given_prompt* create a \#new_prompt* in the same language to include contradictions or inconsistencies within the article thus forcing the summarizer to discern which piece of information is accurate and relevant. \#given_prompt*: cpmcompp> \\ \hline Multiple Topics & Given a prompt \#given_prompt* that represents an article to be summarized, based upon the \#given_prompt* create a \#new_prompt* in the same language that covers multiple topics or subtopes thus make summarization more complicated. \#given_prompt*: cpmcompp> \\ \hline Metaphors Idiom & Given a prompt \#given_prompt* that represents an article to be summarized, based upon the \#given_prompt* create a \#new_prompt* using complex metaphors, idioms and cultural references thus making summarization more challenging. \#given_prompt*: cpmcompp> \\ \hline Long Distance & Given a prompt \#given_prompt* that represents an article to be summarized, based upon the \#given_prompt* create a \#new_prompt* by increasing the distance between related pieces of information in the text as this requires understanding the deeper structure of the text. \#given_prompt*: cpmcompp> \\ \hline Cross Lingual Summary & Given a prompt \#given_prompt* that represents some article about a topic, rewrite the article only and create a \#new_prompt* in any clanguage_2>. Finally ask to provide a summary in the clanguage_1> \#given_prompt*: cpmcompp> \\ \hline Unstructured Data & Given a prompt \#given_prompt* that represents an article to be summarized, based upon the \#given_prompt* create a \#new_prompt* by presenting the article in non-linear or non-chronological format thus increasing the complexity as it becomes challenging to pick out the main points and summarize them accurately. \#given_prompt*: cpmcompp> \\ \hline Personal Opinion & Given a prompt \#given_prompt* that represents an article to be summarized, based upon the \#given_prompt* create a \#new_prompt* by incorporating bias or personal opinion as it greatly complicates the summarization process as the summarizer needs to remain neutral and objective.. \#given_prompt*: cpmcompp> \\ \hline  & \multicolumn{1}{r}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 35: Cross Summarization



\begin{table}
\begin{tabular}{l l}
**Evol Type** & **GPT-4 Prompt** \\ \hline Complex Question & Given a prompt @given_prompt that requires composing a context around some question, based upon the @given_prompt create a @new_prompt in the same language by formulating the question in a more complex way, requiring deeper understanding, reasoning, and inferential abilities. @given_prompt: @grouppt> \\ Advanced Vocab & Given a prompt @given_prompt that requires composing a context around some question, based upon the @given_prompt create a@new_prompt in the same language by using more complex language and advanced vocabulary to increase increase the difficulty level, as it requires deeper understanding of language and words to compose a context. \\ Multiple Themes & Given a prompt @given_prompt that requires composing a context around some question, based upon the @given_prompt create a@new_prompt in the same language by asking question on multiple themes or topics thus making it harder to generate a context around all topics. @given_prompt#: @comppt> \\ Ambiguity & Given a prompt @given_prompt that requires composing a context around some question, based upon the @given_prompt create a @new_prompt in the same language by making question vague and ambiguous thus making it a little harder to compare a context around all topics. @given_prompt: @comppt> \\ More Details & Given a prompt @given_prompt that requires composing a context around some question, based upon the @given_prompt create a@new_prompt in the same language by adding more detail, domain specific knowledge and technical jargons to the question thus making it difficult to generate a context. @given_prompt: @comppt> \\ Increase Reasoning & Given a prompt @given_prompt that requires composing a context around some question, based upon the @given_prompt create a@new_prompt in the same language by asking question that are the intersection of multiple topic this require understanding of all topics and how the topics are related to each other. @given_prompt: @temporppo> \\ Time & Given a prompt @given_prompt that requires composing a context around some question, based upon the @given_prompt create a@new_prompt in the same language by asking question where the context or the answer changes over time, thus assessing how up to date someone is. @given_prompt: @comppt> \\ Abstract Topics & Given a prompt @given_prompt that requires composing a context around some question, based upon the @given_prompt create a@new_prompt in the same language by asking question where the context needs to be generated on some abstract topic where opinion varies from person to person. @given_prompt#: @comppt> \\ Structured Info & Given a prompt @given_prompt that requires composing a context around some question, based upon the @given_prompt create a@new_prompt# in the same language by asking question where the context should be generated in a structured form as bulleted list with topics and sub-topics. @given_prompt: @comppt> \\ \end{tabular}
\end{table}
Table 36: Adversarial Qa

\begin{table}
\begin{tabular}{l p{170.7pt}}
**Eval Type** & **GPT-4 Prompt** \\ \hline Genne Specific & Given a prompt @given_prompt that asks to write a story on a topic, based upon the @given_prompt create a @new_prompt in the same language but, that specifies a genre for the short story, such as science fiction, mystery, fantasy, or historical fiction. @given_prompt: @comppt> \\ Character Constraints & Given a prompt @given_prompt that asks to write a story on a topic, based upon the @given_prompt create a @new_prompt# in the same language but, that requires specific types of characters to be included, such as a detective, a mythical creative, or a historical figure. @given_prompt: @comppt> \\ Setting Restrictions & Given a prompt @given_prompt that asks to write a story on a topic, based upon the @given_prompt create a @new_prompt in the same language but, that limits the setting of the story to a specific location, time period, or environment, such as a futuristic city, the Wild West, or a remote island. @given_prompt#: @comppt> \\ Plot Twists & Given a prompt @given_prompt that asks to write a story on a topic, based upon the @given_prompt create a @new_prompt in the same language but, that introduces a plot twist requirement, such as an unexpected turn of events, a moral dilemma, or a reversal of fortune fine than charnicer. @given_prompt#: @comppt> \\ Narrative Style & Given a prompt @given_prompt that asks to write a story on a topic, based upon the @given_prompt create a @new_prompt# in the same language but, that specifies a narrative style or point of view, such as first-person, third-person limited, or epistaly (written as a series of letters). @given_prompt: @comppt> \\ Word Count Limit & Given a prompt @given_prompt that asks to write a story on a topic, based upon the @given_prompt create a @new_prompt in the same language but, that sets a word count limit for the short story to encourage concise and focused storyelling. @given_prompt: @comppt> \\ Incorporate Dialogue & Given a prompt @given_prompt that asks to write a story on a topic, based upon the @given_prompt# create a @new_prompt in the same language but, that requires meaningful dialogue between characters to develop plot, reveal character traits, orcreate tension. @given_prompt: @cprompt> \\ \hline Theme Integration & Given a prompt @given_prompt that asks to write a story on a topic, based upon the @given_prompt# create a @new_prompt# the same language but, that integrates a specific theme into the story, such as friendship, resilience, betrayal, or the passage of time. @given_prompt#: @comppt> \\ Include Symbolism & Given a prompt @given_prompt that asks to write a story on a topic, based upon the @given_prompt# create a @new_prompt# in the same language but, that encourages the use of symbolism or allegory to convey deeper meanings or themes within the story. @given_prompt#: @comppt> \\ \end{tabular}
\end{table}
Table 37: Soda



\begin{table}
\begin{tabular}{l l}
**Evol Type** & **GPT-4 Prompt** \\ \hline \multirow{2}{*}{Object Interaction} & Given a prompt \#given_prompt\# that involves commonsense physical reasoning, asking to finish a sentence with two possible options based upon the given_prompt\# create a “fnew_prompt\# in the same language that involves object interaction reasoning. \#given_prompt\#: cpcompp\% \\ \hline Object Properties & Given a prompt \#given_prompt\# that involves commonsense physical reasoning, asking to finish a sentence with two possible options based upon the given_prompt\# create a fnew_prompt\# in the same language that requires understanding and reasoning over the object properties. \#given_prompt\#: cpcompp\% \\ \hline Logical Sequencing & Given a prompt \#given_prompt\# that involves commonsense physical reasoning, asking to finish a sentence with two possible options based upon the given_prompt\# create a “fnew_prompt\# in the same language that requires logical sequence reasoning. \#given_prompt\#: cpcompp\% \\ \hline Object Transformation & Given a prompt \#given_prompt\# that involves commonsense physical reasoning, asking to finish a sentence with two possible options based upon the given_prompt\# create a “fnew_prompt\# in the same language that requires object transformation reasoning. \#given_prompt\#: cpcompp\% \\ \hline More Choices & Given a prompt \#given_prompt\# that involves commonsense physical reasoning, asking to finish a sentence with two possible options based upon the given_prompt\# create a “fnew_prompt\# in the same language by adding more options and asking to finish with all correct options. \#given_prompt\#: cpcompp\% \\ \hline Justification & Given a prompt \#given_prompt\# that involves commonsense physical reasoning, asking to finish a sentence with two possible options based on the given_prompt\# create a “fnew_prompt” in the same language by asking to give a detailed step-by-step justification of the chosen option. \#given_prompt\#: cpcompp\% \\ \hline Incorrect Choices & Given a prompt \#given_prompt\# that involves commonsense physical reasoning, asking to finish a sentence with two possible options based upon the given_prompt\# creating a “fnew_prompt\# in the same language by adding more options that are incorrect this making it difficult to identify correct option. \#given_prompt\#: cpcompp\% \\ \hline Double Negatives & Given a prompt \#given_prompt\# that involve commonsense physical reasoning, asking to finish a sentence with two possible options based upon the given_prompt\# create a “fnew_prompt\# in the same language with double negatives thus making it hard to understand and can increase the complexity of the task. \#given_prompt\#: cpcompp\% \\ \hline Theoretical Scenario & Given a prompt \#given_prompt\# that involves commonsense physical reasoning, asking to finish a sentence with two possible options based upon the given_prompt\# create a “fnew_prompt\# in the same language by making the base scenarios less straightforward and more abstract thus making the task more complex. \#given_prompt\#: cpcompp\% \\ \hline \end{tabular}
\end{table}
Table X8: Commonsense

\begin{table}
\begin{tabular}{l p{195.3pt}}
**Evol Type** & **GPT-4 Prompt** \\ \hline Idioms Phrases & Given a prompt \#given_prompt\# that represents two sentences and asks whether the two are paraphrases or not, based upon the \#given_prompt\# create a “fnew_prompt\# in the same language follows and phrases have meanings different from their literal meanings, using them for paraphrasing can add complexity. \#given_prompt\#: cpcompp\% \\ \hline Abbreviations & Given a prompt \#given_prompt\# that represents two sentences and asks whether the two are paraphrases or not, based upon the \#given_prompt\# create a“fnew_prompt\# in the same language by converting certain commonly known phrases or organizations into their abbreviated forms thus making identification more difficult. \#given_prompt\#: cpcompp\% \\ \hline Sentence Structure & Given a prompt \#given_prompt\# that represents two sentences and asks whether the two are paraphrases or not, based upon the \#given_prompt\# create a “fnew_prompt\# in the same language by increasing the complexity of sentences i.e. either rearranging the individual sentences, making use of passive and active voice or changing the sentence structural form. \#given_prompt\#: cpcompp\% \\ \hline Information & Given a prompt \#given_prompt\# that represents two sentences and asks whether the two are paraphrases or not, based upon the \#given_prompt\# create a “fnew_prompt\# in the same language by adding or subtracting relevant details from one sentence which do not change the main theme but add extra entities can make it challenging. \#given_prompt\#: cpcompp\% \\ \hline Variation & Given a prompt \#given_prompt\# that represents two sentences and asks whether the two are paraphrases or not, based upon the \#given_prompt\# create a “fnew_prompt\# in the same language by introducing variations in dialect, accent, slang, or colloquial language usage among these take text complex. \#given_prompt\#: cpcompp\% \\ \hline Negation & Given a prompt \#given_prompt\# that represents two sentences and asks whether the two are paraphrases or not, based upon the \#given_prompt\# create a “fnew_prompt\# in the same language by introducing negativities or double negatives, the meaning of the sentence could be the same but the formation different. \#given_prompt\#: cpcompp\% \\ \hline Time Navigation & Given a prompt \#given_prompt\# that represents two sentences and asks whether the two are paraphrases or not, based upon the \#given_prompt\# create a “fnew_prompt\# the same language by changing the time description (from past to present or future) in paraphrased sentences. \#given_prompt\#: cpcompp\% \\ \hline Cultural Inferences & Given a prompt \#given_prompt\# that represents two sentences and asks whether the two are paraphrases or not, based upon the \#given_prompt\# create a “fnew_prompt\# in the same language by using different cultural inferences in each sentence. The task gets complicated when two sentences infer same conclusion but uses culturally different examples or metaphors. \#given_prompt\#: cpcompp\% \\ \hline Length Variation & Given a prompt \#given_prompt\# that represents two sentences and asks whether the two are paraphrases or not, based upon the \#given_prompt\# create a “fnew_prompt\# in the same language by using different sentence length one can be short and another very long. \#given_prompt\#: cpcompp\% \\ \hline \end{tabular}
\end{table}
Table X9: Pawsx

* [25] M. C. Bader, R. J. Brox, and A. K. Sridharan, "A new approach to the automatic classification of deep neural networks for image recognition," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pp. 1-9, ACM, 2006.
* [26] D. P. Kingma and J. Ba, "Adam: a method for stochastic optimization," in _Advances in Neural Information Processing Systems (NIPS)_, vol. 34, no. 8, p. 709-718, 2014.
* [27] G. Hinton, I. Goodfellow, and Y. Bengio, "Deep learning with tiny data sets," in _Advances in neural information processing systems_, vol. 34, no. 1, pp. 137-146, 2014.

[MISSING_PAGE_POST]

 



### Prompt Taxonomy for Multiturn

Evol-instruct

## Appendix B Licenses

We adhere to Apache 2.0 License from Aya Dataset and Aya Collection and Terms of Use for GPT-4 when constructing our _M2Lingual_ dataset. We confirm that we bear the responsibility in the case of violation of rights and will take appropriate course of actions if needed. Our dataset is licensed through CC-by-NC-SA-4.0 license. The dataset will be hosted on HuggingFace datasets and maintained by the authors.





\begin{table}
\begin{tabular}{l l}
* The follow-up instruction must be challenging in terms of difficulty in comparison with the initial instruction. \\ \hline Ambiguous &
- The follow-up instruction must refer to the previous result obtained from the initial instruction in an ambiguous way (e.g., summarize that under 3 paragraphs...) \\ \hline Reddirection &
- The follow-up instruction must abruptly change the type of the request/task or the thematic/topic of the initial instruction with no transition formula (e.g., let’s shift gears) or even referring to the initial instruction. \\ \hline Generic Rewriting &
- The follow-up instruction must request a change in the \{property\} of the response to the INITIAL INSTRUCTION. \\ \hline Feedback Handling &
- The follow-up instruction must indicate that what the AI model responded to the INITIAL INSTRUCTION was not good enough (you must specify on which random aspect). \\ \hline Random &
- The follow-up instruction must request to change the response content or format in unique and unusual ways (e.g. switch to JSON or YAML or even a custom format illustrated by a template or very specific format description, keep all words starting with certain letter, remove every other word... You must specify this way in the instruction). \\ \hline Context Retention &
- The follow-up instruction must present a request/task that will test the ability of the model to retain the context of the conversation established by the previous instructions. \\ \hline Format Rewriting &
- The follow-up instruction must request a change in the format of the response to the previous instruction. \\ \hline Persona Rewriting &
- The follow-up instruction must request a change in the persona of the response to the previous instruction. \\ \hline Detailed Constraints &
- The follow-up instruction must add detailed constraints, like specifying the desired output format. Also involves providing more specific parameters or criteria to narrow down search results. Examples include specifying keywords, time ranges, locations, categories, or sources. \\ \hline Adjust Output Format &
- The follow-up instruction must ask to adjust the output format as users may request specific formats for the output, such as text-only, summarized results, or structured data formats. \\ \hline Expanding Queries &
- The follow-up instruction must ask to expand on a certain topic as users might want to broaden the search scope to include related topics or synonyms. \\ \hline Refocus Queries &
- The follow-up instruction must be a refocus query as users may wish to refocus the query to target a specific aspect or angle of their original request. \\ \hline Change Context &
- The follow-up instruction must introduce a new topic or context that is related to the current conversation, allowing the chatbot to provide a different perspective or information. \\ \hline Clartification &
- The follow-up instruction must ask for clarification as the chatbot may provide a complex or unclear response, ask for clarification to encourage it to expand on its answer. \\ \hline Chathot Opinion &
- The follow-up instruction must encourage the chatbot to provide its own perspective or opinion on a topic, which can help create a more dynamic and engaging conversation. \\ \hline Open Ended Questions &
- The follow-up instruction must ask open-ended questions that require more detailed and thoughtful responses, encouraging the chatbot to provide more information and keep the conversation going. \\ \hline Complex Queries &
- The follow-up instruction must ask to create a multi-part question or instruction and see how the chatbot manages to break down and answer each part. \\ \hline Pronouns &
- The follow-up instruction must ask a question that uses pronouns like “it,” ”he," or ”she” after some gap in the conversation. The bot should have to remember the noun the pronoun is referring to. \\ \hline Engaging Conversation &
- The follow-up instruction must engage the chatbot in a conversation about a topic that requires knowledge of previous interactions. \\ \hline Recall Information &
- The follow-up instruction must ask the chatbot to recall the details of the earlier turns in the conversation. \\ \end{tabular}
\end{table}
Table 41: Multiturn Evals


* [38] M. Cacciari, G. P. Salambi, and A. Zisserman, "The automated computation of the jet energy scale in multijet events: a review," _JHEP_**07** (2014) 065, doi:10.1007/JHEP07(2014)065, arXiv:1409.0022.
* [39] J. Alwall et al., "MadGraph -- a simulation toolkit for parton shower simulations," _Comput. Phys. Commun._**189** (2014) 1-14, doi:10.1016/j.cpc.2014.02.004, arXiv:1407.4083.
* [40] S. Alioli, R. Frederix, F. Piccinini, and L. Motani, "A study on the production cross section at hadron colliders using the POWHEG method," _Eur. Phys. J. C_**71** (2015) 155, doi:10.1140/epjc/s10052-015-3988-x, arXiv:1407.4083.
* [41] T. Sjostrand, S. Mrenna, and P. Skands, "PYTHIA 6.4 physics and manual", _JHEP_**05** (2006) 026, doi:10.1088/1126-6708/2006/05/026, arXiv:hep-ph/0603175.
* [42] CMS Collaboration, "Event generator tunes obtained from underlying event and multiparton scattering measurements", _Eur. Phys. J. C_**76** (2016) 155, doi:10.1140/epjc/s10052-016-3988-z, arXiv:1512.00159.
* [43] GEANT4 Collaboration, "GEANT4--a simulation toolkit", _Nucl. Instrum. Meth. A_**506** (2003) 250, doi:10.1016/S0168-9002(03)01368-8.
* [44] CMS Collaboration, "Measurement of differential cross sections with proton-proton collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**96** (2016) 012004, doi:10.1103/PhysRevD.96.012004, arXiv:1502.02701.
* [45] ATLAS Collaboration, "Measurements of inclusive W boson production cross sections in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Lett. B_**716** (2016) 1, doi:10.1016/j.physletb.2016.03.021, arXiv:1502.02701.

[MISSING_PAGE_POST]