MELA: Multilingual Evaluation of Linguistic Acceptability (2024)

Abstract: In this work, we present the largest benchmark to date on linguistic acceptability: Multilingual Evaluation of Linguistic Acceptability— MELA, with 46K samples covering 10 languages from a diverse set of language families. We establish LLM baselines on this benchmark, and investigate cross-lingual transfer in acceptability judgements with XLM-R. In pursuit of multilingual interpretability, we conduct probing experiments with fine-tuned XLM-R to explore the process of syntax capability acquisition. Our results show that GPT-4o exhibits a strong multilingual ability, outperforming fine-tuned XLM-R, while open-source multilingual models lag behind by a noticeable gap. Cross-lingual transfer experiments show that transfer in acceptability judgment is non-trivial: 500 Icelandic fine-tuning examples lead to 23 MCC performance in a completely unrelated language—Chinese. Results of our probing experiments indicate that training on MELA improves the performance of XLM-R on syntaxrelated tasks. https://github.com/sjtu-compling/

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2658–2674 August 11-16, 2024 ©2024 Association for Computational Linguistics
https://github.com/sjtu-compling/ MELA

heading: 1 Introduction
text: The acceptability judgment task tests a language model’s ability to distinguish syntactically acceptable sentences like (1a) from unacceptable ones like (1b) in a human language - for instance, the following example on island constraints in English (Ross, 1967).
(1) a. Whose book did you find? b. *Whose did you find book?
As a core linguistic competence, the ability to tell well-formed sentences from ill-formed ones is one of the first that a good language model should have.
Many corpora and benchmarks have been built to evaluate language models’ syntactic ability, using
*First two authors contributed equally to this work. Corresponding authors: Rui Wang and Hai Hu.
either a data-driven approach, where examples created by theoretical linguists in published textbooks are collected, e.g., CoLA—Corpus of Linguistic Acceptability (Warstadt et al., 2019), or a theorydriven approach, where minimal pairs targeting specific syntactic phenomena are generated semiautomatically via some template (Warstadt et al., 2020; Xiang et al., 2021; Hu et al., 2020b).
Recently, there has been growing interest in expanding the data-driven paradigm into other languages. For instance, CoLA-style datasets have been proposed in Russian (Mikhailov et al., 2022), Italian (Trotta et al., 2021), and Chinese (Hu et al., 2023). However, to date there are almost no multilingual benchmarks in this area that can be used to systematically test such abilities of multilingual models.
On the other hand, recently introduced benchmarks for Large Language Models (LLMs) such as GPT-4 (OpenAI, 2023) have mostly focused on application-driven tasks such as world knowledge and commonsense reasoning (Hendrycks et al., 2021a; Srivastava et al., 2022), math reasoning (Cobbe et al., 2021; Hendrycks et al., 2021b), and code generation (Chen et al., 2021; Austin et al., 2021; Zhang et al., 2023). Few studies, however, have investigated these models from a more linguistics-oriented aspect.
To address these gaps, we introduce MELA— Multilingual Evaluation of Linguistic Acceptability, the first large-scale multilingual acceptability benchmark with 46k examples covering 10 languages from a diverse set of language families. Data in four languages are from existing benchmarks mentioned above, and we complement them with newly collected data in six languages. Examples of MELA are demonstrated in Table 1. Following the CoLA tradition, all sentences in MELA are hand-written by linguists in respective languages, taken from textbooks, handbooks and journal articles in theoretical syntax, except for a small frac-
2658
tion of Russian sentences from Mikhailov et al. (2022).
We propose three possible usage cases for MELA, and make preliminary explorations in this paper:
Benchmarking We benchmark various multilingual language models (LMs) on MELA, including BLOOMZ (Scao et al., 2022; Muennighoff et al., 2023), mTk (Wang et al., 2022), mT0 (Muennighoff et al., 2023), Baichuan2-Chat (Yang et al., 2023), GPT-3.5 and GPT-4o (OpenAI, 2023).
Cross-lingual transfer We train XLM-R (Conneau et al., 2020) on different language combinations, finding non-trivial cross-lingual transfer performance even between unrelated language pairs, despite the vast difference in the basic syntax of the 10 languages in MELA.
Syntax acquisition We probe the syntactic capacity of MELA-finetuned XLM-Rs on syntaxrelated probing tasks, which indicates that XLM-R acquires some syntactic knowledge from finetuning on the acceptability judgment task.
In the rest of this paper, we first review relevant literature in §2, and then describe how MELA was constructed in §3. Next, we use MELA to benchmark several open-source and close-source LLMs in §4. We investigate cross-lingual transfer and multilingual fine-tuning in §5. Finally, we probe the XLM-Rs trained on MELA for their syntaxrelated capacity in §6.

heading: 2 Related Work

heading: 2.1 Linguistic Acceptability
text: As we mentioned in §1, large-scale linguistic acceptability datasets are currently available for four languages: CoLA for English (Warstadt et al., 2019), ItaCoLA for Italian (Trotta et al., 2021), RuCoLA for Russian (Mikhailov et al., 2022), CoLAC for Chinese (Hu et al., 2023), NoCoLA for Norwegian (Jentoft and Samuel, 2023), and JCoLA (Someya et al., 2024) for Japenese. Sentences from these datasets are taken from academic works by theoretical syntacticians and are therefore annotated by expert linguists.1
Another line of work in linguistic acceptability is based on minimal pairs, consisting of two nearidentical sentences with minimal differences. Language models are expected to assign a higher probability to the acceptable sentence than the unacceptable one. The minimal pair paradigm is adopted to evaluate specific syntactic issues such as subjectverb agreement (Linzen et al., 2016; Marvin and Linzen, 2018; Varda and Marelli, 2023), reflexive anaphora (Futrell et al., 2019; Hu et al., 2020a), negative polarity licensing (Wilcox et al., 2019; Jumelet and Hupkes, 2018), long-distance dependency (Wilcox et al., 2018; Chowdhury and Zamparelli, 2018), and argument structure (Kann et al., 2019; Tjuatja et al., 2023). Following these works,
1CoLAC also comes with an additional set of crowd labels; Unacceptable sentences in NoCoLA are sourced from grammatical mistakes made by Norwegian learners.
comprehensive benchmarks of minimal pairs are constructed in English resources (Warstadt et al., 2020; Hu et al., 2020b), and then expanded into other languages (Xiang et al., 2021; Song et al., 2022; Someya and Oseki, 2023; Nielsen, 2023).
In this work, we follow CoLA in constructing our benchmark as an initial step towards multilingual evaluation in acceptability judgment, as our goal is to have a wide coverage of syntactic phenomena in the languages selected.

heading: 2.2 Multilingual Evaluation Benchmarks
text: XTREME (Hu et al., 2020c) and XGLUE (Liang et al., 2020) are two of the most popular multilingual evaluation benchmarks. Of the tasks therein, many are constructed by translating English samples entirely or partially into other languages, such as XNLI (Conneau et al., 2018), PAWS-X (Yang et al., 2019), and MLQA (Lewis et al., 2020).
Apart from these NLU benchmarks, the literature has also witnessed an abundance of multilingual generation benchmarks, ranging from summarization (Scialom et al., 2020; Ladhak et al., 2020) to translation (Fan et al., 2021; Goyal et al., 2022). After the popularization of multitask instruction finetuning in language models (Wei et al., 2022; Sanh et al., 2022), multilingual instruction datasets have also been proposed, represented by Supernatural Instruction (Wang et al., 2022) and xP3 (Muennighoff et al., 2023). We refer to Qin et al. (2024) for a more comprehensive review of recent multilingual resources.

heading: 3 MELA: Multilingual Evaluation of Linguistic Acceptability
text: MELA consists of more than 46 thousand acceptability samples across 10 languages from a diversity of language families and groups. Specifically, it contains three Germanic languages: English, German and Icelandic, three Romance languages: Spanish, French and Italian, one Slavic language: Russian, one Sino-Tibetan language: Chinese, one Japonic language: Japanese, and one Semitic language: Arabic. Table 1 shows example sentences and properties of each language in MELA. For dataset statistics, see Table 2.

heading: 3.1 Data collection Procedure
text: High-resource languages. We use four existing datasets for four languages in MELA: CoLA (Warstadt et al., 2019) for English, Ita-
CoLA (Trotta et al., 2021) for Italian, RuCoLA (Mikhailov et al., 2022) for Russian, and CoLAC for Chinese (Hu et al., 2023), each having more than 6,000 data points.2 Since the out-ofdomain samples of RuCoLA are produced by generative models, we additionally collected 1037 Russian samples from The Syntax of Russian (Bailyn, 2011) (with the procedure described below) and add them 50-50 to the development and test sets of the Russian portion to keep a balance between validation-test discrepancy and generalization.
Low-resource languages. Apart from the four existing acceptability datasts, we also collected samples in 6 new languages, all annotated by theoretical syntacticians in their respective languages. These sentences are taken from five books/textbooks in the Cambridge Syntax Guides series, namely The Syntax of German (Haider, 2010), The Syntax of French (Rowlett, 2007), The Syntax of Spanish (Zagona, 2001), The Syntax of Arabic (Aoun et al., 2009) and The Syntax of Icelandic (Thráinsson, 2007). Japanese data were collected from Handbook of Japanese Syntax (Shibatani et al., 2017).
Each book contains roughly one to three thousand example sentences with acceptability judgments made by linguists in respective languages. Graduate students majoring in linguistics in these languages were paid to extract all example sentences with their judgments in these books manually. Note that, following previous CoLA-style corpora, we only keep sentences labeled with * or ?? as our unacceptable sentences. All unmarked sentences are extracted as acceptable sentences.
Following previous acceptability datasets, we remove examples when the judgment is based on coindexing of pronouns, empty categories, prosody or semantic/pragmatic interpretation. We also complete the sentence if it is composed of only a phrase, while keeping the judgment.
For Japanese, we remove examples from its dialects (N=99) and those about classical Japanese (N=13). For Arabic and Russian, as the original sentences are written in transliterations, we also convert them to their respective scripts manually.
The mean time for data collection for one language is about a month, with Icelandic taking about 3 months as there were more examples in the book.
As these books/textbooks and handbooks are 2NoCoLA and JCoLA are not included for they are con-
current with this work.
overviews of the syntax of each language, we believe they cover a wide range of linguistic phenomena in these languages, and can therefore serve as a good resource to evaluate language models’ overall ability to distinguish acceptable sentences from unacceptable ones.

heading: 3.2 Resulting Corpus and Data Split
text: The resulting corpus contains more than 46k example sentences in 10 languages.
For Italian and Chinese, we use the original train/dev/test splits of ItaCoLA and CoLAC, and for CoLAC we use the crowd label following Hu et al. (2023) For English and Russian, we keep the training splits of CoLA v.1.1 and RuCoLA, and use their in-domain development sets as our validation sets, and their out-of-domain development sets as our test sets.
For the six low-resource languages, we decide to adopt two splits for two purposes: fine-tuning smaller models such as XLM-R (v1.0) and benchmarking LLMs (v1.1).3 For v1.0, with the purpose for fine-tuning, we randomly sample 500 sentences from each of these languages to construct a training set, and divide the remaining sentences roughly equally between validation and test sets. For v1.1, we reserve 100 samples from each language as the validation set, and keep all the rest of the examples in the test set, thus producing a larger test set which we believe will make the evaluation more stable. See Table 2 for details of the two splits.

heading: 3.3 Evaluation Metric
text: Following previous works in linguistic acceptability, we evaluate the performance on MELA
3Performance of LLMs on two splits of these data are similar (see Table 8).
by Matthews Correlation Coefficient (MCC, Matthews, 1975), which is a measure of similarity between binary distributions taking values from -1 to 1 and always yielding 0 for any two uncorrelated distributions, regardless of class imbalance.

heading: 3.4 Comparison with Other Multilingual Benchmarks
text: We note that all samples in MELA are constructed individually in each language. While some early multilingual benchmarks opt to translate English sentences into other languages to obtain parallel samples (Conneau et al., 2018; Lewis et al., 2020), this approach does not suit our case. First, the task of linguistic acceptability is highly languagedependent, and syntactic structures acceptable in one language may not be acceptable in another, and thus there is no easy way of translating existing corpora into other languages while keeping the target syntactic phenomena. Second, as Clark et al. (2020) and Hu and Kübler (2021) argue, translation introduces artifacts into multilingual benchmarks and often results in translationese.

heading: 4 Evaluating LLMs with MELA
text: In this section, we report the performance of finetuned XLM-R and several LLMs, open-source or close-source, on MELA.

heading: 4.1 Experimental Settings
text: To establish a supervised baseline, we use XLMRoBERTa (Conneau et al., 2020), which is a multilingual version of RoBERTa (Liu et al., 2019) pre-trained on 2.5TB CommonCrawl corpus covering one hundred languages. XLM-R is fine-tuned on the combined training sets of all languages in
MELA v1.0 with the hyper-parameters described in Appendix A.2.
For open-source LLMs, we consider BLOOMZ (Scao et al., 2022; Muennighoff et al., 2023), two instruction finetuned variants of mT5 (Xue et al., 2021)—namely mTk (Wang et al., 2022) and mT0 (Muennighoff et al., 2023)—and Baichuan2-Chat (Yang et al., 2023) along with its base model. BLOOMZ is both pre-trained and fine-tuned on 46 languages, which only covers 5 languages in MELA: English, Chinese, French, Spanish, and Arabic4. The pre-training corpus of mT5 includes all 10 languages in MELA, but mT0 is fine-tuned on the same instruction dataset as BLOOMZ. mTk’s fine-tuning data, on the other hand, covers nine languages in MELA (except for Icelandic) and includes the English CoLA dataset. For Baichuan2, the exact language distribution of
4Muennighoff et al. (2023) examine BLOOM’s pretraining corpus ROOTS and estimate it to also contain a small amount of Russian, German, Italian, and Japanese.
pre-training and fine-tuning data is not disclosed. For close-source models, we consider GPT-3.5 and GPT-4o (OpenAI, 2023).
There are several decisions to make when evaluating the above LLMs on MELA: prompt selection and the number of examples in the few-shot scenario. After some pilot experiments, which we describe in Appendix A, we opt to use a binarychoice method with the best performing prompt on the development set, and report the results on the test set in both zero-shot and two-shot scenarios.

heading: 4.2 Main results
text: Results of fine-tuned XLM-R and LLMs evaluated on MELA are given in Table 3. We make the following observations.
Observation 1: GPT-4o exhibits a strong multilingual ability for acceptability judgement. It achieves the best performance on each individual language in MELA, exceeding supervised fine-tuned XLM-R. Its performance is 11 points
higher than finetuned XLM-R even in a 0-shot setting. There is a bigger gap between GPT-4o and finetuned XLM-R on low-resource languages than high-resource ones, likely due to the small amount of training data (500 examples) for XLMR. Compared to GPT-4o, GPT-3.5 seems to be more English-centric, with drastic performance drop in non-English languages.
Observation 2: LLMs benefit more from inlanguage examples in two-shot setting. Our results suggest that prompting with two English examples (most of the time) leads to a lower performance than prompting with in-language examples. On Icelandic, for example, the MCC of the 2-shot setting with English in-context examples is even lower than the 0-shot performance for Baichuan2, GPT-3.5, and GPT-4o.
Observation 3: Baichuan2-Base requires incontext examples As shown in Table 3, under zero-shot setting, Baichuan2-Base shows random performance5, while its Chat model exhibits nontrivial performance, even for languages such as Spanish. The Base model benefits more from incontext learning examples though, surpassing the Chat model in two-shot settings.
5Baichuan2-Base always chooses “B. Unacceptable” as the answer, under all prompts we tested.

heading: 5 Cross-lingual Transfer and Multilingual Fine-tuning
text: In this section, we investigate cross-lingual transfer and multilingual fine-tuning of linguistic acceptability with XLM-R. All training and evaluation are done on MELA v1.0 as it requires a training set.

heading: 5.1 Experimental Settings
text: Cross-lingual Transfer To observe the transfer of acceptability judgements across languages, we fine-tune XLM-R on one language, and evaluate on all 10 development sets. We report the median MCC of seven runs for all results to mitigate interrun variance.6
Multilingual Fine-tuning We downsample sentences in each language to the same number, and fine-tune XLM-R in three settings: (1) in-language, where the fine-tuning and evaluation languages are the same; (2) all-language, where the model is finetuned on a mixture of data containing an equal number of sentences from ten languages; and (3) all-but-in-language, where the model is fine-tuned on a mixture of data containing an equal number of sentences from nine languages, except the one being evaluated on.
6Training details can be found in Appendix A.2.

heading: 5.2 Results
text: Results for cross-lingual transfer and multilingual finetuning of XLM-R are reported in Table 4 and Figure 1. We make the following observations.
Observation 1: Cross-lingual transfer is nontrivial. In Table 4, we see that all numbers are (much) greater than 0, suggesting that transferring from language A to language B is possible, even for acceptability judgment tasks. For instance, fine-tuning XLM-R on 500 Icelandic examples results in 23.16 MCC for a completely unrelated language, Mandarin Chinese. A similar conclusion can be drawn from Figure 1, where the green line, which has no in-language training data, demonstrates an increasing trend for all languages, except for French, which plateaus at around 10 MCC.
Observation 2: Size of training set matters, but not always. The overall performance when highresource languages are used as training data (>6k training examples), 29.69 MCC, is higher than when low-resource ones are used (=500 examples), 18.18 MCC, as shown in the last block of Table 4. However, it must be pointed out that sometimes a (14 times) larger training set does not lead to better performance. For instance, in the second column of Table 4, when evaluated on Chinese, 500
examples of German or Spanish achieve roughly 37 MCC, which is on par with having more than 7,000 training examples for Italian and Russian, with 34.86 and 39.77 MCC respectively. Similarly, when evaluated on Icelandic, 500 German examples again demonstrate a performance on par with many thousands of Chinese, Italian or Russian examples (second to last column of Table 4). Thus the transferring performance between two languages seems to be a result of both the language pair in question as well as the size of the training set.
Observation 3: Among low-resource languages, Arabic training data has the lowest average performance (Table 4). This is likely due to the fact that Arabic is from a different language family from all other nine languages. From the last column of Table 4, we observe that German and Spanish training data have the best performance, likely because MELA has three Germanic languages and three Romance languages, which may make cross-lingual transfer easier among these cognate languages.
Observation 4: For Italian, Japanese and Arabic, during multi-task training, adding inlanguage data does not affect performance much. From Figure 1, we see that the green and orange lines cross for these three languages, suggesting that when training with mixed-language
data, in-language examples may not be very critical for these languages.

heading: 6 Edge Probing
text: In this section, we adopt edge probing (Tenney et al., 2019a,b) to explore whether fine-tuning on acceptability judgment tasks injects syntax-related information into the pre-trained XLM-R.

heading: 6.1 Experimental settings
text: Edge probing focuses on structural labeling tasks in the form of span labeling. We choose following tasks: 1) part-of-speech tagging, 2) dependency labeling, 3) constituency labeling, 4) named entity labeling, 5) semantic role labeling, and 6) coreference.7 Take dependency labeling as an example, representations of a dependent and its head, encoded by an XLM-RoBERTa, are used to train a probe classifier to predict the dependency relation between the two words.
We hypothesize that training on MELA can improve the performance of XLM-R on the syntaxrelated probing tasks above, and design the following experiments.
Experiment 1 We train probing classifiers using span representations from XLM-Rs on English probing tasks. We set the pre-trained Mbase (pretrained XLM-R) as the control group, and the other four MELA-fine-tuned Mlangmela as the test group, where lang specifies the training data of which language from MELA were used for fine-tuning.
Experiment 2 For two tasks (pos and dep) with multilingual data available, we experiment on crosslingual transfer as well. We train probing classifiers
7Tasks 1-2 are from UD (De Marneffe et al., 2021); Tasks 3-6 are from OntoNotes (Weischedel et al., 2013).
on representations from Mbase in each of four highresource languages and run zero-shot evaluation on a target language (lang). We repeat the procedure on Mlangmela (see more details in Appendix B).

heading: 6.2 Results
text: In Experiment 1 we train probing classifiers using representations from different XLM-R variants, some of which have been fine-tuned on MELA while the base model has not. Results in Table 5 show that the average performance of XLM-Rbase on the six probing tasks is the lowest across the six edge probing tasks (see the last row). We further observe from Table 5 that the semantic role labeling task benefits most from MELA-fine-tuning.
In Experiment 2, the performances of probing tasks are evaluated in the cross-lingual transfer setting. We compare the the pre-trained XLM-R model (Mbase) and XLM-Rs fine-tuned on the linguistic acceptability judgement task of a specific language (Mlangmela) (see Table 6). The results indicate that the probing classifiers trained on span representations from fine-tuned XLM-R models achieve better performance than the base model. Fine-tuning on one language of MELA helps the model transfer to that language, and more often than not other languages in part-of-speech tagging and dependency labeling.
MELA-fine-tuned XLM-Rs perform better on syntax-related probing tasks in mono-lingual and cross-lingual settings, supporting our hypothesis.

heading: 7 Conclusion
text: In this work we present MELA, the first multilingual acceptability judgement benchmark covering a diverse set of 10 languages, all annotated by expert linguists. By benchmarking multilingual LLMs on MELA and fine-tuning XLM-R in different cross-lingual settings, we find that (1) GPT4o ourperforms supervised XLM-R, especially on low-resource languages, that (2) in-language data is crucial for few-shot evaluation and that (3) crosslingual transfer is non-trivial for all language pairs in supervised fine-tuning. We probe MELA-finetuned XLM-R for the syntax information encoded, finding that training on MELA improves the performance on syntax-related probing tasks, which indicates that language models acquire syntactic knowledge during training on linguistic acceptability judgements.
mela. We conduct a pair comparison between Mbase
and Mlangmela trained on MELA of one language to investigate whether linguistic acceptability helps the cross-lingual transfer in the above two probing tasks. For each cell, probing classifiers are trained on span representations in the language denoted in the second row, which are encoded by the model denoted in the second column, and evaluated on the probing tasks in the same language on which XLM-R is fine-tuned (the first column).
Limitations
Due to the large amount of human labor involved in transcribing and examining the sentences in MELA, the dataset only covers ten languages, of which six are low-resource, with only a small number of training samples. In the future, we intend to expand the dataset by additionally collecting data in other languages, especially non-Latin and nonIndo-European languages, which are currently underrepresented in MELA.
Also, in this work we focused on introducing the MELA dataset and showcasing some of its usages, such as benchmarking LLMs and providing a data resource for cross-lingual research in computational linguistics. We leave the exploration of other use cases of MELA to future work.
Ethics Statement
Sentences in our dataset MELA, including those in English, Italian, Russian, and Chinese consolidated from previous works, are sourced from renounced linguistics publications such as syntax textbooks and journal articles. Therefore, we believe they do not raise any ethical issues such as leak of personal identifiable information.
The sentences in MELA, both acceptable and unacceptable, are only intended for research concerning the acquisition and evaluation of linguistic capabilities (of either humans or language models), and should not be interpreted otherwise. For individual sentences in MELA, the copyright (where
applicable) remains with the original authors or publishers. We ask researchers who use MELA to also cite the original source, i.e., CoLA (Warstadt et al., 2019), ItaCoLA (Trotta et al., 2021), RuCoLA (Mikhailov et al., 2022) and CoLAC (Hu et al., 2023).

heading: Acknowledgments
text: We thank Bingjie Mao, Jinchi Jiang, Wen Shi, Jialin Guo, Chenhui Liu and Licen Liu for their help in data collection. We also appreciate the suggestions and comments from the anonymous reviewers. This study is supported by Shanghai Pujiang Program awarded to Hai Hu (22PJC063). Ziyin Zhang and Rui Wang are partially supported by the National Natural Science Foundation of China (62176153) and the Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102, as the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University).

heading: A Benchmark Details
text: In this section, we provide details about how we make decisions to benchmark LLMs, which includes prompt selection, and the number of examples in the few-shot scenario, along with details of fine-tuning XLM-R.
A.1 Large Language Models
Following MMLU (Hendrycks et al., 2021a), we evaluate MELA in the multiple-choice format. We use prompts that end with “Answer:”. Models are required to produce probabilities for index tokens “A” and “B”. The index token with a higher probability is regarded as the decision of models. We tune prompts with open-sourced LLMs on the validation set in our pilot experiment.
We first experiment on 1,000 samples of MELA (50 samples per label per language), using 4 different prompts in 0/2/4/6/8-shot (equal number of positive and negative examples) scenarios with inlanguage examples provided. The prompt with the highest average MCC is selected to evaluate LLMs on the whole MELA test set (see Figure 2 and 3). The results indicate that 1) prompt-8 is better than others and 2) models no longer improve with more than 2 in-context examples.
Therefore, we carry out formal experiments with prompt-8 (see Figure 4) for both open-sourced LLMs and GPT models in zero and two-shot scenarios (see §4).
Note that mTk includes the 2-shot CoLA task in its training data. We also reuse the prompt for Supernatural Instruction task 6168. We compare the results of mTk’s origin CoLA prompt (see Figure 4) and our multiple-choice prompt (see Table 7).
8https://github.com/allenai/ natural-instructions/blob/master/tasks/ task616_cola_classification.json
A.2 XLM-R Fine-tuning Details
For experiments concerning XLM-R in §4 and §5, we finetune with learning rate 7.5e-6, weight decay 0.075 and batch size 32. To minimize confounding variables and accentuate the interaction across languages in terms of linguistic acceptability performance, we train the model for 5k steps for all experiments in §4 and §5 with 750 steps of linear warmup and cosine learning rate decay over 0.4 cycles, and take the best checkpoint based on validation results.
We note that these hyperparameters are chosen based on previous works on similar tasks (Liu et al., 2019; Hu et al., 2023) and our preliminary experiments. The sheer amount of experiments covered in our work makes it impossible to finetune hyperparameters on each combination of training data, and we thus decide to keep them fixed across all experiments for a fair comparison across languages, which may be suboptimal for certain cases. Hu et al. (2023), for example, report 56.45 MCC for XLM-R on CoLAC development set, while our result is 52.71 with the same training data.
We also note that finetuning language models on linguistic acceptability data leads to large performance variations, regardless of the specific languages (see Figure 5), which corresponds with previous findings in the literature (Raffel et al., 2020). We thus train with seven different random seeds for every experiment in this work to reduce this variance, and the reported scores are computed by first taking the median of these seven runs at each checkpointing step, and then maxing over all the aggregated checkpoints. For experiments on downsampled data in §5, each run also selects a different subset of training data.
A.3 Comparison Between Two Splits
In §3, we have to split the MELA datasets into train, development, and test sets to fine-tune XLM-
R. However, considering training data is no longer necessary for LLM evaluation. Therefore, we decide to make two different data splits. On the one hand, we want a fair comparison between supervised fine-tuning XLM-R and zero/few-shot LLMs. On the other hand, we want MELA to better fit the recent paradigm of LLM evaluation. In this case, we provide a comparison between the performance of LLMs on the two different versions of data (see Table 8). The results of the two versions are similar, by which we make it comparable between fine-tuned XLM-R on v1.0-test set and LLMs on v1.1-test set.

heading: B Edge Probing Details
text: Probing Classifier We follow the same architecture of probing classifier as (Tenney et al., 2019b). We extract contextual representations from each layer of XLM-R (including the embedding layer), and get the scalar mixed representations (in 1,024- dim), see Equation (1) in (Tenney et al., 2019a). Then, the representations are projected in 512-dim with a CNN module. For two-span prediction, we
concatenate representations of two spans into a 1,024-dim tensor. We pass the span representations to the probing classifier, which is a two-layer MLP (hidden state dimension is set to 512).
Probing Dataset For part-of-speech tagging and dependency labeling, we use PUD (parallel sentences in all four languages) in UD V2.13. For the other four tasks in OntoNotes 5.0, we downsample sentences to 2k. All datasets are split into train, development and test sets in a ratio of 7:1.5:1.5. For each sentence, there might be multiple labels, so we present the numbers of sentences, words and labels in Table 9.
Training We train classifiers for all probing tasks with an Adam optimizer at a starting learning rate of 5e-4 for 3,000 training steps with a batch size of 32, and evaluate on the development set every 50 training steps, halving the learning rate if no improvement is seen in 5 evaluation during training.

heading: Future Work
text: In the future, we intend to expand the dataset by additionally collecting data in other languages, especially non-Latin and nonIndo-European languages, which are currently underrepresented in MELA. Also, in this work we focused on introducing the MELA dataset and showcasing some of its usages, such as benchmarking LLMs and providing a data resource for cross-lingual research in computational linguistics. We leave the exploration of other use cases of MELA to future work