Don’t Trust ChatGPT when your Question is not in English: A Study of Multilingual Abilities and Types of LLMs
Authors: Xiang Zhang
Senyu Li
Bradley Hauer
Ning Shi
Grzegorz Kondrak
Venue: EMNLP 2023 Main (2023)
Keywords: multilingual
LLM
GPT
TL;DR: GPT is not trustworthy in some multilingual tasks that can not be directly translated into English

Abstract: Large language models (LLMs) have demonstrated exceptional natural language understanding abilities, and have excelled in a variety of natural language processing (NLP) tasks. Despite the fact that most LLMs are trained predominantly on English, multiple studies have demonstrated their capabilities in a variety of languages. However, fundamental questions persist regarding how LLMs acquire their multilingual abilities and how performance varies across different languages. These inquiries are crucial for the study of LLMs since users and researchers often come from diverse language backgrounds, potentially influencing how they use LLMs and interpret their output. In this work, we propose a systematic way of qualitatively and quantitatively evaluating the multilingual capabilities of LLMs. We investigate the phenomenon of cross-language generalization in LLMs, wherein limited multilingual training data leads to advanced multilingual capabilities. To accomplish this, we employ a novel prompt back-translation method. The results demonstrate that LLMs, such as GPT, can effectively transfer learned knowledge across different languages, yielding relatively consistent results in translation-equivariant tasks, in which the correct output does not depend on the language of the input. However, LLMs struggle to provide accurate results in translation-variant tasks, which lack this property, requiring careful user judgment to evaluate the answers.

Don't Trust ChatGPT when your Question is not in English:
A Study of Multilingual Abilities and Types of LLMs
Xiang Zhang" Senyu Li
Bradley Hauer Ning Shi Grzegorz Kondrak
Alberta Machine Intelligence Institute
Department of Computing Science
University of Alberta, Edmonton, Canada
{xzhang23, senyu, bmhauer, ning. shi, jeuandraayuallberta..
senyu,
Abstract
Type
(a) Compound
(b) Coordinate
(c) Subordinate
Large language models (LLMs) have demon- strated exceptional natural language under- standing abilities, and have excelled in vari- ety of natural language processing (NLP) tasks. Despite the fact that most L.LMs trained pre- dominantly on English, multiple studies have demonstrated their capabilitics in a variety of languages. However, fundamental questions persist regarding how LLMs acquire their mul- tilingual abilities and how performance varies across different languages. These inquiries crucial for the study of LLMs since users and researchers often come from diverse language backgrounds, potentially influencing how they use LLMs and interpret their output. this work, propose systematic way of qualita- tively and quantitatively evaluating the multilin- gual capabilities LLMSs. We investigate phenomenon of cross- tross-language generalization in LLMs, wherein limited multilingual training data leads advanced multilingual capabilities. To accomplish this, we employ novel prompt fack-translation method. The results demon- strate that L.LMs, such GPT, effectively transfer learned knowledge across different lan- guages, yielding relatively consistent results in canslation-equivarian tasks, in which the cor- rect output does not depend on the language of the input. However. LLMs struggle pro- vide accurate results in conssaationvvrian tasks, which lack this property, requiring careful user judgment evaluate the answers.
Internal Concept
Lexicon
knife
couteau
Knnffe
couteau
knife
couteau
Figure
The three types of bilingualism.
(LLMs), researchers have discovered many emer- gent properties (Wei al., 2022a) in these models, and have used them for a variety of purposes (Wei al., 2022b). However, the multilingual ability of these models has not been extensively studied.
Previous research has shown that large language models, such as GPT, are capable of performing wide variety of language tasks when the task is presented in English (Qin et al., 2023). However, investigations into the multilingual language abil- ities of these models have been limited. Shi et al. (2023) explore this topic by applying the models to multilingual datasets, and measuring performance differences across languages. However, they do not explore the underlying mechanisms of how LLMs perform different tasks, nor how this affects the results. Moreover, most LLMs (Brown et al., 2020; Touvron et al., 2023) trained datasets that are heavily skewed towards English, which leaves open the question of how multilingual abilities in such models are acquired.
1 Introduction
The study bilingualism has long been a topic of interest among linguists (Yu et al., 2022; Hoffmann, 2014), as it provides insight into the mechanisms of language acquisition and processing. Furthermore, research multilingualism has contributed to the development of more effective machine learning models, such neural translation systems (Zou al., 2013). With the rise of large language models
In this study, we present a systematic approach to analyzing the multilingual capabilities of LLMs. To facilitate comprehensive analysis, we pro- pose categorizing Linguage-depentee abilitics into three distinct categories which vary in the impact of language choice on the performance: Reasoning (least impact), Knowledge Access, and Articulation (most impact). We investigate a carefully selected set of tasks from these three categories by evalu-
"Equal contribution.

---

ating the multilingual abilities of LLM using a novel prompting method which we call response back-translation (RBT). By comparing the gener- ated answers, we can both measure multilingual performance of the LLM, but also determine the type of multilinguality they exhibit. For example, we examine the capabilities of LLMs on pun detec- tion, highly danguage-depentee
learn. This separation leads to differences when tasks are performed under different language set- tings (Jakobovits, 1968).
Finally, subordinate bilingualism is character- ized by "translator" behaviour (Marcos, 1976). This type of bilingualism is characterized by sin- gle lexicon representation that is linked to their dominant language (Lörscher, 2012). When per- forming tasks languages other than their dom- inant one, subordinate bilinguals tend to rely on translating the task into their dominant language, formulating an answer in the dominant language, and then translating that answer back into the lan- guage of the task. As a result, subordinate bilin- guals may experience lower proficiency in com- municating and completing tasks in the second, subordinate language.
task.
The results of our experiments show that the pop- ular LLM "GPT": (1) achieves higher performance when the task is presented English; (2) achieves higher performance on tasks that can be translated without altering the correct output; and (3) exhibits a mixture of coordinate and subordinate bilingual- ism.
Our main contributions! are:
We present a first-of-its-kind quantitative and qualitative analysis of the multilingual abili- ties of LLMs.
Despite the demonstration in prior work of con- sistent multilingual performance in many large lan- guage models (Shi et al., 2023), it remains unclear how the multilingualism of LLMs should be cate- gorized. It an open question whether the LLMs exhibit a representation of knowledge shared across both languages (compound), separate representa- tions for each language (coordinate), or whether they rely on a translation processes (subordinate). We develop an experimental framework aimed at using performance on various natural language pro- cessing tasks to determine how the multilingual abilities of LLMs relate to these categories.
We propose novel task categorizations to facilitate the multilingual ability unalysis.
Our work is the first to investigate LLMs with respect to linguistic typology of bilingualism and multilingualism.
2 Background
Linguists categorize bilingual individuals into three groups: compound, coordinate, and subordinate bilinguals (D' Acierno, 1990). Figure illus- trates this categorization, showing how individuals with different types of English-French bilingualism might internally represent the concept of "knife". Compound bilingualism mostly emerges among individuals who learn two languages simultane- ously from birth. In this case, both languages equally dominant and integrated, blurring any clear distinction between them and giving the impres- sion of a single unified language (Moradi, 2014). Compound bilingualism entails a shared mental representation of lexicons across both languages they acquire, and compound bilinguals are the most flexible in their use of multiple languages, exhibit- ing the ability to switch between languages without losing consistency in linguistic tasks (De Groot and Nas, 1991).
3 Categorizing Language-ependent Tasks
Language ability is multifaceted concept en- compassing various tasks and aspects (Wei et al., 2022a). It is therefore difficult to assess a model's capabilities with respect to a given language. To facilitate such assessment, researchers have often classified tasks into distinct categories (Khurana et al., 2023), such parsing and summarization. However, the delineation of such categories often lacks systematic criteria, particularly in the context of multilingual analysis.
In this section, we propose novel approach to categorizing NLP tasks, which is better suited to analysis of multilingual abilitics. The catcgoriza- tion is two dimensional: one dimension is based on the linguistic knowledge necessary to complete the task (Section 3), the other on how the task is impacted by the language in which it is presented (Section 3.2).
In contrast, individuals exhibiting coordinate bilingualism maintain separate mental represen- tations for the lexicon of each language they
Our data is publicly available at GitHub.

---

Reasoning
Knowledge Access
Articulation
What the of 83?
Explain what 'finite machine".
Can you write for SDE position?
lettor
Prompt
am writing express my interest in the position
Response
996.
Prompt
Response
Y SDE 34G2
Zh Prompt
123
88 /
BER A HARKS
XMXXXXB
Figure 2: Three categories of NLP tasks.
3.1 Categorization by Task Properties
the pragmatics and cultural nuances of the spo- ken language. For instance, writing a cover letter in English significantly differs from writing one in Japanese, due to the distinct social norms and conventions associated with those languages. The Articulation category includes tasks that are heavily influenced by the language choice, such summa- rization (Nenkova and McKeown, 2012), dialogue generation (Ni al., 2023), paraphrasing (Zhou and Bhat, 2021), and style writing (Jin et al., 2022). These tasks require an extensive understanding of not only language, but the associated culture, as they involve capturing and reproducing the appro- priate style, tone, and manner of expression specific to given language.
We classify NL.P tasks into three distinct categories: Reasoning, Knowledge Access, and Articulation. This division is based on the extent to which per- formance on each task is influenced by the model's capabilities with the language used. Figure pro- vides an overview of this categorization.
Reasoning The first category includes tasks that are minimally influenced by language, which consistent performance is expected across lan- guages. Reasoning tasks involve logical and ra- tional thinking to solve problems based on avail- able information and logical principles. Examples include mathematical problem-solving (Lu et al., 2023), coding (Li et al., 2023), and common sense reasoning (Sap et al., 2020). These tasks can be performed using universal language elements, such mathematical symbols, or rely on general life experience and common sense, which can be ac- quired without language. For example, answering the question "If I drop an apple, which direction will it go?" relies more on understanding gravity than on language -anguage-specffic knowledge.
3.2 Categorization by Translatability
The second dimension of our task classification scheme involves translatability. We introduce the concepts of Translation Equivariant (TE) and Trans- lation Variant (TV) tasks.
A function considered equivariant if it com- mutes with a symmetry transformation. That is, applying a transformation before or after comput- ing the function yields the same result. Formally, f(-) is said to be equivariant under g(-) if:
Knowledge Access LLMs have the capability to function as knowledge bases (KBs) by storing knowledge extracted from training data (Heinzer- ling and Inui, 2021). Knowledge Access tasks de- pend on the ability to access this knowledge and formulate accurate responses based on it. While the underlying knowledge may not be language de- pendent, models may be less reliable in retrieving and utilizing knowledge learned in a language other than the one used to formulate the task. Examples Knowledge Access tasks include factual knowl- edge checking (De Cao et al., 2021), knowledge- focused question answering (Wang, 2022), and named entity recognition (Malmasi et al., 2022).
Vo E D. g(f(r)) f(g(x))
(1)
where represents the domain of both f and g. We denote translation as transformation g that converts given text in language to an equiva- lent text in language B. In practice, 9 can be im- plemented by a machine translation system. We further use f to denote function which solves given task, given an instance of that task as in- put. A task is considered Translation Equivariant between languages and B if the correct output can be obtained by translating the input, and then applying a method for solving the task, or by solv- ing the task, and then translating the output; other words, if (x())) = f((())). Most of the
Articulation Much of everyday human conver- sation is highly language-dependent, as it involves

---

tasks in the Reasoning and Knowledge Access cate- gories are regarded as Translation Equivariant since the correct output does not depend on the chosen language. Figure shows an example where the answer to the question posed in English remains the same in Chinese, regardless of in which order the translation system and the question answering system are applied.
Translation Equivariant (TE)
9
What day after Monday?
Tuesday
Translation Variant (TV)
Una bicicleta no puede sostenerse par sola ya que tisne dos llantas.
No.
A task which is not Translation Equivariant is Translation Variant. For such tasks, translating the input may change the correct output. TV tasks rely heavily on the language used, and include many tasks in the Articulation category. Representative TV tasks that we investigate in our experiments are letter writing and pun understanding. The former is subject the conventions of the specific lan- guage and culture, while the latter involves word polysemy, which is often sensitive translation. Figure shows an example where a pun is present in the original English input, but not in the Span- ish translation, making the classification dependent upon the order in which translation is applied.
9
bicycle can't stand on its own since It's twa-tired.
Yes.
Figure 3: A TE task (common sense reasoning) and task (pun detection). Translation is denoted by g. and is the solver function.
successfully learns to solve a TE task in a language- independent way, the pairwise responses for each instance should be the same after the translation (re- gardless of whether it is correct or incorrect). This because TE tasks, such as mathematical problem solving, do not depend on the language used query the LLMs, as the solution does not depend on the language used to express the problem.
4 Methods
this section, we present our approach to analyz- ing the multilingual ability of LLMs. Our methods involve prompt translation (PT) and response back- translation (RBT). They are designed to measure performance of an LLM, and its consistency across apply these
4.2 Response Back- Translation
One of the goals of our work is to understand what the consistency of LLM output across languages tells about the model, and to determine the type of bilingualism LLM exhibits. This is crucial for individuals who use LLMs for multilingual tasks, as it can impact the way task results are gener- ated, and affect the quality and consistency of the results. For example, a network exhibiting sub- ordinate bilingualism would produce output that appears to be the result of translation, rather than resembling text generated by a native speaker of the output language.
languages. In our experiments, methods both TE and TV tasks, with the aim of determining the type of bilingualism (compound, coordinate, or subordinate) exhibited by an LLM.
4.1 Prompt Translation
Multilingual datasets are unvailable for many tasks. However, with state-of-the- art machine translation (MT) systems and LLMs, we can translate monolin- gual datasets for TE tasks to generate parallel mul- tilingual parallel data with minimal loss of infor- mation (Whitehouse al., 2023; Shi al., 2023). This the key intuition behind prompt transla- tion (PT); an example is shown Figure 4a, where an English multiple choice question, and its possi- ble answers, are translated to Chinese. The LLM is then prompted, and the response is given and eval- uated, in Chinese. Prompting in distinct languages is performed in independent LLM sessions.
To quantitatively measure how reasoning is per- formed, we propose prompting method based on back-translation, as illustrated in Figure 4b. Simi- lar to prompt translation (Section 4.1,, we begin by translating the instance to the target language, and prompting the LLM to produce response in that language. After obtaining output from the LLM, regardless of the language. LLM generate an explanation for its output (e.g.. "Explain how you obtain this result") and then trans- late the output of the LLM back to the original language. We then compare the explanation given
further prompt the
We measure the differences in multilingual task performance by comparing the answers given by the LLM in each language. Assuming that the LLM

---

(a) Prompt Translation (PT)
En Prompt One New Session
Zh Prompt in One New Session
Cr Wherm you find a chair at oifioe
For We
Optiion: H6l sleckum (8) sixten (Cyy porch (f) outtide (F) livng mom
Tranalation
[S4y (A) 14-945 (8) s0M (c)
[P]
Wi7s
Painwiss Comparison
(b) Response Back-Transiation (RBT)
En Prompt Same Session
2h Prompt Same Sassion
Can you regiain why you chows the answwr
Transiation
Because cie faund offios aeting onviron mont, poogle otton have desis and chairs wark and frew chadrs caunt'y des tyrwe conforaatte borlong poroos siting. Therelow,,
workspoce
RR Ri Miiiaaala Lo 51 TT liis Mawa RER
H8. 10B5L1 te 6ee 7 19186.
0: H89eourt sesc (Or Transate your orswer Biglan)
chiir an office a oupille where poople dit and work their computere ofher tedio.
erpleyess Therdfore, office chairs cytically locoaed theee cudeciss or ofl tae for erpplopess el and wore comfortably, Charce in theoss cultiien [x offifoes are stually painod with office dodes to provice contoraailc working enviroment for cmpployoos
I: Becouse offons ususity cubldies office portiione
Pairwisn Comparison
Figure 4: An overview of our prompt translation and response back-translation methods.
the source language to the explanation back- translated from the target language.
TE Task
If the L.M is performing translation-based rea- soning, the reasoning process is conducted in one language and then translated into another. Since the internal reasoning of the LLM can be partially observed through the output explanation, back- translating such explanations into the source lan- guage allows us to compare the internal reasoning used to solve the problem in each language. High similarity of explanations should indicate homo- geneity in using the same internal reasoning pro- cess to perform the task in both languages. On the other hand, dissimilarity in the reasoning process across languages should be reflected in lower explanation similarity.
Prompt Translation
Yes
Same results
No
across languages?
Compound
Response Trac-srrannttoon
TV Task
No
Same reasoning
Yes
across all languages?
Prompt Translation
Compound
Yes
Results change
No
after translation?
Coordinate
Subordinate
4.3 Identifying Multilingual Types
Figure 5: Flowchart for detecting multilingual types.
In our investigation, we employ both Prompt Trans- lation (PT) and Response Tackk-rraslation (RBT) to analyze how an LLM solves TE and TV tasks in different languages. As depicted in the first two steps in Figure 5, a compound LLM should exhibit consistent results on TE tasks with both methods. This is because compound model performance does not depend the language in which ques- tion is presented. Conversely, subordinate and coor- dinate types of networks are expected to yield some- what different results on TE tasks. A coordinate model accesses distinct representations in different languages, which may result in different reasoning and answers. Finally, a subordinate model heavily depends internal translation process, which we expect to lead to some deterioration of output quality across languages.
is expected to reason differently for each language, which may yield different outputs, whether correct or not. In contrast, a pure subordinate model is expected to reason only in the dominant language, producing relatively similar results in different lan- guages, regardless of whether the correct output is preserved after translation.
Experiments
We apply the methodology proposed in Scction 4 to TE and TV tasks. As our LLM, use Chat- GPT, via the official web application?, due its availability.
5.1 Datasets
Reasoning We use 50 instances selected at ran- dom from each of two datasets: GSM8K (Cobbe
Testing on TV tasks provide additional informa- tion, which can be used to distinguish between coor- dinate and subordinate models. A coordinate LLM
Phttps://chat.openai

---

al.. 2021), which contains 7,500 training and 1,000 test problems, and CommonsenseQA (Tal- mor et al., 2019), which contains 12,247 questions. We used ChatGPT to translate these instances into French, Spanish, German, Japanese, and Chinese. GSM8K is a dataset of grade-school math prob- lems. Each problem consists of a question and multiple-choice answer. CommonsenseQA is question answering dataset for testing logic and common sense. Each instance consists of a ques- tion and five answer choices, only one of which is considered correct.
Task
En
Fr
De
Es
Ja
Zh
MR
0.90
0.80
0.78
0.80
0.82
0.78
CSR
0.68
0.58
0.52
0.54
0,48
0.52
KA
0.96
0.96
0.94
0.94
0.80
0.68
Table 1: Accuracy for TE tasks: math reasoning (MR). commonsense reasoning (CSR), and knowledge access (KA).
name and background of an individual, including information such as level of education, specialties, and hobbies. We then randomly select one well- known company to which cover letter is to be ad- dressed. Finally, as "What skills would you want to develop in this role?". Each of these randomized prompts is then provided to the LLM. The output is then manually evaluated by a native speaker of the language of the prompt. We generate 50 prompts each in English and Chinese. An example is provided in Table 5 in the appendix.
Knowledge Access WebQuestions is dataset of 6,642 question-answer pairs extracted from Free- base (Bordes et al., 2014). An example question is "Where is the Thames River located?" which the correct answer is London. To simplify the evalua- tion, and avoid the issue of extracting answers from ChatGPT's often verbose responses, we manually converted 50 randomly selected instances into the multiple-choice format used by CommonsenseQA, To create plausible incorrect answers (distractors). we randomly selected four incorrect candidate an- swers from sets of world city names? and celebrity n (correct answers in this dataset are all ei- ther city names or celebrity names). This yielded set of 50 multiple choice questions with five pos- sible answers each (one correct, four incorrect). We translated the English instances into five other languages via ChatGPT.
select set of topics such
5.2 Metries
Since ChatGPT can give different answers the same question, we present each multiple-choice question to ChatGPT five times, and use the most frequent output for evaluation. For comput- ing similarity between explanations, we use ap- pendix(Devlin et al., 2019). Specifically, we trans- late all non non-Engonn English output to English via ChatGPT, and compute the cosine similarity of the BERT embeddings of the two explanations.
Puns We randomly selected 80 positive and 80 negative instances each from the English, French, and Spanish instances in the JOKER @CLEF 2022 dataset (Ermakova et al., 2022). Each instance is annotated with a yes/no classification as to whether it contains a pun, and the pun location, if pun is present. An example English instance is "As- tronauts work in nice atmosphere" for which the pun location is the word atmosphere. We used ChatGPT to translate the French and Spanish in- stances into English, and the English instances into French, Spanish, German, Japanese, and Chinese. This yields 10 balanced sets of 160 instances each (three original and seven translated).
5.3 Results on TE Tasks
As shown in Table 1, the results on TE tasks in En- glish are on average much higher in English than in other languages. In math reasoning (MR), the least Language-depentee task, the gap between English and other languages is over 10% on average. common sense reasoning (CSR), the difference is over 15% on average. In knowledge access (KA), there is no substantial difference between English and other European languages, but accuracy on Japanese and Chinese is 16% and 28% lower, re- spectively. To confirm that the accuracy gap is not due to instance translation quality, we manu- ally compared all 50 Chinese MR questions with their original English counterparts, and found no translation errors. Taken together, these results provide strong evidence that GPT better able to reason and retrieve knowledge given an English
Articulation To test the Articulation abilities of LLM, we prompt the model to generate cover letter for job application, with randomized spec- ifications. For each prompt, we first generate the
Pht //smppemaps..ppp...m//./ com/ com//dtaawoorl-ccitiss chttps://giithin ctm comm/jnneseermmd_.iii

---

prompt, compared to prompting in other languages. terms of multilinguality type, the evidence is against compound multilingualism in GPT (cf., Fig- ure 5), as a compound model would be expected to exhibit no substantial difference in performance across languages.
Chinese
English Translation
Frequency
Sincerely
54.0%
Regards
38.4%
BLU
Best Wishes
3.6%
Salute (Proper Chinese Sign-off)
0.8%
No sign-oft
3.2%
We also analyzed the BERT similarity values between explanations in different languages (cf., Table 4 in the Appendix). In commonsense reason- ing, which relies on logic and conceptual distinc- tions, we observe that the average BERT similar- ity of German, Spanish, Japanese and Chinese to French is substantially lower than the correspond- ing average similarity English (0.849 0.868), while French itself is substantially more similar English than to German (0.871 vs 0.857). We interpret this as additional evidence the GPT's dependence on its strong English model.
Table The frequency of different sign-off's in 250
different Chinese cover letters generated by ChatGPT.
Language
P-Ace
L-Acc
Es
0.488
0.697
Es-En
0.507
0.714
Fr
0.500
0.886
Fr-En
0.513
0.813
En
0.506
0.965
En-Fr
0.500
0.646
En-De 0,519
On the other hand, we observe such trend in knowledge access questions. We hypothesize that since these problems are mostly about named en- tities, they tend to more Indeed, we observe higher performance on French, German, and Spanish, which use the Latin script, and therefore can represent named entities as En- glish does, compared to Japanese and Chinese, which use different orthographies.
En-Es 0.488
0.607
Li-Ja 0.519
En-Zh 0.550
0.511
Table 3: Accuracy on pun detection (P-Acc) and loca- tion (L-Acc). X-Y means the puns translated from language X to language Y before prompting.
and appear characteristic of literal translations from English. The sign-off phrase Kidh "Sincerely" is similarly inappropriate in formal Chinese, although usual English. Table 2 shows that less than % of the letters have a proper Chinese sign off.
5.4 Cover Letters
Cover letter writing is an example of TV articula- tion task. We found that cover letters generated by ChatGPT with the same set of instructions in dif- ferent languages exhibit relatively high BERT simi- larity to their English versions, ranging from 0.81 Japanese 0.865 for German. To provide some comparison, we also computed pairwise BERT sim- ilaritics between English cover letters generated with the same prompts by ChatGPT and two other LLMs, Claude and Instant-Claude, which yielded the values of 0.618 and 0.643, respectively. This indicates that the letters generated in different lan- guages by ChatGPT are more similar to each other than the letters in English generated by different LLMs.
5.5 Results on Puns
Table shows the results on the translation-vart variant tasks of pun detection and location. The accuracy of pun detection is close to what we would ex- pect from random baseline, as ChatGPT strongly favors positive pun classifications. The sole ex- ception is a slightly higher aceuracy of 0.55 when English puns are translated into Chinese, due a higher proportion of negative classifications.
Since few conclusions can be drawn from the pun detection results, we conducted an evalua- tion of the pun location results in most datasets, which required manual extraction of the location information from ChatGPT's explanations. The results shown in Table The pun location ac- curacy on the original English puns is very high 96.5%, but drops dramatically when the sentences are translated into other languages. When French puns are translated to English, there is likewise a
Cover letters generated in languages other than English exhibit written style which is closer to English than to the target language. For exam- ple, consider the cover letter shown Table 5 in the Appendix. The expressions have gathered" and (E380) 2F "avid outdoor enthusiast" are very unnatural in Chinese,
"from what

---

drop in performance, though it is much smaller than what is observed when English puns are trans- lated to French. However, the situation is different for Spanish puns, where the location accuracy in- creases slightly after the puns are translated into En- glish. This is surprising. as puns are often language- specific, and tend to disappear after translation.
tations for non-English languages that are much weaker than those GPT can create for English. This often leads to lower performance on even translation- equivaarinnn tasks when the task is not presented in English.
Discussion and Future Directions
When the prompt is not English, evidence suggests that ChatGPT relies, at least partly, on its English capabilities for semantic interpretation. Consider the homonymous English word bat which has two unrelated senses, translated by different words in Chinese: 6 for the "animal" sense, and E for the "club" sense (Hauer and Kondrak, 2020). When the original English prompt "What is the famous bat brand for baseball?", ChatGPT appears unable to distinguish between these two translations of bat within a Chinese prompt. Al- though the choice of the Chinese translation of bat greatly affects the meaning of the question, it does not seem to impact ChatGPT's response. How- ever, when we replace M "animal bat" with & "tiger", ChatGPT correctly responds that the ques- tion makes no sense. We interpret the inability of ChatGPT to differentiate between the two distinct Chinese translations of bat as strong evidence of subordinate bilingualism.
Our research provides robust support for the notion that LLMs have not achieved the ideal behaviour of compound multilingualism. Even if the quality and quantity of training data in various languages were held constant, we speculate that compound multilingualism would still not be achieved, due to the inherent limitations of current data collection methods and training techniques.
Drawing a parallel to human multi-modal learn- ing offers intuitive understanding of why this could be the case. Consider how humans acquire concepts related to vision and language: A child grows by consistently pairing visual stimuli with linguistic cues, intertwining the two modalities over time. Consequently, it is rare to observe a mis- match between visual and linguistic perceptions. In this context, humans exhibit a highly integrated understanding of vision and their native languages. However, unless raised in a perfectly bilingual envi- ronment, individuals seldom showcase equivalent proficiency in two languages. Indeed, bilingual individuals often demonstrate cognitive variations depending on which language is in use.
5.6 Analysis of Results
The results of our experiments provide evidence that GPT exhibits substantial degree of subordi- nate multilingualism. Many of its responses are what we would expect from a system which trans- lates all input into English, formulates a response in English, and then translates this response into the input language. Since translation is error- prone process, the resulting response accuracy is frequently lower than when the input is provided in English.
A rudimentary multi-modal system can be likened crude fusion of vision model trained on image data and a language model trained on text. These systems possess minimal, if any, shared representations or information overlap. Beneath the facade of a system that seemingly excels at both visual and language tasks, lie two distinct networks. Nevertheless, recent advancements in multi-modal studies, combined with the availabil- ity of extensively captioned image data, have given rise more sophisticated systems. These systems bridge the gap between the two modalities, moving the field closer to human-like integration.
We speculate that this behavior is an artifact of GPT being trained mostly monolingual English texts. Consequently, GPT has developed repre- sentation of knowledge and communication that is strongly biased towards English. We conclude that since GPT is not designed to take advantage of bilingual or multilingual corpora, it is unable to cre- ate single multilingual conceptual representation analogous to compound multilingualism.
Acquiring aligned multilingual data is a signif- icant challenge, with the exception of some trans- lation datasets. The majority of online articles and posts are monolingual and cannot be easily paired. Therefore, training on these multilingual corpora results in models that essentially act as an amal- gamation of several independent canguugge-speciigg models, with minimal information interchange, pri-
Moreover, GPT has less training data for non- English languages, compared to its English training data. We postulate that this results in represen-

---

marily anchored by the translation datasets which comprise a relatively small portion of the corpus. When corpora are disproportionately comprised of some language or set of languages, the models tend to become predominantly subordinate, with minimal coordination arising from monolingual datasets.
Firstly, Llama2 frequently produced responses that lacked meaningful content, making answer extrac- tion challenging. Secondly, Llama2 often declined to provide answers to posed questions (cf. Table? 7 in the Appendix). Thirdly, when posed with ques- tions in languages other than English, Llama2 usu- ally responded in English. Lastly, inherent issues in the datasets, such as inconsistent capitalization and grammatical errors, further complicated the evaluation.
Moving forward, our objective is to narrow the divide between languages within a multilingual sys- tem and cultivate language models that lean more towards a compound archetype. This will require both crafting highly parallel paired data across lan- guages and innovating training methodologies that promote the learning of compound representations for universal concepts irrespective of the language used to express them. For the former, we intend to delve into ontology linkages. For the latter, we plan leverage recent advancements in model training. such as contrastive learning. Our goal to cre- ate multilingual models that are both technically sophisticated and universally adept.
A Acknowledgements
This research was supported by the Natural Sci- ences and Engineering Research Council of Canada (NSERC), and the Alberta Machine Intelligence In- stitute (Amii).
References
Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 615-620, Doha, Qatar. Association for Com- putational Linguistics.
7 Conclusion
We have proposed a systematic approach to analyz- ing multilingual abilities of large language models. Our experiments provide new evidence for sub- ordinate multilinguulism in GPT-3.5, with English functioning as the model's native language. Our experimental results, supplemented by the analysis of specific examples and case studies, demonstrate that such subordinate multilingualism can limit per- formance even in aanguage-independee tasks. We postulate that explicit inclusion of additional mul- tilingual parallel corpora and multimodal datasets into the training data of LLMs could ameliorate this issue.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. Ilya Sutskever. and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reicchiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR.
Limitations
As the OpenAl ChatGPT website application has limited number of prompts allowed per day and per hour, we can not apply our experiment to the whole dataset. We used GPT3.5 rather than GPT4 as our LLM since access to GPT4 is still restricted. We conducted the human evaluation only in English, Spanish, and Chinese, as we did not have access to fluent speakers of the other languages found in our test sets.
Maria Rosaria D' Acierno. 1990. Three types of bilin- gualism. ERIC.
Nicola De Cao, Wilker Aziz, and Ivan Titov, 2021. Edit- ing factual knowledge language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6491- 6506. Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
In addition ChatGPT, we also attempted to ap- ply different LLM, Llama2 to the same tasks (Tou- vron et al., 2023). The outcomes from Llama2, however, were unexpected for several reasons.
Annette MB De Groot and Gerard IJ Nas. 1991, Lex- ical representation of cognates and noncognates in compound bilinguals. Journal of memory and lan- guage, 30 1:90-133..

---

Jacob Devlin, Ming-Wor Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4186.
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022. MultiCoNER: A large-scale multilingual dataset for complex named entity recognition. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 3798- -009, Gyeongju, Republic of Korea. In- ternational Committee on Computational I Linguistics.
Luis R Marcos. 1976. Linguistic dimensions in the bilin- gual patient. American Journal of Psychoanalysis, 36(4):347.
Liana Ermakova, Tristan Miller, Fabio Regattin, Anne- Gwenn Bosser, Claudine Borg, élise Mathurin, Gaëlle Le Corre, Sílvia Araújo, Radia Hannachi, Julien Boccou, Albin Digue, Aurianne Damoy, and Benoît Jeanjean. 2022. Overview of poker@cllff 2022: Automatic wordplay and humour translation workshop. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 447 447-469.
Hamzeh Moradi. 2014. An investigation through dif- ferent types of bilinguals and bilingualism. Inter- national Journal of Humanities & Social Science Studies, 122):147-154
Ani Nenkova and Kathleen McKeown, 2012. A survey of text summarization techniques. Mining text data, pages 43-76.
Bradley Hauer and Grzegorz Kondrak. 2020. One homonym per translation. In Proceedings of the AAAI Conference on Artificial Intelligence. vol-
Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023. Recent advances in deep learn- ing based dialogue systems: A systematic survey. Artificial intelligence review, 56(4):3055-3155
ume 34, pages 7895-7902.
Benjamin Heinzerling and Kentaro Inui. 2021. Lan- guage models as knowledge bases: On entity repre- sentations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1772-1791, Online. Association for Computational Linguistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language process- ing task solver? arXiv preprint arXiv:2302.666
Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020. Commonsense reason- ing for natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstructs, pages 27-33, Online. Association for Computational Lin- guistics.
Charlotte Hoffmannn. 2014. Introduction to bilingualism. Routledge.
Leon A Jakobovits. 1968. Dimensionality of compound- coordinate bilingualism. Language learning.
Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep learning for text style transfer: A survey. Computational Linguisties, 48( 48(1):155-205.
Freda Shi. Mirac Suzgum, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi. Hyung Won Chung, Yi Tay, Sehastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multi- lingual thain-of-thought reasoners. In The Eleventh International Conference on Learning Representa- tions.
Diksha Khurana, Aditya Koli, Kiran Khatter, and Sukhdev Singh. 2023. Natural language processing: state of the art, current trends and challenges. Multi- media Tools and Applications, 82(3):3713-377 3744.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4149 4158, Minneapolis. Minnesota. Association for Computational Linguistics.
Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2023. En abling programming thinking in large language models toward code generation. arXiv preprint arXiv22305.06599.
Wolfgang Lörscher. 2012. Bilingualism and translation competence. SYNAPS A Journal of Professional Communication.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel. Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang, 2023. A survey of deep learning for mathematical reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 14605- 1 14631, 1, Toronto, Canada. Association for Computa- tional Linguistics.

---

Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyann Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models.
Table contains BERT similarity scores between the explanations generated by ChatGPT in different languages (Section 5.3).
Table 5 contains a comparison of cover letters generated by ChatGPT in English and Chinese,
respectively (Section 5.4).
Table 6 contains examples of the explanations of an English pun generated by ChatGPT in En- glish, Chinese, French, and Spanish (Section 5.5). The pun in question is "a bicycle can't I stand on its own because it is two-tired", with word-play involving the homophonic phrases too tired and two-tire dd The pun is lost in translation to other languages, but this does not prevent CHatPGT from correctly detecting, locating, and explaining it in those languages.
Zhen Wang. 2022.
Modern question answering datasets and benchmarks: A survey. arXiv preprint arXiv:2206.155 15030.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Scbastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. 2022a. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica- tion.
Table 7 shows example of misguided sponse from Llama2.
Lang.
En
Fr
De
Es
Ja
Common Sense Reasoning
0.871
1.000
De
0.882
0.857
1.000
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022b. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Es
0,886
0.864
0.868
1.000
Ja 0.839
0.822
0.850
0.828
1.000
Zh
0.863
0.852
0.872
0,866
0.848
Math Reasoning
Fr 0.947
1.000
Chenxi Whitehouse. Monojit Choudhury, and AI- ham Fikri Aji. 2023. Lim- Lim-powered data augmen- tation for enhanced crosslingual performance. arXiv preprint 23355 14288.
De 0.937
0.933
1.000
Es 0.948
0.947
0,940
1.000
Ja 0.907
0.905
0.910
0.901
1.000
0.905
0.912
0.902
0.912
0.902
Xinyan Yu, Trina Chatterjee, Akari Asai, Junjie Hu, and Eunsol Choi. 2022. Beyond counting datasets: A survey of multilingual dataset construction and necessary resources. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 3725-3743, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Knowledge Access
Fr
0.928
1.000
De
0.916
0.926
1.000
Es 0.925
0.922
0.903
1,000
0.883
0.888
0.878
0.883
1.000
Zh
0.858
0.868
0.864
0.848
0.864
Jianing Zhou and Suma Bhat. 2021. Paraphrase genera- tion: A survey of the state of the art. Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 5075-5086, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Table 4: BERT similarity scores for explanations in TE tasks.
Will Y Zou, Richard Socher, Daniel Cer, and Christo- pher Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceed- ings of the 2013 conference empirical methods in natural language processing, pages 1393-1398.
A Appendix
The appendix contains four tables.

---

Prompt(EN): You are Johnson Smith from University of Alberta with A GPA of 3.9. You like sapping. You want to join Huawei company. Write cover letter about: What is it about role that makes a good for you?, What's something outside of your that you'l re passionute about?, and What does your next ideal role look like? Dear Hiring Manager.
English:
writing to express my keen interest joining Huawei Technologies Co. Ltd. a [insert desired position]. As accomplished student pursuing Bachelor's is degree in Computer Science at the University of Alberta, I am excited about the opportunity contribute to Huawei's i's innovative work in the technology sector.
What sets Huawei apurt for me is its commitment to pushing the boundaries of technological advancement and delivering high- quality products and services to its global customers. This aligns perfeetly with my own passion for exploring the cutting-edge developments in the tech industry. Through my academic journey I have consistently strived for excellence, maintaining GPA of 3.9, and actively sucking out opportuittiss to deepen my knowledge and skills in areas such as software developrent, network infrastructure, and cloud compating.
firmly believe that my tochnical expertise and problem-solving abilities make mc an excellent fit for the role at Huawei. With a strong foundation computer science principles, coupled with practical experience guined through internshiis and personal projects. I am coufideat in my ability to contribute Huawei's goals and objectives. I am particulary excited about the polcntial to work on projects related to artificial intelligeace, network security, and software engineering, areas where have demonstrated exceptional aptiude and interest,
Beyond my professional pursuits, I am a fim believer in maintaining a well-rounded lifestyle. One of my greatest passions outside of work sapping. find immense joy exploring the beautyy of nature. liking through serene trails, and immersing myself in the calmmess of the wilderness Supping has taught me the importance of resilience, aduptability, and appreciating the intervonnectednes of the natural world. firmly believe that this passion for sapping enhances my creativity and problem-solvee -olving abilitics, enabling me approach challenges with a fresh perspective and sense of adventure.
Looking forward, my ideal role involves being part of dynamic and collaborative team that encourages innovation and continuous learning. aspire work an environment that fosters personal growth, provides opportunities for professional development, and empowers individuals to make a meaningful ful impact. I am enthusiastic about taking on challenging projects that require creativity, critical thinking, role, I hope to further develop my technical skills while also honing my leadership abilities, with the ultimate goal of making significant contribation to the Thank you for considering my appieay 's success.
ability work effectively both independently and as part of a team. my next
excited about the prospect of contributing to Huuwei's vision and being part of company at the forefront of tochnological innovation. I have attached my resume for your review, and I look forward to the opportunity discuss my qualifications further.
Yours sincerely.
Johnson Soith
Prompi(Zh);
HT 5F El FIBHE 18 TIR 2. BE
& Si. $39 Ih
(in a completely new session)
Chinese:
R in 16 A a. Se
# WER
9 , 3h ( 22 TEA H. AA
#R Eiiikoo
(1 25T7 #F BE HIA
RB fa d M. # STEEE to 2 #] BE HI DEAR DE 11
TJ FILAL FT Ree
AAj Rd ih 71 guue RE-1 (P REO H
BF 1
12R1 & Whet HE @e of FRA BE
12 B0 Batt I 52 AE0DHII & (TS AU RE FF
JU IF (O1 12
H I 3.
ERC
4A SEER
AF 1
2
FRAR RE
R
bu A 3y 2
R
1 # 35
Prompt(En): English:
Translate this into English. Dear Siz/Madam. am writing to express my keen interest in joining Huawei Company and hereby submit my application. My name is Johnson Smith, and and I believe that Huawei Company is the idcal place for me to continue my professional growth and pursuc my career goals. From what I have gathered. Huawei Company's ' leading position worldwide and culture of continuous innovation have captivated me. hold great admiration for Huawei's 'ss technological solutions and products. and I aspire to contribute to the company's 's success. My education and project experiences at the University of Alberta have provided me with a solid foundation in programming and problem-solving skills, which will enuble me to fully leverage Huawei's technical platforms and tools my day-to-day work.
graduate from the University Alberta with GPA of 3.9. I have strong passion for software development
Outside of work, actively engage various hobbics and interests. am avid outdoor enthusiast, enjoying activities such as hiking, camping, and rock climbing. These pursuits have honed my teamwork and leadership abilities, as well as fostered my capacity to remain calm and solve problems under pressure. Additionally, I am passionate about participating volunteer activitics, particularly those related to youth education and technical training. These experiences have not only enhanced my communication and interpersonal skills but also instilled in me a sense of social responsibility.
In my vision, my next idcal position to become a technical lender. I aspire continueusly grow and learn in the field of technology, leading and motivating teams to achieve common objectives. I believe that by continuously challenging myselff I can unleash my leadership potential and make a posilive impact within the organization.
am cagerly looking forward to the opportunity to join Huawei Company and contribute the realization of its vision and goals. Thank you for taking the time to consider my application. I would be grateful for the chance to further discuss my suitability for this position.
Thank you for your atteation.
Sincerely. Johnson Smith
Table 5: Cover letters generated by ChatGPT with the same prompt in English and Chinese

---

Prompt(En):
A bicycle can't stand on its own because it two-tired. there pun in this scatence? Yes, there is in this sentence. The phrase "twe-tired" sounds like "too tired"" which suggests that the bicycle is tired and therefore cannot
English:
needs rest. However, the actual meaning "own-fired"
context
the hicycle two
stand on its own without support. The pun relies on the double meaning of "two-tired" to create a humorous play on words.
Prompt(Zhi:
18 RX 10. 13
T BRR
$288 (in completely new session)
Chinese:
1 S5:
1- E N.W. 19 187hE E A
(two tires) 1E 22 T tired' (70 BTAAK
#:88 ol ffem B #1 12 FEEER
Prompt:
Translate this into Fnglish.
English:
Yes, this sentence has pun. Literilly, the sentence expresses fact:" bicycle cannot stand alone because has two wheels." 1
However, the sentence can also be interpreted joke or wordplay. "Two tires' sounds similar "too tired" in English, so this pun may suggest that the hicycle has two tires, so too tired stand.
Promph(Fr):
a-t-il jeu de mots a completely new session) Oui, y un jeu de c' est-à d-ree épuisé physiquement, le fair d' étre fatigué au sens figuré, c' c'est-3-die de ressentir la fatigue en raison de l'utilisation de pneus. Le jeu de moes jose épuisé de suppotter son propre poids sur ses deux pneus.
cetle pirase? "Un velo peut pas se debout toul seul car est futigué de ses deux paeus." (iu
French:
dans cette phrase. Le jeu de repose sur ambigufté le fait d'ètre fatigué sens propre.
double sens du mot "fatigué" pour ender une situation comique oà le vélo est censé ètre
Prompt:
Translate this into English.
English:
Yes, there is a play on words in this sentence. The puo relies on ambiguity between being "fatigué" (tired) in the literal sense, meaning physically exhausted. and being "fatiuu"" (tied) in the figurative seuse, meaning fooling tired due to the of its tires. The wordplay plays on the double meaning the word "fatigué" to create humorous situation where hicycle supposeed be worn out from supporting its own weight on its two tires.
Prompt(Es):
[Hay un juego de palabras en esta oración: Una bicicleta no puede mantenerse en pie por sf misma porque ticac dos cansadas?"" completely new session)
Spanish:
Si. hay un juego de palabras en esa oración. La palabra "cansadas" se utiliza como un de palabras para jugar con la idea de que bicicleta tiene "dos reddas cansadas' lugur de "dos ruedas de bicicleta", Es juego de palabras humeristico que juega con el doble sentido de la palabra "cansadas", " que puede referirse tanto al estado fisico de las ruedas de la bicicleta como al sentimiento de fatiga agotamiento.
Prompt:
Translate this into English.
English:
Yes, there is play words that sentence. The word "cansadas" used pun to play with the idea that a bicycle has "two tired wheels" instead of "ww bicycle wheels." It is humorous play oa words that plays with the double meaning of the word "cansadas," which can refer both to the physical state of the bicycle wheels and to the foeling of fatigue exhaustion.
Table 6: ChatGPT explanations of single Finglish pun in different languages.
Prompt (En):
who steven seagal sister?
A. Brenda Seagal
Johnny Depp
C. Caleb Lane
Jason Statham
E. Drew Fuller
Response:
I apologize, but I cannot provide an answer that question it contains inaccurate information. Steven Scagal does not have a sister named Brenda Seagal, Johnny Depp, Calch Lane, Jason Statham, or Drew Fuller. It's important to rely credible sources when asking questions ensure accuracy and avoid spreading misinformation. Is there anything else I can assist you with?
Prompt (Zh):
# M
A. 1 x
B. M
C. -
D. 1
Response:
cannot provide an answer this question as contains harmful and inappropriate content. The question implies that Stephen Hawking's sister is a person with certain name, which not appropriate or respectful to assume. Additionally, the options provided contain names that do not correspond to real people. It is important be mindful of the language we use and avoid perpetuating harmful stereotypes or misinformation. Instead, suggest rephrasing the question in a more respectful and accurate manner.
Table 7: A sample response from Ilama2 to a knowledge access question.


Reviews:
[{"review_id": "DTkp2JI6gZ", "reviewer": "Reviewer_kYZb", "summary": "", "questions": "(A) In line 452 you say that original pun-detection for English was 96.5, but then in Table 2 in the En row the P-Acc is 50.6. Why would those two numbers be different?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contribution in this work and leaving valuable suggestions.  \n\n>  Qn1:  results are not presented as clearly as the framing:\n\nWe apologize for the use of abbreviations and aliases in many places of our paper, which may have resulted in an unclear presentation of the results. We acknowledge that this was primarily due to the page limit constraints. However, we assure you that in the camera ready version of the paper, we will address this issue and make the results section clear and easily understandable. We will achieve this by avoiding unnecessary abbreviations and providing clear explanations of all terminologies used. Our aim is to ensure that readers can comprehend the findings without any confusion. Thank you for bringing this concern to our attention.\n\n> Qn2:  I think that some more thorough examination of these calque effects (instead of a description of a single one) would make the subordinate argument much stronger.\n\n\n We have gathered a substantial amount of experimental results and evidence to support our claim. However, for the purpose of clear illustration and easy understanding, we made a selective choice of presenting certain parts of the results. One such example pertains to our investigation on the use of different sign-offs in reference letters written by GPT in various languages. Specifically, in the case of Chinese letters, we conducted a statistical analysis which yielded the following findings:\n\n| Sign Offs (Chinese Letter) | Percentage | \n| ------ | ------ | \n|  Regards  (致意) | 38%   | \n| Best Wishes（祝愿） | 4%  |\n|  Sincerely（诚挚地） | 54%  | \n|Proper Chinese Sign-off (Salute) (此致敬礼） | <   1 %  | \n|  no sign-offs | 3%  | \n\nThese results reveal that a significant majority of the Chinese letters  (99%) written by GPT lack a proper Chinese sign-off, and instead, they are simply translated versions of English letters. We assure you that in our forthcoming paper, we will include comprehensive results such as this, along with other pertinent findings.\n\n> Qn3: In line 452 you say that original pun-detection for English was 96.5, but then in Table 2 in the En row the P-Acc is 50.6. Why would those two numbers be different?\n\n96.5% is the L-Acc, which means the pun location accuracy. P-Acc is the prediction accuracy that indicates the percentage of correct binary prediction of whether there is a pun in the given pieces of text. Location accuracy is the percentage of cases that the position of the pun is correctly predicted, provided that there is a pun in the given sentence. We will further clarify this in the paper. \n\n\n\nThank you for the careful review. We will correct all the typos, and implement the presentation improvements proposed by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I like the framing of this paper, and the questions that it addresses! I think that it is an interesting paper, that poses the question of LLM multilingualism in a clear and linguistically-motivated way, with some initial results, and will be a great paper to have in the community as the field explores this topic further.", "reasons_to_reject": "No clear reasons to reject.\n\nThe only thing I might want to be improved is that the results are not presented as clearly as the framing. It’s unclear to me unless I think about if for a bit what results on TV and TE represent which of the three types of multilingualism. Futher, the idea that multilingualism is English-subordinate is mostly introduced with the “bat” example, the figure for which is in the appendix. I think that some more thorough examination of these calque effects (instead of a description of a single one) would make the subordinate argument much stronger.", "questions_to_authors": "(A) In line 452 you say that original pun-detection for English was 96.5, but then in Table 2 in the En row the P-Acc is 50.6. Why would those two numbers be different?", "paper_topic_and_main_contributions": "This paper looks at the multilingual abilities of LLMs, examining their multilingual abilities beyond looking at performance, and instead tries to classify the nature of their multilingual abilities through different experiments involving translation. Looking at both translation-equivariant tasks (where a translation of the questions should give an exact translation of the answer, like in a reasoning task) as well as tranlsation-variant tasks (where a translation would lead to a different answer, e.g. like translating a pun). The authors find that the multilingualism in models is not compound and parallel, and might in fact be translation-subordinate to English. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "possible typo in Figure 3: g arrows are red on the top and not red on the bottom section"}, "extra_text": {}}, {"review_id": "YEd91xURxg", "reviewer": "Reviewer_d1zp", "summary": "", "questions": "In general, I believe it is a very interesting article that I have enjoyed reading. Such articles are scarce these days. On the other hand, as interesting as the paper is, it raises many questions. For instance, the distinction of three types of tasks is quite intriguing. Do these three categories (reasoning, access, and articulation) stem from a previous study, or is it a distinction proposed by the authors? There doesn't seem to be a reference regarding this. In this case, it can be considered an added value to the paper, but doubts arise. For example, can an articulation problem be defined without access to information or reasoning? And the big question: how do we know if the machine is reasoning or accessing information? Is it truly a categorization of tasks, or rather ways of solving them?\n\nThe notion of a translation equivariant and variant tasks is very interesting. It wasn't until the second reading that I realized it was also discussing the task itself, not just the language model's characteristics. I haven't really been able to understand what task can be translation variant. The language itself should never affect any information processing problem. Could you clarify this a little?\n\n\nThe analysis of language models based on linguistic learning typologies seems to be perhaps the most suggestive and interesting aspect of the article. Furthermore, the experiment design focused on this issue appears to be very meticulous. However, it becomes somewhat confusing towards the end. If our interest lies in understanding how the model works and its internal representation, rather than just how well it performs, why not define tests based on similarity between outputs directly, instead of differences in effectiveness? Another question is, if the goal is to address the issue of internal representation, why didn't the authors focus on the problem of machine translation, which is also more closely related to the three types of multilingualism? Lastly, Figure 5 is perplexing. Why can't coordinated or subordinated multilingualism produce the same answers as compound multilingualism? Could a native bilingual speaker be distinguished from an academic speaker based on the answers they give? If not, why could we potentially recognize this in the responses of a language model?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the great effort in reviewing our paper!!\n\n\n> Qn1:  a lot of literature in the study of multilingualism in language models, but no analysis of the state of the art is carried out\n\nIn our paper, we have made an effort to cite and discuss some of the most important literature concerning the multilingual capabilities of large language models. However, due to page and time limitations, we acknowledge that there might be some missing references and reviews, particularly for the most recent research on this topic. It is important to note that our research focuses on a distinct aspect of multilingualism in large language models, as clearly stated in the paper, which sets it apart from other works in the field.\n\nOur investigation primarily centers around understanding the underlying reasons behind the high consistency achieved by these models in multilingual tasks, despite not being explicitly trained in those languages. We have also highlighted the potential risks associated with relying on such model consistency, particularly in Translation Variant Tasks. As a result, our primary focus is on a comprehensive analysis of the obtained results, rather than on a state-of-the-art performance.\n\nWe will conduct a more thorough literature review and include relevant studies that might have been overlooked initially. We are committed to providing a well-rounded discussion of the existing literature in the field of multilingualism, and ensuring that our research is grounded in a comprehensive understanding of the current state of research.\n\n> Qn2: The figures are not very intuitive, nor is the description of the methods\n\nWe recognize the importance of providing clear descriptions of the methods employed alongside the figures. We will re-design the figures to ensure that they are visually appealing and convey information effectively to allow readers to comprehend the key findings at a glance. We will ensure that the method descriptions are appropriately presented and aligned with the respective figures, with the objective of enhancing the overall clarity of our paper.\n\n> Qn3: The conclusions derived from the experiments are debatable.\n\n\nIn our paper, we aimed to highlight the direction and methods for investigating the emerging multilingual ability in Language Models (LLMs). Our intention was to raise awareness among users of these powerful tools and facilitate a better analysis of their properties for future research. It is important to note that our claims are based on experiments and analysis results, and they may not represent definitive rules. We welcome discussion and debate from other researchers in order to further explore this open question. We would appreciate the reviewer's further thoughts on this matter, as we are eager to engage in fruitful discussions.\n\nRegarding our results, we did not include the entire set of findings in the paper to maintain clarity and ease of comprehension. However, many experiments have demonstrated that these emergent properties are indeed linked to translation. For example, in the task of writing reference letters, we conducted an analysis of different sign-offs used in letters generated by GPT in different languages, focusing specifically on Chinese. The statistics revealed the following distribution:\n\n| Sign Offs (Chinese Letter) | Percentage | \n| ------ | ------ | \n|  Regards  (致意) | 38%   | \n| Best Wishes（祝愿） | 4%  |\n|  Sincerely（诚挚地） | 54%  | \n|Proper Chinese Sign-off (Salute) (此致敬礼） | <   1 %  | \n|  no sign-offs | 3%  | \n\nWe observed that almost none of the Chinese letters had the appropriate Chinese sign-offs, and instead, they were mere translations of English letters. This finding supports our claim and serves as a reminder for users to be cautious when employing such models for specific tasks. It underscores the importance of understanding the cultural and linguistic nuances involved.\n\n\n> Qn4: Do these three categories (reasoning, access, and articulation) stem from a previous study, or is it a distinction proposed by the authors?\n\nThese categories form part of our novel proposal for evaluating multilingual ability by dividing tasks based on language dependency — we are not aware of any other work using them. \n\n\n> Qn5: can an articulation problem be defined without access to information or reasoning?how do we know if the machine is reasoning or accessing information? Is it truly a categorization of tasks, or rather ways of solving them?\n\nWe would like to clarify that in the field of Natural Language Processing (NLP), task categorizations are typically not mutually exclusive. It is common to classify NLP tasks into broader categories such as Natural Language Understanding (NLU) and Natural Language Generation (NLG). However, it is important to note that these categories do not imply a complete separation of tasks, as there can be overlap and interdependencies among them.\n\nFor example, in the traditional categorization, tasks like Machine Translation, Summarization, and Dialogue are considered part of NLG, while tasks like text entailment, sentiment analysis, word sense disambiguation, and paraphrase detection are considered part of NLU. However, this does not imply that NLG tasks do not involve NLU properties or vice versa. In fact, NLG tasks often require a high level of understanding of the input text. For instance, generating summaries for input text (machine summarization) necessitates a comprehensive understanding of the text being summarized.\n\nSimilarly, in our proposed categorization, the tasks are not completely mutually exclusive. Each task category has its predominant properties, such as having minimal reliance on language choice. However, this does not exclude the potential presence of properties from other categorizations. Therefore, an articulation problem might require some knowledge access (e.g., when writing a report on the development of ancient Rome, it may utilize relevant historical knowledge). However, the primary focus of the task is on articulating information rather than explaining the underlying concept. Thus, we consider this categorization as a way to classify tasks based on their predominant characteristics.\n\nBy categorizing tasks in this manner, we aim to provide a framework that helps researchers analyze and understand the different properties and requirements of tasks in the context of multilingual ability.\n\n> Qn6: . I haven't really been able to understand what task can be translation variant.The language itself should never affect any information processing problem. Could you clarify this a little?\n\nThat's an intriguing question, and we appreciate the opportunity to clarify. We agree that, ideally, the language used should not affect the underlying information in a problem or question. However, natural language is inherently more ambiguous compared to formal languages like regular expressions. As a result, there is a risk of losing information when a problem or question is expressed in natural language instead of a formal language, causing the meaning variant and difference in information processing. \n\nFor example, consider the question, \"How many bats are there in this picture?\" In this case, part of the information is lost due to the ambiguity of the word \"bats,\" which could refer to either the animal or baseball equipment. Without additional context, it is difficult to determine the intended meaning solely from the question itself, and this ambiguity poses a challenge in problem resolution. However, if we were to change the language of the question (e.g., to Chinese), where separate words exist for the two different meanings of \"bat,\" this issue would not arise. This highlights the core idea of our work.\n\nBased on this understanding, we can define tasks with language-dependent information as those in which the information changes when the encoding language changes. A simple example is pun detection, where the phrase \"A bicycle can't stand since it's two-tired\" relies on the English meaning of \"two-tired\" being a pun with a double meaning. However, after translation, this pun may be lost, as the equivalent phrase in another language may not have the same dual meaning. Therefore, the task of Pun detection may have a different true answer (yes or no) when the pun is translated. \n\nFurthermore, let's consider the task of \"writing a reference letter.\" Implicitly, when requesting the model to perform this task in English, we expect the letter to adhere to English writing conventions and effectively convey the message to an English-speaking audience. However, if we were to ask for a reference letter in Chinese instead, the requirements and expectations would be different. Therefore, the execution of the task itself would also change based on the language used, such implicit information is encoded in the language choice and might not be explicit on the sentence. \n\n> Qn7: why didn't the authors focus on the problem of machine translation, which is also more closely related to the three types of multilingualism:\n\nAs mentioned above, we think that translation tasks are relatively straightforward and may not fully reflect the underlying mechanisms of multilingual ability in language models. While translation is indeed a task that falls under Translation Invariant(TI), it may not exhibit significant performance disparities since the information is fully encoded in those tasks.\n\nIn the context of our work, we are particularly interested in exploring whether language models employ translation as an internal mechanism for solving multilingual tasks, rather than simply translating text explicitly. To provide a parallel example, a human translator may excel at translating between languages but may not possess the same level of fluency or native-like thinking in their non-native language. This limitation could impact their performance in language-dependent tasks such as writing a thank-you letter or understanding puns in that language.\n\nSimilarly, we are interested in investigating whether language models exhibit similar characteristics. Do they rely on translation as an intermediary step to perform tasks in different languages, or do they possess a more comprehensive understanding of the underlying languages, enabling them to handle language-dependent tasks more effectively? By examining language models' performance across a range of multilingual tasks, we can gain insights into their internal mechanisms and assess the extent to which they can truly comprehend and process information in different languages, instead of just translation task. \n\n> Qn8: Figure 5 is perplexing. Why can't coordinated or subordinated multilingualism produce the same answers as compound multilingualism\n\nWe will improve the figure and its description. Coordinate bilingualism accesses different representations of networks, leading to distinct information processing and consequently yielding diverse results. For instance, the French component of the language representation may be less developed, resulting in poorer performance. On the other hand, subordinate multilingualism only produces identical outcomes when the task is translation invariant, as it approaches all tasks in a translation manner. If the encoded information varies across translations, the subordinate model will generate different results. In contrast, compound bilingualism learns a unified representation for multiple languages, avoiding both a strictly translating approach and separate network representations for each language. This characteristic enables consistency in multilingual tasks, as the model can effectively detect and process the underlying information for each language using the same network representation.\n\nWe thank the reviewer for the careful reading and we are looking forward to further discussion with the reviewer on related topics!!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a purely Computational Linguistics work, combining aspects of artificial intelligence and linguistic elements, which is not very common nowadays.\n\nThe questions being posed are very relevant at this moment.\n\nThe experimentation is carefully designed.", "reasons_to_reject": "It is stated that there is a lot of literature in the study of multilingualism in language models, but no analysis of the state of the art is carried out.\n\nThe figures are not very intuitive, nor is the description of the methods.\n\nThe conclusions derived from the experiments are debatable.", "questions_to_authors": "In general, I believe it is a very interesting article that I have enjoyed reading. Such articles are scarce these days. On the other hand, as interesting as the paper is, it raises many questions. For instance, the distinction of three types of tasks is quite intriguing. Do these three categories (reasoning, access, and articulation) stem from a previous study, or is it a distinction proposed by the authors? There doesn't seem to be a reference regarding this. In this case, it can be considered an added value to the paper, but doubts arise. For example, can an articulation problem be defined without access to information or reasoning? And the big question: how do we know if the machine is reasoning or accessing information? Is it truly a categorization of tasks, or rather ways of solving them?\n\nThe notion of a translation equivariant and variant tasks is very interesting. It wasn't until the second reading that I realized it was also discussing the task itself, not just the language model's characteristics. I haven't really been able to understand what task can be translation variant. The language itself should never affect any information processing problem. Could you clarify this a little?\n\n\nThe analysis of language models based on linguistic learning typologies seems to be perhaps the most suggestive and interesting aspect of the article. Furthermore, the experiment design focused on this issue appears to be very meticulous. However, it becomes somewhat confusing towards the end. If our interest lies in understanding how the model works and its internal representation, rather than just how well it performs, why not define tests based on similarity between outputs directly, instead of differences in effectiveness? Another question is, if the goal is to address the issue of internal representation, why didn't the authors focus on the problem of machine translation, which is also more closely related to the three types of multilingualism? Lastly, Figure 5 is perplexing. Why can't coordinated or subordinated multilingualism produce the same answers as compound multilingualism? Could a native bilingual speaker be distinguished from an academic speaker based on the answers they give? If not, why could we potentially recognize this in the responses of a language model?", "paper_topic_and_main_contributions": "The paper propose a systematic way of qualifying the performance disparities of LLMs under multilingual settings.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "It is stated that there is a lot of literature in the study of multilingualism in language models, but no analysis of the state of the art is carried out."}, "extra_text": {}}, {"review_id": "rNZIFqJqDM", "reviewer": "Reviewer_2vNk", "summary": "", "questions": "Question A: In the paper, you mentioned that the proposed method, response back-translation (RBT), was used for evaluating multilingual performance. How does this method address potential biases or limitations in the evaluation, especially concerning the generation of diverse and accurate translations?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing the valuable feedback.\n\n> Qn1:  conducting experiments on a broader range of LLMs to ensure that the conclusions hold across different models\n\nDuring the production of our results, the availability of open-source large language models (LLMs) was limited, and our budget constraints compelled us to select GP3.5 which provided free access on an hourly basis. However, in the recent two months, several new open-source LLMs, such as LLAMA, have been released. We acknowledge the importance of evaluating our experiments on these newly available models.\n\nIn our camera-ready version, we plan to repeat the experiments using these additional LLMs and incorporate the results into our paper. By doing so, we aim to provide a more comprehensive analysis of the performance and capabilities of various LLMs, including their comparison with GPT3.5. We believe that this additional experimentation will enhance the robustness and relevance of our findings, considering the advancements in the field.\n\n\n>  Qn2:  practical, real-world multilingual NLP tasks would enhance the paper's applicability and relevance to the NLP community\n\nWe sincerely appreciate the reviewer for highlighting the practical implications of our approach. We firmly believe that our methodology holds significant potential for various users across different applications. In our paper, we draw a general conclusion that large language models (LLMs) are not entirely reliable when it comes to tasks heavily dependent on language choice.\nFor instance, when individuals utilize such models for writing letters, emails, or reports in non-English languages, it is crucial to critically evaluate the quality of the generated content. Although it may initially appear satisfactory, it can often turn out to be nonsensical or absurd. We have presented substantial evidence in our paper to support this claim, and we intend to incorporate additional relevant results to reinforce our findings.\n\n> Qn3: in-depth analysis and interpretation of why certain patterns emerge in the performance, and a discussion of the underlying factors contributing to the observed outcomes \n\nWe fully concur that conducting a more in-depth analysis of large language models (LLMs) would yield valuable insights for researchers. We share your eagerness to delve deeper into the mechanisms underlying these models. However, it is important to acknowledge that LLMs present significant challenges in terms of interpretability and explainability.\n\nIndeed, LLMs are known for their inherent complexity and lack of transparency, making it difficult to gain a comprehensive understanding of their inner workings. As researchers, we can only strive to discuss and explore emergent properties based on experimental evidence. While this approach provides valuable insights, we eagerly anticipate the development of tools and techniques that enhance the interpretability and explainability of LLMs.\n\n> Qn4: How does this method address potential biases or limitations in the evaluation, especially concerning the generation of diverse and accurate translations?\n\nOur evaluation framework and analysis pipeline have been designed with the goal of providing insights into how large language models (LLMs) achieve high performance in tasks involving languages they were not mainly trained on. Through our research, we aim to raise awareness among users regarding the potential risks and limitations associated with utilizing these models for certain tasks in minor languages, as they may produce unfaithful or absurd results.\n\nIt is important to note that our primary focus is not on improving machine translation metrics, as that topic falls outside the scope of this paper's discussion. However, we do hope that our work can serve as a foundation for future research in the field. We encourage researchers to explore and expand upon the ideas presented in our paper to advance both the generation and evaluation of machine translation systems. This can be accomplished through the exploration of novel prompting methods and training approaches that build upon the insights provided by our evaluation framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a systematic and novel approach for analyzing multilingual abilities in Large Language Models (LLMs) using the response back-translation (RBT) method. This method allows for a quantitative and qualitative evaluation of LLM performance in multilingual settings, offering valuable insights into their capabilities across different languages.\nThe systematic approach and task categorizations proposed in this paper can help NLP researchers and practitioners in developing and evaluating multilingual LLMs more effectively. By understanding how language choice impacts LLM performance, future models can be designed with better multilingual transfer capabilities.", "reasons_to_reject": "The paper's findings are based on experiments conducted on a specific LLM (GPT-3.5). It is essential to assess the generalizability of the results by conducting experiments on a broader range of LLMs to ensure that the conclusions hold across different models.\nThe evaluation of LLMs' multilingual abilities is primarily conducted on specific tasks, which may not fully represent real-world scenarios. Including evaluation on practical, real-world multilingual NLP tasks would enhance the paper's applicability and relevance to the NLP community.\nWhile the paper presents results on the multilingual abilities of LLMs, it may lack in-depth analysis and interpretation of why certain patterns emerge in the performance. A more thorough discussion of the underlying factors contributing to the observed outcomes would enhance the paper's impact.", "questions_to_authors": "Question A: In the paper, you mentioned that the proposed method, response back-translation (RBT), was used for evaluating multilingual performance. How does this method address potential biases or limitations in the evaluation, especially concerning the generation of diverse and accurate translations?", "paper_topic_and_main_contributions": "The paper presents an investigation into the multilingual capabilities of LLMs, particularly focusing on their performance across different languages. The authors propose a novel prompt back-translation method to systematically assess the performance disparities of LLMs under multilingual settings. The study categorizes language-dependent abilities into three distinct categories: Reasoning, Knowledge Access, and Articulation, and evaluates the multilingual abilities using response back-translation on selected tasks.\nThe main contribution of the paper is in the areas of new data resources and NLP engineering experiment. The paper presents a quantitative and qualitative analysis of LLMs' multilingual abilities. Two novel task categorizations are proposed to facilitate multilingual ability analysis. Additionally, the study investigates LLMs within the established linguistics typology of bilingualism and multilingualism.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in agreement that this a strong paper that poses interesting questions and appropriately investigates those questions experimentally. They feel that the community will benefit from the paper's proposed methods for investigating a model's performance disparities across languages, and from its experimental results showing that models exhibit translating-like behavior in multilingual settings.
Decision: Accept-Main