Inductive Linguistic Reasoning with Large Language Models

Authors: Raghav Ramji, Keshav Ramji
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: language models, linguistic reasoning, prompting, analogical reasoning, linguistics puzzles

Abstract:
Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. In this work, we investigate the abilities of such models to perform abstract multilingual reasoning through the lens of linguistic puzzles on extremely low-resource languages. As these translation tasks involve inductive and deductive reasoning from reference instances, we examine whether diverse auxiliary demonstrations can be automatically induced from seed exemplars, through analogical prompting. We employ a two-stage procedure, first generating analogical exemplars with a language model, and then applying them in-context along with provided target language exemplars. We explore various combinations of language models as analogical generators and reasoning agents, testing different model sizes and specialized multilingual LLMs. Our results on the modeLing dataset show that analogical prompting is effective in eliciting models' knowledge of language grammar similarities, boosting the performance of GPT-4o by as much as 8.1\% and Llama-3.1-405B by 5.9\% over chain-of-thought approaches. These gains are realized with self-generated analogical demonstrations as well as those generated by weaker multilingual models. We also report several findings about interesting phenomena which drive linguistic reasoning performance, suggesting that such puzzles are a valuable benchmark for new reasoning methods.

Decision: reject
Reviewer scores: 6, 5, 5, 1

Reviews and discussion:
<thread>
creation date:
1729858158331

thread_type:
official_review

<thread_object>
creator: reviewer_kxr4

<content>
summary:
The paper explores the capabilities of LLM in performing linguistic reasoning on low-resource languages through language puzzles. This study uses the ‘analogical prompting’ approach, which enhances the reasoning capabilities of these models by using analogy-generated examples to improve performance in translation tasks, particularly in low-resource languages.

The idea is very interesting, and this is the first contribution that transfers the idea beyond English. However, there are some really serious points that emerge (detailed below). This does not put the paper in a good light and it strongly needs revision.

strengths:
The idea is interesting because using the analogical reasoning approach proposed in ‘Large language models as analogical reasoners’ on multilingual tasks is a methodology that has been little explored and apparently shows promise.


The authors propose a good experimental setting and a comprehensive discussion, however with difficulty one understands some passages

weaknesses:
Among the paper's weaknesses are:

- The image Figure 1 is very confusing and is really difficult to read as it is of very poor quality.

- Many steps should be carefully explained e.g. the heart section (section 2 introduces the method that should emulate multilingual analogical reasoning). No examples are given in this section and the problem is not formalised, confusing the reader.

- The experiments, although many, are poorly introduced and the thread is not understood.

questions:
How did you conduct the evaluation?

Do you plan to release the code publicly?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730389403064

thread_type:
official_review

<thread_object>
creator: reviewer_sdwx

<content>
summary:
The paper explores analogical prompting to solve modeLing (Chi et al., 2024), a dataset containing International Linguistics Olympiad-style problems. Through experiments with proprietary and open-source models using different prompting strategies, the authors demonstrate that few-shot chain-of-thought prompting with explanatory rationales yields optimal performance. They further suggest including analogical exemplars ( language family information obtained through LLM prompting) in prompts can enhance model performance.

strengths:
1. There are limited works on solving Linguistics Olympiad problems.  This paper's methodology is valuable as a benchmark for future studies.
2. The study presents comprehensive experiments across various models and prompting techniques, with a clear presentation of results.

weaknesses:
1. The paper's contribution is primarily empirical, with limited conceptual innovation. The approach of using analogical prompting to boost performance is not very inspiring, as it mainly involves augmenting prompts with self-generated information [1].

2. The authors tested their method only on machine translation tasks, overlooking other question formats in IOL, such as multiple-choice and cloze questions. A more suitable benchmark than modeLing would be [2] or [3].

3. It is widely known that closely related languages help with cross-lingual transfer [4] [5]. This paper, however, does not seem to provide any novel insights in this area.


References:

[1] Sun, Z., Wang, X., Tay, Y., Yang, Y., & Zhou, D. (2022). Recitation-Augmented Language Models. ICLR 2023.
 
[2] Sánchez, E., Alastruey, B., Ropers, C., Stenetorp, P., Artetxe, M., & Costa-jussà, M. R. (2024). Linguini: A benchmark for language-agnostic linguistic reasoning. arXiv preprint arXiv:2409.12126.

[3] Bean, A. M., Hellsten, S., Mayne, H., Magomere, J., Chi, E. A., Chi, R., ... & Kirk, H. R. (2024). LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages. arXiv preprint arXiv:2406.06196.

[4] Dan Malkin, Tomasz Limisiewicz, and Gabriel Stanovsky. 2022. A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank. NAACL 2022.

[5] Vésteinn Snæbjarnarson, Annika Simonsen, Goran Glavaš, and Ivan Vulić. 2023. Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese.NoDaLiDa 2023.

questions:
1. In Table 1, zero-shot scores are near zero across all models, which is unexpected, given that BLEU metrics are relatively lenient. Any insights into why this might be the case?

2. Line 311-313: "Our findings suggest that when equipped with the right tools (analogical demonstrations) from effective multilingual reasoners, strong deducers can thrive.". However, in Table 2, using Aya-23-35B as the generator yields better results than Llama-405B (which performed better in prior evaluations) when GPT-4o is the deducer. Does this imply that Aya excels at language identification rather than machine translation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730395590288

thread_type:
official_review

<thread_object>
creator: reviewer_uiez

<content>
summary:
This paper investigates the possibility of using few shot learning so that LLMs can generalize their knowledge to new and highly under resourced languages at inference time.  The authors introduce a new method of prompting called 2 stage analogical prompting, according to which for a given language problem P for a novel or very under resourced language L they first get a model to infer what family of languages L belongs to, then they get the model to select languages in L's family and to produce language problems similar to the given one P.  These results are then fed into either the same or a different model to then solve the original language problem P.  The authors show that their 2 stage analogical prompting delivers superior results to other  state of the art prompting (CoT) and methods without CoT.

strengths:
The 2 stage analogical prompt is interesting and suggests that perhaps models might leverage information about related but more represented languages to solve the given linguistic problems in the test set.  There is also an interesting difference between larger models like Llama 405B or GPT4o and smaller models; the analogical exemplars work for the larger models but not the smaller ones, pointing to an ability of the larger models to adapt the analogical examples to the given linguistic problem that the smaller models lack.

weaknesses:
The paper's main weakness is the disconnect between the empirical investigations, which seem sound enough, and the desired conclusion that is given here: "In summary, our results suggest that the ability of the model to deduce from inductively learned rules is the key performance driver."  In other parts of the paper the rules referred to here would seem to be grammar rules.  There is little in the paper to suggest in the results that any grammar rules have been really learned or what the form of the grammar rules might be.  For example the rules could involve simple agreement or complex long distance effects governing ellipsis, gapping, or some other complex grammatical phenomenon.   At least this reviewer would like to see a much more detailed study in which (i) the grammar rules at issue are clearly stated, (ii) we have results for patterns that are governed by the rules (iii) we have results for constructed examples that violate those rules.  I would expect that for examples that violate the rules the models would either fail to produce an output or flag it in some way, if they had learned the grammatical rules.  The paper provides no such data, and so we can't really conclude anything about the mechanism that the models used to infer correct solutions to the language problems posed.

Another problem with this paper is the reference to linguistic problems that aren't really very well described.  One can gather that at least some of the problems are translation problems.  But are they all translation problems?  If so, how on an olympiad test would a participant be able to get a good translation for a completely unknown language without any clues?   The whole experimental basis of the paper is kind of murky and needs to be cleaned up for those readers who are unfamiliar with the linguistic olympiads.  

The strengths of the paper could be improved by looking into more detail as to what 2 step analogical reasoning is doing.
I might of missed it but it seems that the paper itself doesn't contain a discussion of what happens when the language family is omitted but the examples are provided.  It would have been nice to have a more detailed stucy of the analogical reasoning itself.

questions:
Please describe in more detail the test linguistic problems in this study.

What are rationales in the particular case of linguistic problems?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730596634269

thread_type:
official_review

<thread_object>
creator: reviewer_entm

<content>
summary:
The paper explores LLMs' linguistic reasoning using linguistic puzzles on extremely low-resource languages. Its key contribution is a two-stage analogical prompting method, where the model first generates examples from related languages and then applies these to deduce grammar rules in a target language.

strengths:
**Originality**

The paper introduces an innovative approach to evaluating linguistic reasoning in LLMs through analogical prompting. It applies this method to extremely low-resource languages and further evaluates generating exemplars through a different LLM, increasing overall performance.

**Quality**

The paper presents experimentation across multiple models and prompting strategies.

**Clarity**

The paper is well-structured, with clear explanations of each experimental setup, metric, and finding.

**Significance**

The paper highlights advancing the understanding of LLMs' reasoning capabilities across diverse languages. The focus on low-resource languages underscores the broader implications of this work for multilingual AI and low-resource language preservation.

weaknesses:
1. Section 4 mentions that each response was manually evaluated to provide exact match scores, but this evaluation process lacks details. Specifically, there’s no mention of how many responses were reviewed, how many LLMs were involved, the number of evaluators, or their inter-annotator agreement. Without this, it’s challenging to assess the reliability of the manual evaluation.

2. Section 5.2 mentions other linguistic reasoning datasets, yet these were not utilized in the experiments. Incorporating additional benchmarks would provide more reliable and generalizable results.

questions:
1. The paper briefly mentions that frontier models like GPT-4o and Llama-3.1-405B-Instruct often successfully identify language families. How accurately do LLMs identify language families, and how often do they correctly solve queries when the language family identification is accurate?

2. The results show that the mixture setting—where analogical exemplars are generated by one model and applied by another—outperforms the self-generation setting, but the paper does not delve deeply into why this occurs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732354450248

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (Part 2/2)

comment:
3. *Discussion of 2-Stage Analogical Prompting with Inferred Language Families*: To examine in further detail what happens during 2-stage analogical prompting (inferring the language families, without specifying that family through an oracle label, as in Figure 2b), we analyze the language family labels produced by Llama-3.1-405B-Instruct and GPT-4o in the first stage of our analogical prompting procedure. Note that this is prior to the model identifying specific languages from said family and producing the analogical exemplars. We compare these labels against the oracle labels in the table included in Appendix F, to yield a correctness score; we include these tables in Appendix G of our revised paper. Llama-3.1-405B-Instruct's language family correctness out of the 272 samples, relative to the oracle labels in Appendix F is $\frac{249}{272} = 91.54\%$, while GPT-4o's rate is $\frac{202}{272} = 74.26\%$. This reinforces our belief in the Llama models being strong multilingual reasoners. However, the model does not necessarily need to identify the correct language family in order to produce the correct answer. For instance, the Aya-35B exemplars applied in the weak-to-strong setting do not include any explicit family labels, jumping immediately into choosing similar languages and generating exemplars, which proves effective, as exhibited in Table 2. 

### Revisions to Paper
We have posted a revised draft of our paper, incorporating the valuable feedback of the reviewers. We list the changes made below, which may also be visible in green in the revision:
* Added pointers in Sections 3.1 and 3.2 to their corresponding experiments in Sections 4.1 (baselines) and 4.2 (analogical prompting), respectively.
* Rephrased the final paragraph of Section 4.2.
* Updated Figure 1 to better illustrate the analogical prompting method and improve its clarity.
* Added a subsection (2.1) to discuss the linguistic reasoning (“Rosetta Stone”) puzzles studied in the work, with an example.
* Analyzed the language families identified in the self-generated analogical examplars with inferred families experiments and included this breakdown in Appendix G.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732354500159

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (Part 1/2)

comment:
We sincerely thank the reviewers for their valuable feedback, comments, suggestions, and questions. We would like to clarify certain aspects of our work that were raised by the reviewers, and which we have sought to address in the revised version of our paper. 

1. *Evaluation Method*: Our evaluation is primarily performed using exact match, given the stated concerns over corpus-level BLEU and ChrF2 scores (included nonetheless in the appendix), despite being a fairly strict criterion. This aligns with the scoring procedures for many computational linguistics competitions / linguistics olympiads. The introduction of a human annotator (an author of this work) to assess the generated response against the gold response is solely required to handle parsing issues in instruction following that arise with smaller models such as Llama-3.1-8B-Instruct and Aya-8B, to confirm that models are not being unfairly penalized despite producing the correct answer, but not in the desired boxed format as in the instruction. To ensure that the evaluation protocol was standardized across the board, this was repeated for all experiments, although stronger models (e.g. GPT-4o, Llama-3.1-405B-Instruct) were very adept at instruction following; hence, the exact match scores by parsing from the boxed responses and by human verification were the same for all experiments with those models as the generator. To reiterate, no human annotators were introduced to solve the problems on the Olympiad to verify their correctness, or determine where the model went wrong if the final answer was incorrect, as this would require an extremely experienced expert, as we note in our limitations section. 

2. *Linguistic reasoning problems analyzed in our work*: The machine translation tasks -- more specifically, “Rosetta Stone” puzzles -- that form the primary focus of our work are present in a few benchmarks: PuzzLing Machines, modeLing, and LINGOLY. As we note in Section 5.2, the authors of modeLing suggest that this work was developed in part due to concerns of leakage of the problems in the PuzzLing Machines dataset, which is older (2020) and whose content may have been included in the vast web-scraping performed for the curation of pre-training corpora. This has motivated our selection of the modeLing dataset, which consists entirely of newly written problems by experts to ensure the quality of the problems as well as avoid leakage. By contrast, the problems in LINGOLY are drawn from the UK Linguistics Olympiad (UKLO), which may still be susceptible to leakage; to this effect, the authors introduced a “no context” baseline, which is somewhat akin to our zero-shot baselines. However, the “no context” performance is seemingly non-zero for most mid-size and large / frontier models, as inferred by their exact match and $\\Delta_{NC}$ scores, unlike in modeLing, suggesting that either leakage is present, or the models are extensively familiar with the languages being tested upon. This led us to rely on modeLing as our dataset of focus. Nonetheless, we appreciate the reviewers’ feedback on this matter, and with the spirit of expanding the generalizability of our findings, we are currently working on evaluating our method on the LingOly dataset, which we will include in the camera-ready version.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533825447

thread_type:
official_comment

<thread_object>
creator: area_chair_vb4e

<content>
title:
Action Required: Respond to Author Rebuttals - Nov 27

comment:
Dear ICLR Reviewers,

The author discussion phase is ending soon. Please promptly review and respond to author rebuttals for your assigned papers. Your engagement is critical for the decision-making process.

Deadlines:
- November 26: Last day for reviewers to ask questions to authors.
- November 27: Last day for authors to respond to reviewers.
- November 28 - December 10: Reviewer and area chair discussion phase.

Thank you for your timely attention to this matter.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732791471622

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update: Paper Revision

comment:
Dear Reviewers,

Thank you very much again for your valuable feedback, which has been helpful in improving our submission. Based on Reviewers SDWx and Entm's points raised on the generalizability of our findings, we have further evaluated our two-stage analogical prompting approach on the LINGOLY dataset [1] as well. The results reinforce the efficacy of our method, with remarkable improvements over the baseline and in fact, outperforming the state-of-the-art reported in their paper, to the best of our knowledge. 

We include the results tables with GPT-4o below (comparing against the baseline with the same model in the LINGOLY paper), which are also reflected in both tabular and pictographic forms in Appendix D of our revised paper; these rely on the exact match metric. Note that the "breakthrough" level is the easiest set of problems and "Round 2" is the most difficult, which serves as the UK invitational qualification exam for the IOL. The columns represent the different question types present, which expand beyond the Rosetta Stone puzzle setting we explored with modeLing. Please note that all empty cells correspond to combinations for which problems do not currently exist in the LINGOLY dataset.

Baseline Results for GPT-4o (reported in LINGOLY):

|              | Computational | Text | Monolingual | Match-up | Pattern | Rosetta |
| :------------: | :-------------: | :----: | :-----------: | :--------: | :-------: | :-------: |
| Breakthrough |               | 100% |             |          | 47%     | 79%     |
| Foundation   | 0%            |      |             | 100%     | 67%     | 62%     |
| Intermediate |               |      |             |          | 58%     | 34%     |
| Advanced     |               |      | 0%          | 33%      | 53%     | 26%     |
| Round 2      |               |      | 0%          | 30%      | 27%     | 12%     |


Our Results with GPT-4o, 2-stage analogical prompting:

|              | Computational | Text | Monolingual | Match-up | Pattern | Rosetta |
| :------------: | :-------------: | :----: | :-----------: | :--------: | :-------: | :-------: |
| Breakthrough |               | 100% |             |          | 80%     | 86%     |
| Foundation   | 0%            |      |             | 100%     | 69%     | 80%     |
| Intermediate |               |      |             |          | 83%     | 64%     |
| Advanced     |               |      | 19%         | 50%      | 73%     | 51%     |
| Round 2      |               |      | 14%         | 42%      | 49%     | 41%     |


Deltas (Difference between our result and baseline):

|              | Computational | Text | Monolingual | Match-up | Pattern | Rosetta |
| :------------: | :-------------: | :----: | :-----------: | :--------: | :-------: | :-------: |
| Breakthrough |               | 0%   |             |          | +33%    | +7%     |
| Foundation   | 0%            |      |             | 0%       | +2%     | +18%    |
| Intermediate |               |      |             |          | +25%    | +30%    |
| Advanced     |               |      | +19%        | +17%     | +20%    | +25%    |
| Round 2      |               |      | +14%        | +12%     | +22%    | +29%    |


Encouragingly, we find that our results significantly outperform the baseline by a sizable amount across all difficulty levels, and across all tasks. Moreover, the results outperform the Claude-3 Opus state-of-the-art scores reported in the LINGOLY paper on every single setting, with the exception of the Breakthrough Rosetta Stone (easiest problems). Specifically, we find that our 2-stage analogical prompting method enables GPT-4o to solve questions of the monolingual type which it could not before (0% —> 19% and 14%); furthermore, the correctness rates jump considerably for some of the hardest categories over the baseline (1.81x improvement in Round 2 Pattern, 1.96x in Advanced Rosetta Stone, and 3.42x in Round 2 Rosetta Stone). It is especially worth noting that the Round 2 Rosetta Stone results corroborate with our findings on modeLing as reported in our paper. **These findings suggest that our method generalizes across both datasets and question types.**


### **Paper Revisions**

We have made the following revisions to our paper, to make reference to these results:
* The results have been added to Appendix D, with both tables and bubble plot-style visuals to highlight the performance gains across tasks and difficulty levels. 
* We have added a note in Section 3.3 of the question types included in LINGOLY, which expand beyond the "Rosetta Stone" translation problem style. 

[1] Bean, A. M., Hellsten, S., Mayne, H., Magomere, J., Chi, E. A., Chi, R., Hale, S. A., & Kirk, H. R. (2024). LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages. arXiv preprint arXiv:2406.06196.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates LLMs' linguistic reasoning capabilities through a novel two-stage analogical prompting method applied to low-resource language puzzles. The approach first generates examples from related languages and then uses these to deduce grammar rules in target languages. 

Some reviewers acknowledge the strengths of the work, including comprehensive experimentation across multiple models and prompting strategies, clear presentation, and potential significance for multilingual AI and low-resource language preservation. The results show notable improvements over baselines. However, reviewers raise several concerns: 1) the paper's contribution is primarily empirical, with limited conceptual innovation beyond augmenting prompts with self-generated information; 2) there is a disconnect between the empirical results and the claimed conclusions about grammar rule learning, with insufficient analysis of the specific grammatical phenomena being tested; 3), the evaluation is limited to machine translation tasks, overlooking other important linguistic puzzle formats, and the results on cross-lingual transfer, while promising, do not provide substantial novel insights beyond existing work. While the authors have addressed some concerns through additional explanations and planned evaluations on other benchmarks, the questions about theoretical novelty and broader applicability remain (reviews from Reviewer KXR4 is not included in the consideration since no response from the reviewer). 

Given these limitations, I agree with most reviewers suggesting the work falls marginally below the acceptance threshold of ICLR.