Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?
Authors: Seth Aycock
David Stap
Di Wu
Christof Monz
Khalil Sima'an
Venue: ICLR 2025 Spotlight (2025)
Keywords: llms
translation
low-resource
grammar
long-context
linguistics
Primary area: foundation or frontier models, including LLMs
TL;DR: We show LLMs fail to exploit grammatical explanations for translation, but benefit from typological knowledge for linguistic tasks.

Abstract: Extremely low-resource (XLR) languages lack substantial corpora for training NLP models, motivating the use of all available resources such as dictionaries and grammar books. Machine Translation from One Book (Tanzer et al., 2024) suggests that prompting long-context LLMs with one grammar book enables English–Kalamang translation, an XLR language unseen by LLMs—a noteworthy case of linguistics helping an NLP task. We investigate the source of this translation ability, finding almost all improvements stem from the book’s parallel examples rather than its grammatical explanations. We find similar results for Nepali and Guarani, seen low-resource languages, and we achieve performance comparable to an LLM with a grammar book by simply fine-tuning an encoder-decoder translation model. We then investigate where grammar books help by testing two linguistic tasks, grammaticality judgment and gloss prediction, and we explore what kind of grammatical knowledge helps by introducing a typological feature prompt that achieves leading results on these more relevant tasks. We thus emphasise the importance of task-appropriate data for XLR languages: parallel examples for translation, and grammatical data for linguistic tasks. As we find no evidence that long-context LLMs can make effective use of grammatical explanations for XLR translation, we conclude data collection for multilingual XLR tasks such as translation is best focused on parallel data over linguistic description.

Published as a conference paper at ICLR 2025
CAN LLMS REALLY LEARN TO TRANSLATE A Low- RESOURCE LANGUAGE FROM ONE GRAMMAR BOOK?
Seth Aycock! 1:2
David Stap"
Di Wu²
Christof Monz
Khalil Sima'an'
ILL., University of Amsterdam LLTL, University of Amsterdam
aycock@uva.nl
ABSTRACT
Extremely low-resource (XL.) languages lack substantial corpora for training NLP models, motivating the use of all available resources such as dictionaries and grammar books. Machine Translation from One Book (Tanzer al. 2024) suggests that prompting long-context LLMs with one grammar book enables Kalamann translation, XLR language unseen by LLMs a noteworthy case of linguistics helping an NLP task. We investigate the source of this translation ability, finding almost all improvements stem from the book's parallel examples rather than its grammatical explanations. We find similar results for Nepali and Guarani, seen fow-resource languages, and we achieve performance comparable an LLM with a grammar book by simply fine-tuning an encoder- decoder translation model. We then investigate where grammar books help by testing two linguistic tasks, grammaticality judgment and gloss prediction, and we explore what kind grammatical knowledge helps by introducing a typological feature prompt that achieves leading results these more relevant tasks. We thus emphasise the importance of task-appropriate data for XLR languages: parallel examples for translation, and grammatical data for linguistic tasks. As find evidence that long-context LLMs can make effective use of grammatical explanations for XLR translation, conclude data collection for multilingual XLR tasks such translation hest focused on parallel data over linguistic description.
INTRODUCTION
Most of the world's languages are extremely low-resource (XLR), severely lacking in suitable corpora for NLP tasks (Ranathunga de Silva 2022), such as parallel data for machine translation (MT). However, over 50% of languages have both dictionary and grammar (Nordhoff & Hammarström 201T). While human-rcadabe grammar texts are difficult to incorporate into most NLP models due to their non-standard, unstructured format. Large language models (LLMMs) can handle free-form textual instructions and provide a potential solution to this data mismatch. After pre-training on trillions of tokens (in mainly high-resource languages), LLMs can learn tasks from only few in-context Given this, interest in exploiting grammar texts
examples (Brown et al. 2020 Wei et al., 2022).
in-context for NLP tasks is growing (Ramos al.
2024 Tanzer et al. 2024 Zhang et al. 20246).
Machine Translation from One Book (Tanzer et al.
2024 claims LLMS can learn to translate between Kalamang (ISO 639-3: kgv) newly-doummeneed language unseen in LLM training data and English (eng) via in-context learning with only a grammar book. We note that kgv has over 3,000 parallel sentences, dictionary with over 3,000 definitions (Visser 2020). 500-page grammar book (Visser) 2022) consisting of grammatical explanations and over 1000 parallel glossed examples,
and nearly 100 typological feature specifications (Skirgård et al. 2023a b). This level of resources is comparable to more than thousands of XLR languages have (Joshi al. 2020 OLAC 2024), thus we expect most of these are also minimally represented in LLMs' pretraining data. Given this, finding methods to effectively exploit the available kgv resources could have wide-reaching implications for XLR NLP. this paper, we question the claimed utility of grammatical explanations for XLR MT with LLMs, then ask where and what kind of grammatical knowledge helps. We show that:

---

Published as a conference paper at ICLR 2025
Parallel examples are essential for translation We disentangle grammar books' parallel examples from grammatical explanations, finding explanations add no significant advantage over parallel data: adding +0.7 CHRF++ into kgv, and into eng scores fall -0.3 points adding explanations to parallel sentences; and quality drops up to 8 points with parallel data removed. Our findings generalise to Nepali (npi) and Guarani (gug), where the book's s parallel sentences outperform the full book by up to 4 CHRF++. LLMs fail to effectively exploit grammatical explanations for translation.
Fine-tuning matches long-context LLMs We fine-tune small translation models on the parallel data, achieving competitive results within 0.2 CHRF++ of the performance of Gemin ni with grammar book into kgv, and beating Ll amm--3.... 1-8B settings with access to the same data by up to 20 points. Parallel examples (especially with glosses) are both more token-efficient and readily available than grammar books, and enable computationally cheaper methods than long-context LLMs.
Typological prompting outperforms explanations and helps linguistic tasks We introduce a novel typological feature prompt, and for kgv and npi translation we find our method is more effective than explanations into eng, but not into XLR languages. On kov grammaticality judgment, our typological prompt improves up to 3% over the book's 1 1.2k parallel sentences and 8% over the whole book. For gloss prediction, parallel sentences again beat the book by up to 5.3% morpheme accuracy, and adding typology achieves leading performance on this task. Therefore LLMs can exploit grammar for relevant linguistic tasks-if provided in a useful form- not for translation.
Task apsr-pppreeritte data is therefore essential. In the current paradigm, we recommend that data collection for XLR MT is thus better focused on parallel data over linguistic description, given the advantages in token efficiency, computational cost, and availability.
2 RELATED WORK
Grammar for low-resource machine translation Translation of low and extremely-low resource languages (here meaning <100k and 10k parallel examples respectively) with L.MMs currently of significant interest (Cahyawijaya et al. [2224; Court & Elsner 2024; [yer et al. 2024). Methods
include fine-tuning (Xu et 2024), dictionary prompting (Ghazvininejad et al. 2023), and retrieval- augmented few-shot prompting (Merx al. 2024). Alongside advances in long-context LLMs,
recent work has introduced grammar information in context for various tasks: Guo et al. (2024) test
textbook-style prompt with LLM-generaal parses, seeing limited gains against parallel sentences; and Zhhng et al. (2024a) add a singular syntactic rule to their prompt with small effects. Zhang al. (2024)) chain morphological analysis, dictionary and an LLM--ummarised grammar book in-context, observing small gains from book passages over dietionary- only setup. Others meanwhile or as a hybrid rule-based use grammars with LLMs for data augmentation (Lucas et al. 2024)
translation system (Coleman et al. 2024).
Machine Translation One Book (MTOB) (Tazerr et al. 2024) introduces a translation test set for the newly documented XLR language Kalamang (thus unseen by LLMs), plus grammar book and additional parallel sentences. MTOB suggests long-context LLMs can exploit linguistic knowledge (a grammar book) for XLR translation, a potential step forward in leveraging underused resources for XLR languages. However, several issues mean MTOB leaves open questions over L.Ms' ability to exploit linguistic information for XLR tasks. The test sets of 50 short, easy examples are potentially too small for making wider generalisations, and the human bascline is somewhat flawed as they may learn from examples at test time; relatedly, Gemini Team al. (2024) ask the non-fluent human baseline to rate model outputs and their own kgv predictions, potentially biasing the evaluation. Furthermore, despite CURF++ being the de-facto standard in XLR translation (Maillard et al., 2023 Costa-jussà et al. 2024; Edman et al. 2024). .TOB uses CURF which unlike CURRF++ does not factor in word order. MTOB's s results would benefit from further ablations, since the signal from parallel sentences and explanations is not disentangled, nor is strong translation approach tested. Finally, we note that the kgv grammar book not designed for language learning, but for describing theoretical linguistic phenomena-wwit In this paper, we tackle these issues by combining the test sets, using automatic CHRF++ scores, disentangling the parallel/non-paralle signal, and testing two tasks better aligned for grammar books.
MTOB's 's authors note limits L.MMs to a basic competence.
Linguisties in NLP Incorporating linguistic information into NLP models is a long-standing goal with mixed results (Lakoff 1978. Raskin 1985 Uszkoreit 2009 Opitz et al. 2024). Past work sees
2

---

Published as a conference paper at ICLR 2025
gains from adding syntactic knowledge into translation models using constituency parses (Curryy
& Heafield 2019), grammar supertags Nädejde et al. 2017). or tree-structured models (Sartran
et al. [2022). One useful form of linguistic information typology, available for many languages in
standardised feature databases (Dryer & Haspelmath 2013 Skirgard et al. 2023a b); features describe
languages in terms of phenomena such as word order rules, verb tenses, and noun cases. Trained linguists condense fine-grained textual descriptions from grammar books into discrete, categorical, and cross-linguistically consistent feature specifications. Typological features have been incorporated into NLP models with some success in the form of embeddings (Malaviya et al., 2017] Östling & Tiedemann 2017) but several studies find minimal positive effects on performance PPonti et 2019) Ustún et al. 2022). To test whether LLMs follow this trend, we construct novel prompt that uses readily available typological feature specifications for source and target languages, as an in-context
and language-invariant method for bridging cross-lingual grammatical differences.
Interlinear gloss prediction Language documentation involves describing the underlying grammar of language given its surface forms (Ginn et al. 2023). A standardised data format for this analysis is Interlinear Glossed Text (IGT), comprising a morphologically segmented transcription (where morphemes are the smallest units of meaning), aligned interlinear gloss with subword- lexical and grammatical information, and a sentence-level translation (Comric et al. 2015 Mortensen et al., 2023); Kalamang example is shown in Example T [Visser 2022h. We note that glossing is designed for trained linguists rather than language learners. Glosses have been widely applied in NL.P tasks,
including as a pivot for translation (Zhou et al., (020)), dependency parsing (Georgi et al. 2012).
grammar generation (Bender et al. 2014), morphological analysis (Moeller et 20,0, | Shhnniiyya & Palmer 2023), and linguistic resource development (Beermann al.) [2000] Buchholz et al.) (2024).
Predicting IGT is therefore a well-motivated grammatical task, and segmented IGT is a valuable linguistic resource. IGT prediction is most relevant for XLR languages where it is impactful in
assisting annotators for documentation and preservation (Ginn et al., 2023). Prior methods include supervised neural models (Zhao et al. [2020] Girrbach] 2023). or adapting multilingual language
models (He et al. [023] Ginn et al. 2024a b). Since IGT is costly to generate, past work has
scraped it from books (Nordhoff 2020; Nordhoff & Krämer 2022); we follow this method to extract
Kalamang glosses from the grammar book. One of our contributions involves testing gloss prediction to determine whether LLMs can use grammatical knowledge for more relevant tasks.
(1) bal se sor=at na dog IAM fish=OBJ consume 3sG IAM consume=PLNL finish 'The dog ate the fish, after he ate.'
se nan=i
koyet
Transcription Interlinear Gloss Translation
METHODOLOGY
3.1 GRAMMAR BOOKS FOR TRANSLATION
Our methods are guided by open questions over the use of grammar books for XLR translation. First, we manually filter the grammar books into parallel examples and word pairs, and explanatory. descriptive text, to disentangle the signal from translations and grammatical explanations (see Appendix/AAfoo for kgv book extract). This novel ablation is necessary to understand which specific aspects of grammar books are useful for XLR MT. We ask whether L.LMs really learn effectively from the grammar explanations, or if most translation supervision stems only from the book's 's parallel examples. We combine the directional test sets into single 100 example test set to improve the generalisability of these results, and evaluate with CHRF++ (Popović) 2017) to take word order into accoun [ evaluation set, FLORES (Costa-jussa) et al. 2024) and likely a low data weight in LLMs; while not unseen, these experiments broaden our results seen low-resource languages more generally.
We also test eng-npi and qug translation, low-resource languages with established
3.2 NEURAL MACHINE TRANSLATION APPROACHES
To compare the LLM- approach with a standard MT approach for learning to translate a language as yet unseen by the model, run experiments fine-tuning NLLB-1.3B (Costa-jussà et al. 2024)
We omit human evaluation (cf. Gemini Team al. 2024 given the infeasibility of engaging proficient
Kalamang speakers. See Appendix for a small-scale qual qualitative analysis of several kgv-eng examples.
3

---

Published as a conference paper at ICLR 2025
on the parallel data sourced from the grammar book. We expect similar results to be achieved with the same resources using a small, specialist encoder-decoder model, which would confirm that the useful translation signal stems from the parallel sentences contained within grammar books-whitd constitute less than 20% of the kgv grammar book's 's total tokens (see Table for token counts).
3.3 TYPOLOGICAL FEATURE PROMPTING
In asking what kind of grammatical knowledge can aid LLMs XLR tasks, we introduce text- based method for incorporating typological information into prompts, differing from previous work on continuous typological embeddings (Oncevay et al. 2020). We extract categorical typological feature specifications from Grambank Skirgärd et al. (203b) for kgv, npi, gug, and eng, and use rule-based template to construct a prompt containing features for each language and a short explanation. For an example of the prompt format, see Appendix [ Most languages with grammar books have some typological feature specification, since features are distilled by annotators from external resources. Our method isolates high-level grammatical tendencies of a language from the specific instantiations of those features (i.e. parallel examples). We hypothesise that our method, when combined with the grammar book's parallel sentences, will least match the performance of the grammar book. We expect that providing explicit features such as word order rules removes some reasoning requirements for the LLM. Conversely, typological features will not have relevant parallel examples, some reasoning and retrieval is still required, potentially tempering the advantages.
3.4 GRAMMATICALIT JUDGMENT
To test the LLM's ability to acquire knowledge and understanding of Kalamang grammar from the book, we introduce discriminative grammar judgment experiment. We ask the model to choose the original Kalamang test sentence against a modified example, with three successively easier settings: swapping two adjacent words (SWAPas)), two random words (SWAPran), and shuffling all words (SHUFFLE). We acknowledge that while we cannot guarantee all corruptions are ungrammatical (since no author speaks Kalamang), we assume the uncorrupted examples are linguistically unmarked sentences. For all settings we expect O-SHOT model to achieve approximately 50% accuracy, while for high-resource languages we would expect near 100% accuracy. We expect the grammar book to have greater positive impact in this setting where grammatical knowledge is explicitly rewarded.
3.5 INTERLINEAR GLOSSED TEXT PREDICTION
To explore another more relevant task for exploiting grammar explanations, we test IGT prediction with the grammar book against few-shot and supervised baselines. This experiment tests whether LLMS can learn grammar from book to the extent that we see a difference in performance on grammar-focused task. IGT requires both lexical translation and grammar analysis, without any generation in the language at hand. This makes IGT prediction more appropriate task to perform from a descriptive, non-didactic grammar text. We argue that IGT prediction accelerates XLR documentation more than translation, and is likely have more direct impact for both first language (LI) speakers and linguists, not to mention the potential downstream uses, e.g. POS tagging and MT. IGT prediction is also well defined task with strong baselines from shared task (Ginn al. 2023) and clear evaluation metrics, primarily morpheme accuracy MMMiilln--Majrr [020] /hhao et al. 2020); our experiments build this prior work. Finally, we argue grammar books intuitively suited to IGT prediction more than translation, because their unique contribution is glossed text, rather than just parallel sentences. We use all available sentences with IGT from Dictionaria as our test set, and for our supervised baselines, we process the grammar book IGT examples into a training and development set. We expect the grammar book to provide marginal gains over raw parallel sentences because the grammar book explicitly explains the glossed examples therein.
4 EXPERIMENTAL SETUP
4.1 DATA
We use the preprocessed Kalamang (xgv) grammar book (Visser 2022 Tanzer et al. 2024). with additional processing of irregularities (particularly for glossing) introduced in La conversion.
4

---

Published as a conference paper at ICLR 2025
We similarly preprocess grammar text in Nepali (npi) (Bal 2004) and Paraguayan Guarani (gug) (Estigarribia (020). We prompt with the entire grammar, B in-context (where the subscript indicates the data subset). Following Nordhoff & Krämer (2022), extract parallel glossed examples and bilingual word/phrase pairs from the book based on text formatting into a parallel subset, BOOK para (p)- The remainder of the book contains grammatical explanations without parallel examples, labelled BOOK para (-p)). Subset statistics are shown in Table We preprocess kgv-eng parallel examples from the grammar book into an unsegmented parailel data format, giving PARAbook (used for 5*-SHOT examples and in full as prompt) and PARA
TGT which includes glosses (1239 examples) for excerpts of prompt types, see Appendix E We additionally test prompts with PARArai (400 examples) and WORDLIST (W) (3813 examples). Additionally. we sample 500 examples from Dictionaria (Visser 2020 as the development set for fine-tuning, In total there are 3.3k eng=kgv parallel example<" we focus on the 1.2k in PAR Abook for fair comparison with BOOK settings. For testing, we use our combined 100 example test set for kgv, and FLORES devtest for npi and gug (1012 examples) (Guzmán et al., 2019), with few-shot examples from FLORES dev. For IGT prediction, we preprocess 1221 examples from the grammar book with glosses for training (5623 words) and development (612 words) sets (split 90: 10% by sentences). Following Ginn et al. (2023), introduce test set of 97 glossed examples (447 words) from different source, Dictionaria, which were manually inspected for correct alignment.
Table 1: Dataset statistics for grammar book subsets, in lines and space-separated tokens.
Language
Split
Lines
Tokens
B00Kpam
4489
17858
kgv
B00Knon-jon
2282
81268
B00Kpum
759
5333
npi
B0OK
2896
23233
B0OKpam
5718
49122
gug
B00Kaen-pan
3295
57338
4.2 MODELS
In our experiments we use the API-only al. 2024). We justify this choice due Gemini's 's context window of 1M tokens, significantly larger than other models, which can handle the entire grammar book, and use Flash over Pro due to prohibitive cost differences. We also use the smaller, peighweight Llama-3.1-88 base and instruction -uundd models (Dubey et al., (022), with a context of 128k tokens. This is insufficient for kgv and gug BOOK,ll but fits BOOK para and BOOK para: plus the np BOOKan- We test Insma-Instruct (I.1 ama 1amam)),....... T), and fine-tune Ilama base with LoRA (Hu et al.] (021] on PARAnook (Llama- ft) with prompt masking for epochs with a constant learning rate of Ic-4, batch size 4. and LoRA a 16, r 16, targeting all linear projections. For NMT baseline, we fine-tune I.LB--.33-DDisilll (NLLB) (Costa-jussá et al. 2024) on PARA For kgv grammaticality judgment and IGT prediction,
(henceforth Gemini) (Gemini Team
the same Gemini model as above.
4.3 EVALUATION
We evaluate translation automatically with CHRF++ (Popović 2017). We favour CHRF++ over CHRF, used in Tanzer et al.] (2024), since it takes into account word order as well as character n-gram overlap. We report scores for trimmed responses after the first newline character distinguish translation quality from overgeneration and chat explanations (Aycock & Bawden 2024) and use forceful prompt (detailed in Appendix
ensure the translation is produced on the first line.
httpps: : /diitiit i ionaiiina.........oo aaarria.cco .a..cll 1d.. wew..door//ccttc but ioutiions... oosi/nn/a amanq
Data (including grammar book splits) and code are made available this link
5

---

Published as a conference paper at ICLR 2025
Table 2: Translation results for eng=kgv with Gemini, Alama-Instruce (L-I) and fine-tuned (L-ft),, and prompt tokens counted with NLTK's tokenizer (Bird et al. 2009). Highest B00K scores are underlined, highest overall are bolded. Grey rows indicate settings with data other than the book's 8 parallel data; indicates tests ruled out by context length. #W4W tests are not run with Gemin i but are included for comparison. The subset of the book's parallel sentences almost matches or outperforms the whole grammar book, while its grammatical explanations perform poorly.
CHRF++
Tokens
Setting,
eng kgv
kgv-eng
Model
Gemini
1-I
L-ft
Gemini
L-I
I-ft
BASELINES
0-SHOT
11.0
2.7
18.5
12.7
12.5
23.0
W4W
18.9*
18.2*
PARALLEL DATA
WORDLIST (W)
29.1
13.6
19.5
27.9
20.8
26.8
9.0k
5*-SHOT PAR Abook
38.9
15.0
24.6
33.4
21.1
23.0
0.8k
PARAbook
26.6
7.3
13.0
33.1
22.9
26.9
15.6k
W
34.7
6.8
14.4
34.7
27.5
30.5
24.6k
PARAmmi
40.7
13.8
17.9
46.6
31.3
37.6
29.4k
PARABoot
33.7
20.3
28.8
32.8
24.7
33.1
22.7k
GRAMMAR BOOK SUBSETS
BOOK₁
34.4
34.4
99.6k
38.3
39.6
PARAmiin
43.7
46.1
113 4k
BO0K (p)
30.8
9.7
19.0
34.7
22.1
28,8
18.3k
BOOKke-pmn para (.p)
22.6
3.3
10.0
27.5
14.3
16.7
81.3k
TYPOLOGY
TYP O-SHOT
10.8
3.4
13.6
13.9
14.3
17.6
68.4k
BOOKKμr
31.4
35.2
86.7k
PARAbbok
32.9
33.0
84.0k
PARA PARApockttiin
40.6
44.9
100.6k
4.4 BASELINES
For translation experiments, we several basclines: 0-SHOT translation with a standard translation prompt; word-for-word translation with fuzzy dictionary lookup (W4W);: 5 retrieved examples per word (5*-SHOT) based longest common subsequences following Tanzer et al. (2024); prompting with the full WORDLIST (W), parallel examples, PARAsooke parallel examples with glosses, PARA and processed training set examples, PARAAuin- For IGT prediction, we use baseline frequency-basee based classifier (TOP-CLASS), fine-tuned RoBERTa token classifier (Ginn et al. (023) (SMP-BASE); hard-attention glossing model (TOCL-MORPIL (Girrbach) 2023); and BYT5-FT and GLOSSLM-FT models (Gin et al.) 2024b) fine-tuned
IGT
kgv IGT training and development sets. We provide segmented input, and English translations to models which accept them.
4.5 EXPERIMENTS
Our central research question investigates the contributions of grammatical explanations and parallel data to translation performance. We therefore prompt models with BOOKa and its filtered subsets. We test our typological feature prompt, TYP, to replace BOOK For npi and gug, we repeat the book settings above. We fine-tune translation models with the PARAbook parallel data for comparison with B0OKgam settings. For grammaticality judgment and IGT prediction tasks, we similarly test Gemi ní with the kgv Book and TYP prompts.

---

Published as a conference paper at ICLR 2025
Table 3: Translation results for eng-ippi and eng=iug with Gemini and Llama-I I. Best BOOK pu (white rows) scores are underlined, best overall are bolded; - indicates tests ruled out by context length. While a and monKnan-pura decrease performance from O-SHOT, Bo0Kpan has neutral or positive effect into and from npi respectively, with similar trend seen for gug.
CHRF++
Setting,
eng-npi
npi-eng
eng-gug
gug-eng
Model_,
Gemini
L-I
Gemini
L-I
Gemini
L-I
Gemini
0-SHOT
42.5
28.6
65.2
51.1
26.6
23.6
5"-SHOT
43.2
37.6
64.9
57.3
29.2
13.7
43.1
23.4
BOOKи
42.6
24.3
64.4
48.9
22.2
38.7
B00Kpm (p)
42.5
28.6
64.9
52.6
25.8
6.7
41.8
11.8
no0ka (-p)
41.8
24.5
64.5
48.4
19.3
5.6
34.5
10.1
TYP 0-SHOT
42.4
23.2
64.6
49.5
21.1
4.3
33.9
23.4
TYP + BOOKppra
41.8
22.0
64.9
49.1
21.9
34.5
RESULTS & ANALYSIS
Grammar versus parallel sentences for translation We disentangle the signal from grammar books' explanations and parallel sentences for translation. Our kgv results in Table show that most or all performance improvements stem from the book's parallel sentences, with quality plummeting when parallel data is removed. With Gemini into eng, BOOK, marginally outperforms BOOKatl and beats BOOK "p by CHRF++, while into kgv, BOOK, outperforms BOOk by over points, and BOOKa performs 3 points better than BoOK2- However, we show statistically in Section [.] that this small improvement is modelled directly by an increase in test set vocabulary coverage, rather than from the grammatical explanations. Additionally, this gap closes with the PARA book prompt, which preprocesses and structures the parallel data in BOOK, into kgv-gloss-ens triples. PARAbooK performs particularly well for Llama-1 I. with over 10 points improvement over BOOK, into kgv. Due to context restricting kgv BOOKall tests, conclusions with Llama-2 I are limited, but we find again that BOOk performance lags far behind BOOKg- We note that baselines including O-SHOT show kgv translation is non-trivial. We also find that additional parallel data further improves translation quality, and note -*-SIITT is generally competitive despite its short average prompt, achieving the best BOOK, score into kgv with Gemini. Thus for kgv translation, both LLMs on test mainly learn from the book's 's parallel sentences, failing to exploit the grammatical explanations.
KGr
We observe similarly strong trend for npi and gug, low-resource languages with high-quality FLORES test sets, in Table BOOK, settings largely match or outperform BOOK for both models and languages (except L. ama-I in gug-eng where the model often fails output translations the first line for BOOK settings). Few settings beat O-SHOT and differences between Gemin n settings (especially npi) smaller than for kgv; perhaps the model's prior competence (and shorter npi grammar book) mean there is less to be gained. However, analysing BOOK settings in isolation shows that both ВоOк and BOOK-p have detrimental effect of up to 7 points below 0-SHOT, while BooK, has neutral or small positive impact in both npi and gug. Finally 5*-SHOT is again effective, especially for Lla Llama-I into npi and gug, likely due the greater vocabulary coverage of the example set. These results generalise our findings for kgv to seen low-resource languages: we find no evidence that LLMS can effectively exploit grammatical explanations for translation.
Fine-tuning versus in-context learning We test standard MT approach for adding a new language by fine-tuning NLLB, small MT model, on the book's 's parallel data, shown in Table NLLB achieves competitive improved performance compared prompting Gemin' n with the same preprocessed parallel data, PARAnook- We also test backtranslation (BT). standard method to boost performance in MT (Sennrich et al.) 2016). single BT iteration with PARAnain has a negative impact into kgv, likely due to the poor quality of the initial model introducing excessive noise. However we see boost CHRF++ into eng, we expect because of the strong English language modelling of NLLB. Further, adding small 400 example parallel training sees large gains of 4-8 points. These results suggest the MTOB benchmark can be adequately addressed as a standard XLR MT problem with simple data preprocessing, small pre-trained model, and fine-tuning on single GPU for 1 hour.
7

---

Published as a conference paper at ICLR 2025
Table Translation results for eng-igg with NLLB, an MT model, fine-tuned on PARAbook data; equivalent in in-context learning results with Gemini are shown for comparison. Fine-tuned NLLB achieves competitive results with an LLM given the same parallel data, especially into kgv.
CHRF++
Setting,
eng-kgh sag-kgv
kgv-eng
Model_,
Gemini
NLLB
Gemini
NLLB
PARAbook
26.6
34.2
33.1
28.6
PARAnii
33.4
38.7
38.5
36.9
BT PARAmin
32.0
31.6
also fine-tune Llama base on PARAhook to give Ll ammamm-ff ft, with results Table 2 We find all Llama-ft settings beat equivalent L1 allam-Il tests with BoOKall data, except for PARA settings with glosses which marginally outperform L1 L-ama-ft O-SHOT results. Prompting Llama-ft with parallel data in-context further improves performance over 0-SHOT by up to 10 points. We additionally fine-tune Gemin ní. on Apook with results in Appendix [ finding niminii-ft underperforms NLLB and Gemini with the same data in-context by 6-T2 CHRF++; expect this is because it is already extensively instruction-tunee Thus taattuunnn--pattuaaartt of small MT models-is cheap method for achieving competitive results with prompting instruction-tuner long-context LLMs, given the same parallel data.
1GT
Typological prompting for linguistic tasks Given the limited contribution of grammatical expla- nations to translation performance, we introduce a novel prompting method summarising languages' typological features. This prompt is intended to replace BOOK-p. thus on results when combined with BOOK, data. Our results for eng-kgv translation in Table 2 show expectedly poor O-SHOT performance due the lack of any Kalamang text. Into kgv, our prompt beats BOOK, but not BOOKal: however into eng, our prompt with BOOK, achieves the best transla- tion results for settings with book parallel data. For npi Table 3 TYP BoOK, is less effective than BOOKall into npi, and marginally outperforms it into eng up to 0.5 CHRF++, though BOOK, alone performs best; similarly in gug tests, BOOK, outperforms TYP BOOKp, which beats or matches Book-p -p- The performance of typological prompting for translation is therefore inconsistent, supporting the above finding that LLMs fail to effectively exploit grammatical information for MT.
primarily focused
SwaPadi
2012
65%
O shot
Test Setting
10*-shot
Bookui
Swapran
75%
Bookp
Book
Typ + Hook,
Shuffle
80%
30
40
50
60
70
80
90
100
Accuracy (%)
Figure 1: Grammaticality judgment accuracy in kgv; for reference in eng tests, Gemini scores 100%, 99%, and 100% respectively. Our prompt TYP + BoOK, performs best overall suggesting grammar can help LLMs for linguistic tasks.
To determine whether grammar is not useful for MT or LLMs cannot exploit grammatical explanations more broadly, we test two more relevant tasks: grammaticality judgment and IGT prediction. In Figure grammaticality judgment results in kgv with Gemini show all settings perform similarly poorly on SWAP though improving on O-SHOT by around 7%. Generally, 10*.SHOT worse than prompts with Book,, likely because diverse sentences may help here more than overlapping
8

---

Published as a conference paper at ICLR 2025
Table 5: IGT prediction results in kgv for supervised baselines and Gemint settings. Our TYP +
B0o prompt achieves the highest morpheme accuracy and high scores on other metrics, while BOOKall performs poorly overall.
Model
Morph Acc.
Word Acc.
Stem F1
Gram
CHRF++
TOP-CLASS
Ginn
2023
44.0
39.7
40.6
57.8
34.5
SMP-BASE
Ginn & almer
2023
45.2
41.7
39.7
58.9
34.3
TUCL-MORPH
Gurbach
2023
43.6
38.8
40.0
50.7
35,4
BYT5-FT
Xue al
2022
40.8
48.6
40.9
45,4
49.0
GL
et
2024b
43.8
47.7
41.5
50.4
49.1
-0OT
43.9
43.7
44.3
45.2
46.4
BOOKall
40.1
31.5
38.7
43.4
40.5
BoOk pa (p)
45.4
42.1
44.0
49.0
45.0
Bo0Kaon-pra (-p)
21.0
8.8
23.9
15.6
26.0
TYP + B00K
46.1
40.9
44.2
50.5
44.8
vocabulary, which helps more for MT. For BoOk settings we observe that BOOK, matches or outperforms BOOKall across all three tests by up to 5%, and consistently beats BoOK-p. by up to 18% in SHUFFLE tests. So far, the LLM still fails to exploit grammatical explanations effectively and learns mainly from parallel examples. However, our TYP BoOK, setting performs best over the three tests by up to 3% over BoOK,- These positive results suggest that LLMs can learn from grammar, given the right kind of grammatical knowledge and a relevant task.
For kgv IGT prediction, we compare Gemin n settings with supervised baselines in Table The leading performer in morpheme accuracy, the key IGT metric, again our typological prompt TYP + BOOK,, scoring 6% above BOOKan. 0.5% over BOOK,, and 25% over BOOK-p. Additionally, our prompt beats all supervised systems by 1-5%, suggesting in-context learning with typological knowledge and parallel glossed examples is strong method for XLR IGT prediction. Results for other metrics show slightly differing trends, with supervised models showing stronger word accuracies and Gram Fl scores (since most are closed-set classifiers). Generally though, Book-p shows extremely poor performance, while BOOK,, -HHSH and TYP + BOOK, settings perform consistently well, often beating supervised baselines. We note that TYP BOOK, scores show competent performance for both grammatical (on morpheme accuracy and Gram F1) and lexical aspects (via Stem F1) of IGT prediction, suggesting all-round competence on this task. These results reinforce our findings that while parallel sentences still provide most of the useful signal, LLMs can exploit grammatical specifically typological information for linguistic tasks.
5.1 ANALYSIS
Type coverage and Token efficiency We investigate whether any added performance from gram- matical explanations statistically significant or can instead be attributed to greater test set type coverage in the prompt. We distinguish between types, meaning unique words in vocabulary, and tokens, Lec individual occurrences of types a text. We univariate least squares regression models to CHRF++ scores with test type coverage the independent variable, for both directions, shown in Figure 2 All settings fall within the 95% confidence interval of the regression lines, and the models significant in both directions (p 0.005, F-test the Pearson correlations are also significant (p < 0.005). Thus maximising target vocabulary coverage (via parallel sentences) in-context is the most efficient method for improving LLM-based XLR translation. These linear regressions show that translation performance can be directly modelled by test set vocabulary coverage, and that the book's 's grammar explanations provide no significant advantage over its parallel sentences. See Appendix for full statistics on our prompts' test set type coverage.
We then explore whether the improved translation scores can be attributed to a longer (or shorter) prompt, by testing for relationship between prompts' total tokens and translation quality in terms of CURR++ for (OOK(all/p/->) with Gemini. The resulting linear models are not significant in either direction (p 0.997. p 0.78 into and from kgv, F-test), with no significant Pearson correlations.
FFrr details of these and following statistical tests, Appendix B

---

Published as a conference paper at ICLR 2025
eng kgv
kgv eng
50
y=0.37x + 14.38
y=0.31x + 13.46
Pows
Confidance Interval
Booly W
Confideace
40
Piess
Pos 1 Per.
Tp- W.+
40
Typ- Mi--W-PP.
Booke +.
Boskk
Buska
Typt Book,
Typ
Ploaa Boskk
Chri44
Plac
ChrF4
30
Typ Dorky
Sont,
30-
Book..
Paws
20
15
S-chet
10 0-datt
80
100
20
40
60
80
100
Test: Type Coverage (%)
Test Type Coverage (%)
Figure 2: Regression models of CHRF++ score against test type coverage for eng-kgv and kgv- eng translation with Gemi ní. Prompt settings are labelled with abbreviations for clarity. The plots show that translation performance can be statistically modelled by test set vocabulary coverage.
The grammar book is therefore both a token-inefficiee way to learn (with similar performance despite nearly 5x more tokens than kgv BOOK,), and cost-inefficient dataset to generate, compared to using its parallel sentences. The needle-in-a-haystee increasing context, retrieval of relevant information (i.e. similar parallel examples) becomes harder (Hsich et al. 2024), so while BOOk, is a subset of BOOKgll there a greater ratio of relevant to irrelevant information in the prompt-assunn exploited for translation.
problem could partially explain this: with
grammatical explanations cannot be effectively
Discussion We note that our results do not indicate LLMs cannot understand books in general; rather, we find no quantitative evidence that the results here and in MTOB show L.MS can effectively exploit grammar books (or linguistic knowledge) for translation. Indeed, we show that LLMs can exploit grammatical information in the form of typology for more relevant, anguistticallyfocusss tasks. More broadly, from an educational perspective, translation a problem-solvine task aiming to reach goal state (translation) via series of actions given an initial state (source) and optionally rules on applying actions. Humans tend to learn this kind of task more efficiently via worked-examples (van) Gog et al. 2019), i.e. with explicit explanations, rather than pure discovery learning, meaning without explicit guidance (Mayer (004). Our results however indicate that for translation, LLMs learn more effectively from unannotated parallel examples (i.e. discovery) than from grammar principles with explained examples (i.c. cauplee-bssedl Our results thus tentatively support a divergence between learning strategies for translation between human learners and L.LMs learning in-context. We suggest that this may partially stem from prompts with parallel data aligning more closely with LLMs' instruction-tune tunnnn data than grammar book explanations.
CONCLUSION
We find no evidence that LLMs can effectively exploit grammatical explanations for low and extremely low-resource MT in Kalamang, Nepali, and Guarani, instead finding that LLMs rely on the parallel sentences within the book. This runs counter the claim of prior work including MTOB which use grammar books to enable LLMs' performance on XLR tasks. We show that fine-tuning small MT models matches the performance of costly long-context LLMs. Further, we show statistically that grammatical explanations add no significant advantage above the increased type coverage they provide, and that grammar books are less token-efficient for prompting than parallel sentences. However, LLMs can exploit grammatical information, given appropriate task-e.g, e.g. grammaticality judgment or IGT prediction-ant more useful grammatical data the form of our typological prompt, which achieves leading results on these linguistic tasks. We therefore emphasise the importance of task- approprrtee data: parallel data for MT, and grammatical, preferably typological, knowledge for linguistic tasks. Moreover, suggest data collection efforts for multilingual XLR tasks, at least for MT, are better focused on parallel data over linguistic description, which enables less costly, more token-efficient translation.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We emphasise that this work does not aim to address social problems, and instead investigates the empirical utility of grammar books resources for XLR NLP. We operate the assumption of continued consent of the Kalamang community to use their language in our research, as discussed in Tanzer et al. (2024). In Sections 2 and 3.5 we discuss the utility of our work for both linguists and LI speakers, specifically relating to the IGT prediction task in its capacity to improve language documentation processes,
ACKNOWLEDGEMENTS
This work was funded in part by the UvA's Language Sciences for Social Good project, the City of Amsterdam, and the Netherlands Organization for Scientific Research (NWO) under project numbers VI.C.192.088 and 2023.017. The authors would like thank members of the Language Technology Lab for many constructive discussions, particularly Vlad Niculae for detailed feedback on our paper. The authors are grateful for the helpful feedback provided by the anonymous reviewers.
REFERENCES
Seth Aycock and Rachel Bawden. Topic-guided Example Selection for Domain Adaptation in LLM- based Machine Translation. In Neele Falk, Sara Papi, and Mike Zhang (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 195-195, St. Julian's s, Malta, March 2024. Association for Computational Linguisties. URL https: /////aaaanttt canthology..o ogy wrtp/org//2000 wacleaallls. ssr.
Bal Krishna Bal. Structure of Nepali Grammar. PAN Localization, Working Papers 2004-2007, pp. 332-396, 2004.
Dorothee Beermann, Lars Hellan, Pavel Mihaylov, and Anna Struck. Developing a Twi (Asante) Dictionary from Akan Interlinear Glossed Texts. In Dorothee Beermann, Laurent Besacier, Sakriani Sakti, and Claudia Soria (eds.), Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under- Resourced Languages (CCURL), pp. 294-297, Marseille, France, May 2020. European Language Resources association. ISBN 979-10-955488 (11---9555661111111 URL https:
3020.s1tu-1.41
Emily M. Bender, Joshua Crowgey, Michael Wayne Goodman, and Fei Xia. Learning Grammar Specifications from IGT: A Case Study of Chintang. In Jeff Good, Julia Hirschberg, and Owen Rambow (eds.), Proceedings of the 2014 Workshop the Use of Computational Methods in the Study of Endangered Languages, pp. 43-53, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/W14-20
URL httth: wwwaacaaanhhloccoooooooo logy
org/ 205//14-220 2206
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc., 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse et al. Language Models are Fhw-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpthtttttsppsssssssssee http://///pppppsssssssssss//r//p haash/
Matthew J. Buchholz, Julia Bonn, Claire Benet Post, Andrew Cowell, and Alexis Palmer. Boot- strapping UMR Annotations for Arapaho from Language Documentation Resources. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference Computational Linguistics, Language Resources and Evaluation COLING 2024). pp. -4547 Torino, Italia, May 2024. ELRA and ICCL. | httpt:ttt////.... / antho oow.oooo..ogogooooooooo 2024 rec ma n. 220 w
11

---

Published as a conference paper at ICLR 2025
Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. LLMs Are Few-Shot In-Context Low- Resource Language Learnrrs In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 405-433, Mexico City, Mexico, June 2024, Association for Computational Linguistics. doi: 10. 11/53////2222..aalllaeaee 24. /tt/tttps///allatthllles org/2022 naacl- maaaccl-oong.... 24
Jared Coleman, Bhaskar Krishnamachari, Khalil Iskarous, and Ruben Rosales. LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages, May 2024. URL http:
1/arxiv.ore http://w..g/aab//20...0. 08997
ar arXiv:2405.08997 [cs].
Beraard Comrie, Martin Haspelmath, and Balthasar Bickel. The Leipzig Glossing Rules: Conventions for interlinear hooopheme--b-morrhhee glosses. 2015. Uttpttts::/ www eva. mpg. de/
pdf]
Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett et al. Scaling neural machine translation to 200 languages. Nature, pp. 1-6, June 2024. ISSN 1476-4687. doi: 10.
URL https
7/ 7/www. , nawure..ture...... com ar ieiticll.... s 586 023-07335---
Sara Court and Micha Elsner. Shortcomings of LLMs for Low-Resouree Translation: Retrieval and Understanding are Both the Problem, June 2024. URL
15625 arXiv:2406. 15625 [cs].
Anna Currey and Kenneth Heafield. Incorporating Source Syntax into Transformer-Base Neural Machine Translation. In Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Fourth Conference on Machine Translation (Volume I: Re- search Papers), pp. 24-33, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W1955550
URL wttttssss/ccaaneeetteeee
1W19-52033
Matthew S. Dryer and Martin Haspelmath. WALS online (v2020.3), 2013. URL https://doi
org/10. 10. 5281 1/ zenodo. 7385533
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez et al. The Llama 3 Herd of Models, August 2024. URL http::/arxiv. org/ abs 2407 21783 arXiv:2407.2177 [cs].
Lukas Edman, Gabriele Sarti, Antonio Toral, Gertjan van Noord, and Arianna Bisazza. Are Character- level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation. Transac- tions of the Association for Computational Linguistics, 12:392-410, April 2024. ISSN 2307-387X. doi. org 10. www21112ttaoo a 00651
doi: 10.1162/tacl_ a_00651. URL
Bruno Estigarribia. A Grammar of Paraguayan Guarani. Grammars of World and Minority Lan- guages. UCL Press, London, UK, August 2020. ISBN 978-1-78735-287- doi: 10.14224/111 ac ww/ wttp:///////eee///1o00000
9781787352872. wtt.httss::///tttttt http://ddcccooccoooo/ooooo scovery. ucl
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, June 2024. URL http::/ org abs / /203. 05530 arXiv:2403.05530 [cs].
Ryan Georgi, Fei Xia, and William Lewis. Improving Dependency Parsing with Interlinear Glossed Text and Syntactic Projection. In Martin Kay and Christian Boitet (eds.), Proceedings of COLING 2012: Posters, pp. 371-380, Mumbai, India, December 2012. The COLING 2012 Organizing Committee. URI //s/:t://// antho ogy.org/C12-2003
12

---

Published as a conference paper at ICLR 2025
Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. Dictionary- Phrase-level Prompting of Large Language Models for Machine Translation, February 2023. URL http:://aaa org/ httg/bs/23000 07856 arXiv:2302.07856 [cs].
Michael Ginn and Alexis Palmer. Robust Generalization Strategies for Morpheme Glossing in an Endangered Language Documentation Context. In Dieuwke Hupkes, Verna Dankers, Khuyag- baatar Batsuren, Koustuv Sinha, Amirhossein Kazemnejad, Christos Christodoulopoulos, Ryan Cotterell, and Elia Bruni (eds.), Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP. pp. 89-98, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2023. 1/2023.genbench. 2023. gen.enchnnecch.1
.ewbennc-bbeeeeee 1.7. URL https
ac http:///aaaaantoooooo.co org/
Michael Ginn, Sarah Moeller, Alexis Palmer, Anna Stacey, Garrett Nicolai, Mans Hulden, and Miikka Silfverberg. Findings of the SIGMORPHON 2023 Shared Task on Interlinear Glossing. In Garrett Nicolai, Eleanor Chodroff, Frederic Mailhot, and Çağm Çöltekin (eds.), Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology. pp. 186-201. Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18853/v1/203: htt.:////112220...gmmopoooooo 1.200. URL https:://aclatttologgg./o// 2223.sigmorpponn112222
Michael Ginn, Mans Hulden, and Alexis Palmer. Can we teach language models to gloss endangered languages?, June 2024a. URL http::tttttt.....too org. abs /400 18895 arXiv:2406. 18895 [cs].
Michael Ginn, Lindia Tjuatja, Taiqi He, Enora Rice, Graham Neubig, Alexis Palmer, and Lori Levin. GlossLM: Multilingual Pretraining for Low-Resource Interlinear Glossing, March 2024b. URL
http://aaw...../r////////333333/ 06399 MiXiv:2403.06390 [cs].
Leander Girrbach. Tü-CL at SIGMORPHON 2023: Straight-Through Gradient Estimation for Hard Attention. In Garrett Nicolai, Eleanor Chodroff, Frederic Mailhot, and Çağrı Cö1- tekin (eds.), Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pp. 151-165, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: 010665//11/2222.mmmmmoorrr 1.11.. URL
https: htappp//ccclathhooor..orogo
gmorphon-11111
Ping Guo, Yubing Ren, Yue Hu, Yunpeng Li, Jiarui Zhang, Xingsheng Zhang, and Heyan Huang. Teaching Large Language Models to Translate on Low-resource Languages with Textbook Prompt- ing. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference Computa- tional Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 15685-15697, Torino, Italy, May 2024. ELRA and ICCL. URL https: https::/accaatthoooogyy.
frec-main.oo 1362
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishray Chaudhary, and Mare' Aurelio Ranzato. The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6098 6111, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1/1/5//////11.... 1632. URL
org/D15 1-33
Taiqi He, Lindia Tjuatja, Nathaniel Robinson, Shinji Watanabe, David R. Mortensen, Graham Neubig, and Lori Levin. SigMoreFun Submission to the SIGMORPHON Shared Task on Interlinear Glossing. Garrett Nicolai, Eleanor Chodroff, Frederie Mailhot, and Çağrı Çöl- tekin (eds.), Proceedings of the 20th SIGMORPHON workshop Computational Research in Phonetics, Phonology, and Morphology, pp. 209-216, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: 10. 111.6//////22330gmmmmrrhpo 1.22 URL hittps://e http:://cccannhooo..ororg// httpa/tttt/////llatbbboooo...o.
Cheng-Ping Hsieh, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What's the Real Context Size of Your Long-Context Language
13

---

Published as a conference paper at ICLR 2025
Models?, August /BL.Ett:://axx..oor//////////// [cs].
arXiv:2404.06654
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low- Rank Adaptation of Large Language Models, October 2021. URL http://arke org/ abs / 2000. 09685 arXiv:2106.0960 [cs].
Vivek Iyer, Bhavitvya Malik, Pavel Stepachev, Pinzhen Chen, Barry Haddow, and Alexandra Birch. Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation, August 2024. URL http:://axxv..rr/ppp/p// ab4p 12780 MiXiv:2408.1278 [cs].
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The State and Fate of Linguistic Diversity and Inclusion the NLP World. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282-6293, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/v 1/2020. acl- www.020.a--maai.55006 m60n. URL 2020 aww.amal..maaa..mmmmm 560.
Tom Koemi, Vilém Zouhar, Christian Federmann, and Matt Post. Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2019,2014, Bangkok, Thailand, August 2024. Association for Computational
htRl.ttts::/ httptt/peeee..aaaaaaaooooooor//p wwt.:924.cclloooo
George Lakoff. Some Remarks AI and Linguistics. Cognitive Science, 2()) 2 1978. ISSN 1551-6709. URL https://onl iine wineeirrraaa harlllbrrrrraleee..r.e. s1551 61551679960 6755660000
Agustín Lucas, Alexis Baladón, Victoria Pardiñas, Marvin Toraals.. Santiago Góngora, and Luis Chiruzzo. Grammar-based Data Augmentation for Low-Resource Languages: The Case of Guarani-Spanish Neural Machine Translation. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6385-6397, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL
Jean Maillard, Cynthia Gao, Elahe Kalbassi, Kaushik Ram Sadagopan, Vedanuj Goswami, Philipp Koehn, Angela Fan, and Francisco Guzman. Small Data, Big Impact: Leveraging Minimal Data for Effective Machine Translation. In Proceedings of the 61 Annual Meeting of the Association for Computationnal Linguisties (Volume 1: Long Papers), pp. 2740-2756, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.
acl-ool...llolonn... 1544
//ap::://aaaaae URL httpt:tt/////. /ac aawacaannhhhooooooo Sescchooogge log.. 2020 aacl acog. 154
Chaitanya Malaviya, Graham Neubig, and Patrick Littell. Learning Language Representations for Typology Prediction. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2529-2535, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 18653/v 1268. /ttp::tts:///caaatt www.aattoooooooog///oooo 1268
Richard E. Mayer. Should There Be Three-Strikes Rule Against Pure Discovery Learning? American Psychologist, 59(1):14-19, 2004. ISSN 1935-990X. doi: 02.1037/0003-066X.55
Angelina McMillan-Major. Automating Gloss Generation in Interlinear Glossed Text. In Allyson Ettinger, Gaja Jarosz, and Joe Pater (eds.), Proceedings of the Society for Computation in Linguistics 2020, pp. 355 366. New York, New York, January 2020. Association for Computational Linguistics. URL https:/ http://aaaaaaaahhaaecee
org. 2020 sci1-1.42
Raphaël Merx, Aso Mahmudi, Katrina Langford, Leo Alberto de Araujo, and Ekaterina Vylomova. Low-Resouree Machine Translation through
LLM Prompting: A Study on the Mambai Language. In Atul Kr. Ojha, Sina Ahmadi, Silvie Cinková, Theodorus Fransen, Chao-Hong Liu, and John P. McCrae (eds.), Proceedings of the 2nd Workshop on Resources and
14

---

Published as a conference paper at ICLR 2025
Technologies for Indigenous, Endangered and Lesser-resourced Languages in Eurasia (EURALI) @ LREC-COLING 2024, pp. 1-11, Torino, Italia, May 2024. ELRA and ICCL. URL https
/aallanchoooyy.om org/ 2024. eurali-1.1
Sarah Moeller, Ling Liu, Changbing Yang, Katharina Kann, and Mans Hulden. IGT2P: From Interlinear Glossed Texts to Paradigms. In Bonnie Webber, Trevor Cohn. Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), ) pp. 5251-5262, Online, November 2020. Association for Computational Linguistics. doi: 10. 2tt55///122222..mmmmmmoom 2020. wwnnp-mmiinn.....m...m...mmmmmmm 424
URL attpss:/aaclatthollee.
org/
David R. Mortensen, Ela Gulsen, Taiqi He, Nathaniel Robinson, Jonathan Amith, Lindia Tjuatja, and Lori Levin. Generalized Glossing Guidelines: An Explicit, Human- and -eahiale (tem-and-Procee Convention for Morphological Annotation. In Garrett Nicolai, Eleanor Chodroff, Frederic Mailhot, and Çağrı Çöltekin (eds.). Proceedings of the 20th SIGMORPHON workshop on Computeational Research Phonetics, Phonology, and Morphology, pp. 38-67, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18653/v 1tt33/1112220 /022 1.7. URL http:///aaaaathoooy....om org/ 2023. sigmorphonn-1.
Sebastian Nordhoff. Modelling and Annotating Interlinear Glossed Text from 280 Different En- dangered Languages as Linked Data with LIGT. In Stefanie Dipper and Amir Zeldes (eds.), Proceedings of the 14th Linguistic Annotation Workshop. pp. 93 93-104, Barcelona, Spain, Decem- ber 2020. Association for Computational Linguistics. URL https: //acl www.ccannnooooly.ooooooooooooooe org/ 2020.. 1.9
Sebastian Nordhoff and Harald Hammarström. Glottolog/Langdo:: Defining dialects, languages, and language families as collections of resources. In First International Workshop on Linked Science 2011, Bonn, Germany, October 2011. LISC. URL htttp:://dl.hett hhnd///hh...aa.l.eee net/11888/
00 10-0011000000001----00 000000000011------3333
Sebastian Nordhoff and Thomas Krämer. IMTVault: Extracting and Enriching Low-resource Lan- guage Interlinear Glossed Text from Grammatical Descriptions and Typological Survey Articles. In Thierry Declerck, John P. McCrae, Elena Montiel, Christian Chiarcos, and Maxim lonov (eds.), Proceedings of the 8th Workshop on Linked Data in Linguistics within the 13th Language Re- sources and Evaluation Conference, pp. 17-25, Marseille, France, June 2022. European Language Resources Association. URL htt.s: /aaaaccaantthoo wogthhoooyy....m ogy. wwg. 2wwp:/022... 1d1-1.3
Maria Nädejde, Siva Reddy, Rico Sennrich, Tomasz Dwojak. Marcin Munczys-Dowmunt, Philipp Koehn, and Alexandra Birch. Predicting Target Language CCG Supertags Improves Neural Machine Translation. In Ondřej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, and Julia Kreutzer (eds.), Proceedings of the Second Conference on Machine Translation, pp. 68- 79, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18633/111117-4777, URL.attss://claatthoooog ott.aa/r.hooooooooooo///0100000
OLAC. OLAC resources in and about the Karas language, 2024. URL http:/ www
language wrchivesc.cc.... wwt.:////gg/aaagggggaeaeeee
Arturo Oncevay, Barry Haddow, and Alexandra Birch. Bridging Linguistic Typology and Mul- tilingual Machine Translation with Multi View Language Representations. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pp. 2391-2406, Online, November 2020. Association for Computational Linguistics. doi: 10. hwtp:///.222.0.2.222. 187. URL www.lllyyy.oooooog..o. 2ttp://222..mm.... ma in. 187
Juri Opitz, Shira Wein, and Nathan Schneider. Natural Language Processing RELIES on Linguistics, May 2024. URL URL htstt::/aaxivv.cr/aabb//// 02tp//5522500556533 arXiv:2405.0596 [cs].
Edoardo Maria Ponti, Helen O' O'Har,, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. Modeling Language Variation and Universals: A Survey Typological Linguistics for Natural Language Processing. Computational Linguistics, 45 (3):559-601, September 2019. doi: 10.1162/coli a 00357. URLhttppe h/ap:////cnn pTg/J19-3000
15

---

Published as a conference paper at ICLR 2025
Maja Popović. chrF++: words helping character n- n-gramm. In Ondřej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Ji- meno Yepes, Philipp Koehn, and Julia Kreutzer (eds.), Proceedings of the Second Conference on Machine Translation, pp. 612-618, 8, Copenhagen, Denmark, September 2017. Association for Computational Linguisties. doi: 10. N0116633/1/W17-47770 URL htt.hhhtsss/eet h ogy
Skg/w17-4777
Rita Ramos, Everlyn Asiko Chimoto, Maartje ter Hoeve, and Natalie Schluter. GrammaMT: Im- proving Machine Translation with Grammar-Informet In-Context Learning, October 2024. URL
http://arxiv. org/ abs/2410 18702 arXiv:2410.18702.
Surangika Ranathunga and Nisansa de Silva. Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (eds.), Proceedings of the 2nd Conference of the Asia- Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 823-848, Online only, November 2022. Association for Computational Linguistics. URL https: 2022. aa22.aacl-ma.. mann.. 62.
Victor Raskin. Linguistics and Natural Language Processing. In Proceedings of the first Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages, 1985.
URL httpt::://aaaaaattttttt///
tmi-1.17
Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, and Chris Dyer. Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale. Transactions of the Association for Computational Linguistics, 10:1423-1430 2022, doi: 10.1162/tacl a 00526. URL https: www.antooogg.oorgg/oooooooooo tacl-1
81
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving Neural Machine Translation Models with Monolingual Data. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 96,96, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16- 1009. URL
Bhargav Shandilya and Alexis Palmer. Lightweight morpheme labeling in context: Using structured linguistic representations to support linguistic analysis for the language documentation context. In Garrett Nicolai, Eleanor Chodroff, Frederic Mailhot, and Çağn Çöltekin (eds.). Proceedings of the 20th SIGMORPHON workshop on Computationall Research in Phonetics, Phonology, and Morphology, pp. 78-92, Toronto, Canada, July 2023, Association for Computational Linguis- http//acllantoool.....oo.o
tics. doi: 10. 20118855311/2222 www.:023...immmoooooooo 1.9. URL
Si sigmorphen--2 1.9
Hedvig Skirgård, Hannah J. Haynie, Damián E. Blasi, Harald Hammarström, Jeremy Collins, Jay J. Latarche, Jakob Lesage. Tobias Weber, Alena Wiizlack-Makarviich, Sam Passmore, Angela Chira, Luke Maurits, Russell Dinnage, Michael Dunn, Ger Reesink, Ruth Singer, Claire Bowern, Patience Epps. Jane Hill, Outi Vesakoski et al. Grambank reveals the importance of genealogical constraints on linguistic diversity and highlights the impact of language loss. Science Advances, 9(16) April 2023a. doi: 10.1 26/sciadvvadg66177 brq//oo//00 1126 6 httpsscaadv..acc.. adq
URL https: //ww:/wwwww.... science
Hedvig Skirgård, Hannah J. Haynie, Harald Hammarström, Damián E. Blasi, Jeremy Collins, Jay Latarche, Jakob Lesage, Tobias Weber, Alena
Michael Dunn, Ger Reesink, Ruth Singer, Claire Bowern, Patience Epps. Jane Hill, Outi Vesakoski, Noor Karolin Abbas, Sunny Ananth, Daniel Auer, Nancy A. Bakker et al. Grambank v1.0, March 2023b. URL
5281/ 2221//zeodo.. zenodo. 7740140
Garrett Tanzer, Mirac Suzgum, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi A Benchmark for Learning Translate a New Language from One Grammar Book. In Proceedings of the Twelfth International Conference on Learning Representations, 2024. URL/tttps //tp:////peeeeeeeeeee
16

---

Published as a conference paper at ICLR 2025
Hans Uszkoreit. Linguistics in Computational Linguisties: Observations and Predictions. In Timothy Baldwin and Valia Kordoni (eds.), Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious Vacuous?, pp. 22
25, Athens, Greece, March 2009. Association for Computational Linguistics. URL https 7/aclantho www.aaathhoooooooooooooo. wTw.000090099000000
ogy.
Tamara van Gog, Nikol Rummel, and Alexander Renkl. Learning how solve problems by studying
examples. In John Dunlosky and Katherine A.Editors Rawson (eds.). The cambridge handbook of
cognition and education, Cambridge handbooks in psychology, pp. 183-208. Cambridge University Press, Cambridge, 2019.
Eline Visser. Kalamang dictionary. Dictionaria, (13): 2020. doi: 10.528 URL
Eline Visser. A grammar of Kalamang. Language Science Press, Cambridge, MA, USA, January 6499927
2022. ISBN 108-3-96110-34--0 URL http:sss//o....r//11.235.223
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of- Prompting Elicits Reasoning in Large Lan- guage Models. Advances in Neural Information Processing Systems, 35:24824-24837, Decem-
ber 2022. /RLLitttpsa//pproeeeddnee neur
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models, February 2024. URL http://arxiv. org/ wtw.////.... 11674 arXiv:2309. 1674 [es].
Linting Xue, Aditya Barua, Noah Constant, Rami AI-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistiex, 00:291-3060 2022. doi: 10. 10.1162/tacL__00046
httpshts:://// http:/::///laaattoooooo....o.o wttrg/2022.taaa 1.17
Chen Zhang, Xiao Liu, Jiuheng Lin, and Yansong Feng. Teaching Large Language Models an Unseen Language the Fly, June 2024a. arXiv:2401.19161 [cs].
Kexun Zhang, Yee Man Choi, Zhenqiao Song, Taiqi He, William Yang Wang, and Lei Li. Hire Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions, February 2024b. URL httpps::///xxx..orr///oaao/
18025 1xXiv:2401.180% [cs].
Xingyuan Zhao, Satoru Ozaki, Antonios Anastasopoulos, Graham Neubig, and Lori Levin. Automatic Interlinear Glossing for Resoureed Languages Leveraging Translations. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference Computational Linguistics, pp. 5397-5408 Barcelona. Spain (Online), December 2020. Interna- tional Committee Computational Linguistics. doi: 10. 10.18653/w1/2220.cooo main.471. URL maw.nn-mmmamn111
https: //tp::///clllcc. htttp:://aclanthet http://.......oooooooooo/////...
Zhong Zhou, Lori Levin, David R. Mortensen, and Alex Waibel. Using Interlinear Glosses as Pivot in Low Resource Multilingual Machine Translation, March 2020. URL http::/
abs/ boss/191... 02709
101iv1111..07000 [cs].
Robert Östling and Jörg Tiedemann. Continuous multilinguality with language vectors. In Mirella Lapata, Phil Blunsom, and Alexander Koller (eds.), Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 644 649, Valencia, Spain, April 2017. Association for Computational Linguistics. URL
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. UDapter: Typology-based Language Adapters for Multilingual Dependency Parsing and Sequence Labeling. Computational Linguistics, 48(3):555 592,, September 2022. doi: 10.1162/coli a 00443. URL https:// Ceg.anttolooyy.og/20000
17

---

Published as a conference paper at ICLR 2025
A KALAMANG GRAMMAR BOOK EXTRACT
In Figure 3 provide brief extract from the Kalamang grammar book (Visser 2022) where the first paragraph exemplifies BOOKnon-para and examples 17 and 18 show the format of B0o
Clitics are mainly inflectional, and include postpositions as well as aspect and moad markers. Among the derivational clities are attributive =ten, which derives adjectives from verbs but also attested in non-verbal predicates ($6.3.5) and causative ($114.4). Cliticisation always occurs after affixation. A derived noun, for example, can carry a postposition. Lenget 'villager' from leng 'village' and agent nominaliser becomes lenget=at when it the last constituent of the object NP, as in (17). Amkeiret 'birth parent' from amkeit 'to give birth' and agent nominaliser -ef inflected with animate lative -kongga, as (18).
(17) ma sontum leng-et-at
merengguen
3sG person
gather
'He gathered the village people.
[narr27_3:1]
(18) don wa me se amkett-et-kongga
thing FROx
give_ birth-NMIZEAN.IAT
'This thing comes from the birth parent.'
[conv20_366 00:003
Figure brief passage from Visser 2022). showing the format of nooKnon-para (above) and B00K para (examples 17 and 18) explaining morphological feature of Kalamang.
B STATISTICAL TESTS
As discussed in Section 5.1 we fit linear regression models to CHRF++ score with test set type coverage as the independent variable for Gemini eng=kgv translation experiments. We find the models are significantly useful in both directions according the F-test, (p < 0.005). For eng-kgv: F(1, 15) 79.3, 12 0.84, 2.3 10-7, and for kgv-eng: F(1, 15) 98.1, n2 = 0.87 5.7 x 10 The Pearson correlations of these results are also significant, where for eng-kgv: 0.92. 1.1 10-? 7, and for kgv-eng:
0.93, 2.8 10
Finally, in modelling CHRF++ with prompt tokens as the independent variable for BOOK(all/p) p/ we find the resulting linear models not significant according the F-test. For eng-kgv, p 0.997, F(1, 1) 0.00, R² 0.00; and for kgv-eng, = 0.78, F(1,1) = 0.13, R² = 0.11.
There is no correlation between the number of tokens and the observed CHRF++ score.
C TEST SET ANALYSIS
To illustrate the weakness of the kgv test set, we generate eng-xxx test sets in: Dutch (nld), German (deu), French (fra), and Spanish (spa) using Google Translate and test Gemini' 's performance on these sets to find the upper bound. Table shows the test set is weak and a score below 50 C+RF++ falls far below the observed high-resource upper bound. The 100 example set also falls well below standard translation test sets in size, usually 500-1000 examples (Costa-jussà et al.) 2024). We addressed the issues of simplicity and size by testing the npi and gug FLORES test sets.
D TYPOLOGICAL FEATURE PROMPT
We provide an extract of the kgv-eng typological feature summary constructed from Grambank (Skirgård al. 2023b) in Table
https://cloud. www.le...ooooooo.ooooooooooo htt.://......g.....oooo/oaaaaaaa
18

---

Published as a conference paper at ICLR 2025
Table CHRF++ and BLEU scores of Gemini zero-shot tests the translated 100-example kgv test set, plus our best kgv results.
Setting
BEST
O-SHOT
Language
kgv
kgv
nld
deu
fra
spa
Direction
CHRF++
43.7
46.6
11.0
12.7
80.5
73.7
80.1
67.1
87.8
72.2
84.0
73.1
BLEU
12.2
22.5
0.0
0.0
61.1
53.6
63.4
44.4
80.0
51.6
74.2
53.6
Table 7: An extract of our typological feature prompt TYP constructed from Grambank data, specify- ing features and descriptions for both source (kqv) and target (eng) languages where available.
The following typological features describe the grammatical features of Kalamang and English including word order, verbal tense, nominal case, other language universals. Each feature is assigned a value that indicates the extent to which the language tends to exhibit that feature.
Feature ID: GB020 Are there definite specific articles? Kalamang Value: absent, Code Kalamang coded for this feature, meaning the feature absent.
This feature indicates Kalamang does not obligatorily encode the grammatical function of definite articles.
English Value: present, Code
English coded for this feature, meaning the feature is present. This feature indicates English obligatorily encodes the grammatical function of definite articles.
Below is short summary of the grammatical feature, an explanation of the process for assigning the feature's code, and examples of the feature from other languages including interlinear glossed text.
Are there definite or specific articles?
Summary An article is marker that accompanies the noun and expresses notions such as (non- )specificcyy and (in)definiteness. Sometimes these notions of specificity and definiteness are summed in the term 'identfiability". The formal expression is irrelevant; articles be free, bound, marked by suprasegmental markers such tone. Articles different from demonstratives that demonstratives markers have clear spatial deictic function. As demonstratives can grammaticalize into definite or specific articles, they form natural continuum, making it hard to define discrete categorics, but to qualify article marker should be used in some cases express definiteness without also expressing spatial deictic meaning.
in paradigm
Procedure Code there morpheme that can mark definiteness or specificity without also conveying a spatial deictic meaning.
2. Code 0 if the source does not mention a definite article and you cannot find one in examples or texts in an otherwise comprehensive grammar.
3. Code if the grammar does not contain enough analysis to determine whether there definite article not.
4. If you have coded for GB020 and 0 for GB021 and GB022, please write comment explaining the position of the definite or specific article. This is the end of the summary for feature GB020: "Are there definite specific articles?".
Feature ID:: [...]
This is end of the typological feature summary for Kalamang and English.
E PROMPT EXAMPLES
To further clarify the difference between prompt settings, we provide brief excerpts in Table
19

---

Published as a conference paper at ICLR 2025
Table Excerpts from various prompt settings for kgv-eng translation. All prompts also include the text from the 0-SHOT setting.
Setting
Example
0-SHOT
Kalamang is language spoken on the Karas Islands in West Papua. Translate the following sentence from Kalamang to English: [source]
Now write the translation. If you are not sure what the translation should be, then give your best guess.
Do not say that you do not speak Kalamang. Do not say you not have enough information, you must make a guess. If your translation is wrong, that is fine, but you have to provide a translation.
Your translation must be the first line of your response, with no other text before the translation. Only explain your reasoning after providing the translation. is crucial that you only give the translation on the first line your response, otherwise you will fail. Now write the translation:
Kalamang: [source] English:
5*-SHOT
To help with the translation. here is translated sentence with words similar to [source word] in list of translated Kalamang-Fnng English reference sentences:
Kalamang: [source].
English translation: [target]
To help with the translation, here is a translated sentence...
WORDLIST
To help with the translation, here a Kalamang-Enghh word list:
Kalamang: =a = English: focus marker
Kalamang:
English: filler
Kalamang: a' a English: yes
Kalamang: adat English: tradition
Kalamang: ade English: pejorative interjection
Kalamang: adi English: interjection of pain
PARAbook
To help with the translation, here are some example Kalamang English parallel sentences:
Kalamang: Bal se sorat koraru.
English translation: The dog has bitten the fish.
Kalamang: Mu kiem.
English translation: They run.
Kalamang: Ma reitkon purapi anat kamatet.
English translation:
He sent one hundred and fifty thousand rupiah.
PARA
To help with the translation, here are some example Kalamang English parallel sentences: Kalamang: bal se sor=at koraru Interlinear gloss: dog IAM fish=OBJ bite English translation:
The dog has bitten fish.
Kalamang: mu kiem = Interlinear gloss: 3PL run = English translation: They run.
Kalamang: ma reitkon purap-i an=at kamat=et = Interlinear gloss: 3SG hundred fifty- OBJQNT ISG=OBJ send=IRR English translation: He sent me one hundred and fifty thousand rupiah.
BOOKpara
To help with the translation, here the full text of Sallammnng-Fnggiin grammar book:
hal se sor=at koraru
dog 1AM Oish=OBJ bite 'The dog has bitten the fish.'
mu kiem 3PL run "They run.'
BOOK Koon-parn
To help with the translation, here the full text of a Kalamang-English grammar book:
This description of Kalamang (ISO 639-3 code kgv, glottocode karal499), Papuan language of the Greater West Bomberai family, is spoken by around 130 people in East Indonesia. The majority of speakers live on the biggest of the Karas Islands, which lie just off the coast of the Bomberai Peninsula in West Papua province. The language is known Karas in older literature...
20

---

Published as a conference paper at ICLR 2025
F PROMPT VOCABULARY STATISTICS
In Table 9 we show test set out-of-vocabule vutcaul-v (OOV) type counts (i.e. unique words) and corre- sponding test type coverage in the input prompt for each setting. If the prompt includes word that is in the test set in the target language, we count that as an in-vocabulary type, and words which do not appear in the prompt as OOV; our denotation of OOV is therefore unrelated to the model's vocabulary. We additionally include token counts (individual occurrences of types) for each prompt.
Table 9: Test set OOV type counts and type coverage, plus token counts, for all prompt settings in eng=kgv translation.
eng-kgv
kgv-eng
Setting.
00V
Coverage (%)
OOV
Coverage (%)
Prompt Tokens
O-SHOT
374
0.0
395
0.0
W4W
374
0.0
395
0.0
0
WORDLIST (W)
171
54.3
164
58.5
9011
5*-SHOT PARAbook
201
46.3
127
67.8
852
PARAbook
201
46.3
127
67.8
15561
W
124
66.8
87
78.0
24572
PARAgai
93
75.1
62
84.3
29407
PARA book
227
39.3
120
69.6
22686
BOOK₁ W
203
45.7
77.0
99579
142
62.0
69
82.5
108590
PARAGaio
106
71.7
46
88.4
113425
BOOKpan para
219
41.4
121
69.4
18309
BOOK pon-ppra
243
35.0
133
66.3
81270
TYP O-SIIOT
374
0.0
395
0.0
68426
BOOKpara
219
41.4
121
69.4
86735
+ PARAbook
201
46.3
127
67.8
83987
+ PAR PARAbunkttrain
93
75.1
62
84.3
100581
G ADDITIONAL FINE- TUN-IG RESULTS
Table IO shows translation results for fine-tuning the instruction-tune Gemini on PARAbook data, and tested in a 0-SHOT setting. Included are results with Llama- ft and fine-tuned NLLB. Fine- tuning the small MT model more effective than tuning an L.M in this particular 0-SHOT setting.
Table 10: Translation results for eng=kgv with Gemin ni, Ll ama base, and NLLB, fine-tuned on the preprocessed PARAhos data. We observe tuning translation model is more effective than tuning an LLM (whether pretrained already instruction-tunet - in this setting.
CHRF++
Setting,
eng-kgv
kgv-eng
Model_,
Gemini-fr
Llama- ft
NLLB
Gemini-fi
L1 ama - ft
NLLB
20.2
18.5
34.2
19.3
23.0
28.6
H LIMITATIONS
In addition to those noted in the main paper, we acknowledge the following limitations of this work. While combine the Kalamang test sets to give 100 example set, this is still far below standard test set for MT, often 1-2k sentences. In Kalamang, we are limited by the availability of additional data. However do test Nepali and Guarani with the FLORES devtest of 1012 examples which
21

---

Published as a conference paper at ICLR 2025
provides more realistic low-resource translation settings. Nepali and Guarani experiments also address generalisation issues of focusing only on one XLR language. Regarding evaluation, we note that many differences in CHRF++ score were fairly small, and reported in Koemi al. (2024) difference in CHRF (note, not CHRF++) of 3.05 is required for more than 90% of humans to agree that system is better than another in practice; this emphasises the need for future experiments and qualitative analyses (see Appendix D for a small scale qualitative analysis).
Further, the majority of our translation experiments are run on Gemini 1.1.1.5F1.. 5-Fassh, an APl-only LLM. Given the nature of our long-contexi experiments, we are necessarily limited in our choice of model-ar at the time of running experiments and to our knowledge, no other model family can handle context lengths over 200k tokens which is necessary for the entire Kalamang book. We run selected short-context experiments with the open-weight L,1 amalm.ma3... 1-8B model to improve the generalisation of our results, and we leave tests with other long-context models to future work. We finally note that while ideally we would have a larger kgv test set, running long-context inference of paid API models for > lk examples becomes prohibitively expensive. This limitation applies to the entire method of long-context LLM prompting, justifying the fine-tuning of smaller, open- weigh,, local models for XLR translation instead especially for members of these language communities who are unlikely to have access to large API models but may have access to free GPUs through services such Google Co and Kaggle"
I QUALITATIVE EVALUATION
Table 17 shows 7 test set examples of Kalamang to English translation with various Gem n prompting settings. We note again that a qualitative evaluation of English to Kalamang translation is not possible without a Kalamang speaker among the authors. We also note that the test set examples have been available online from Dictionarie (Visser) (2000) and its related Github repository (9 since November 2020. We argue this does not compromise our results, since we always compare performance with the book to O-SHOT settings; whether or not the model has already seen the test set is less relevant if the O-SHOT performance extremely poor, as is the case for kgv. Let us now qualitatively discuss each one in turn.
In Example [ O-SHOT only translates the borrowed word 'fiber' and the name (visible to due capitalisation), but is otherwise irrelevant. 5*-SHO gets some vocabulary correct such as 'boat' and 'grandfather', but misses the overall meaning. While BOOK-p manages some correct lexical translation, many words are incorrect and the overall meaning is lost. Both BOOKall and BOOK, get the general meaning correct, but BOOK, is more accurate, correctly generating 'two' and 'are' over 'is', and more naturally predicting 'the red one'. BOOKp is therefore marginally more grammatically correct and fluent, in relation to the reference target.
Example 2 shows predictably poor performance in the 0-SHOT setting. For 5*-SHOT, the model manages some correct lexical translations but the sentence-level meaning is lost. BOOK and BOOK, get most of the meaning; however they both miss some lexical translation (e.g. 'sacrifice' rather than 'medicine") and incorrectly predict verb tenses. BOOK-p only correctly translates few words (including 'child' and "born'), and generates an unrelated sentence.
In Example 3 0-SHOT is again an inadequate translation (despite being fluent). Here, the SHOT setting is also off-target in meaning, being unable to find translation for 'Desili' and instead using as name. Book-p also fails to translate this word, and the output is irrelevant. BooK and BooK, predict a similar meaning, close to the target; however, BOOK, correctly generates the tenses of past continuous 'planing' and the present simple 'cut', instead of the simple past 'planed' and 'went to cut'. Therefore here the parallel, glossed examples in BOOK, help to predict correct grammar moreso than the grammatical explanations in BOOK,n and BOOK-p-
Example f shows a largely irrelevant 0-SHOT translation, with the correct proper noun. 5*-SHOT gets the possessive 'father', and the meaning of 'one hundred', but the overall meaning is lost. The Book settings are similar and get different aspects of lexical and sentence-level meaning correct. BOOKal is
researcch google. ccm/
https://ww.kaaglee.tt httss:/www.keeeee..oomm
22

---

Published as a conference paper at ICLR 2025
the worst among them, predicting inadequate output. BoOK-p correctly translates the meaning of "one hundred", but fails to translate "walorkawat'; and BooK, is the only setting to mostly correctly translate 'coconut leaves', but misses the meaning of 'father's ' family' and "one hundred'.
In Example 5 0- O-SHOT'E is completely wrong. SHOSHOT and BOOK, outputs are identical, and close to the meaning but lack the reference's specificity. BoOKat and Book p however both predict negation, and output a grammar book-style sentence showing the indeterminate gender of the pronoun with 'He/She', which is penalised against the reference; BOOK-, also misses the meaning of sickness.
Example 6 again illustrates the largely inadequate 0-SHOT performance. Here, the SHOSHOT setting is fairly lexically accurate with 'beach' and 'tall' (against 'long'), but misses the sentence-level meaning. BOOK-p fails to translate 'beach' but gets some of the meaning; while Bookall and BOOKp both get some aspects correct: the former keeps the beach's name but misses the word 'beach', and the latter misses the name but predicts 'beach'.
Finally, in Example we see another failure of the 0-SHOT setting. With 5 SHOT the model gets the verbs correct but misses some vocabulary (i.e. 'the bay") and the general meaning. Both BOOK and BoOK-., predict 'District Officer' for 'Camat', which the Indonesian translation, while the reference denotes this as a given name, showing the model relying on previously observed unrelated vocabulary when lacking translation the prompt. BOOK, is closer to the reference meaning for the first clause, though with the present perfect 'has come' instead of the simple past 'came', and misses some meaning in the second clause. BOOKal is further away from the reference in the second clause referring 'what' rather than 'Camat/him' and BoOK p misses 'the bay' and has an incorrect subject for 'know".
In summary, 0-SHOT is predictably irrelevant but fluent; SHOT tends to give correct lexical translations with incorrect sentence-level meaning; BOOK predicts some higher-level meaning but lacks lexical translation adequacy; and BoOkan and BooK, produce the best translations which largely complete at the sentence-level, with BOOK, sometimes generating more precise grammar and lexical translations.
23

---

Published as a conference paper at ICLR 2025
Table 11: Examples of source, target, and predicted outputs for 0-SHOT, 5*-SHOT, BOOK (all/e/
settings, in kgv-eng translation with Gemini.
Setting
Output
kgv src
Fiber eira metko, kon tama Dian taraunkin? Kon kerkapten.
eng igt
There are two fibre boats there, which one Dian's 's grandfather's 's? The red one.
0-SHOT
The fiber is wet, is it Dian's ' fault? It's broken.
-HOS
there fibre boat there, where is Dian's ' grandfather? It's getting dark.
(1)
BoOKall
The fiber boat is there, which one is Dian's ' grandfather's? The one that is red.
BOOK,
Those two fiber boats are over there, which one is Dian's grandfather'ss '?? The red one. there fiber over there, which one is Dian's s? It's the one with the cover.
BOOK-p
kgv sre
Wa me kariak kaia kon untuk emumur kalo tumun amkeiret mu wara nanet. This is blood medicine for mothers, when they have given birth they drink this. You should be careful when you go out at night.
eng igt
0-SHOT
5 5*-SHOT
This one is a kind of food for women if child is born they eat it.
(2)
BoOkall
This is the blood of a woman who has given birth to a child.
BOOK,
This is blood sacrifice for the women that the children will be born.
BOOk-p
This is story about a child who was born on the island.
kgv src
Desili koyet an koi panggutki tenaunat tawara. After planing, use the panggut to chop the keel.
eng igt
0-SHOT
The young man is very strong.
5H-S
Desili is diligent and she is planting cassava.
(3)
BoOKall
After planed, went to cut the keel.
BOOK,
After planing, again cut the keel.
Book-p
Desi is already here and she is asking for drink.
kgv src
Kalamang esun mu walorkawat reitkon. Kalamang's & father's family have one hundred coconut leaf stems.
eng tgt
0-SHOT
The Kalamang people are very strong. Kalamang's 's father is waiting for the hundredth day.
5*-SHOT
(4)
Bookall
The Kalamang people are fishing for lobsters.
BoOK,
The Kalamang man has many coconut leaves.
BoOk p
The Kalamang people have one hundred walorkawat.
kgv sre
Ma kaleuna ning. He has a kidney-disease. You are very kind. He is sick.
eng tgt
0-SHOT
5*-SHOT
(5)
BOOKall
He/She is not sick. He is sick.
BOOK,
BoOk
He/She is not going.
kgv sre
Os Kamburkadok me kahen. The beach at Kambur is long.
eng tgt
0-SHOT
The Kamburkadok are coming. The one on the beach is tall.
**-SHOT
(6)
BOOKal all
The sand on Kamburkadok is far away.
BOOK,
The sand the beach is far away.
BOOk "
The Kamburkadok is far away.
kgv sre
Camat mu lukta, in arep neko komahal, Camat and family came, we in the bay didn't know.
eng tgt
0-sHOT
The chief is sick, and the people are worried.
5*-S1IOT
They came first, we don't know where they went.
(7)
BoOKall
The District Officer came to us, we don't know what's inside the bay.
BOOK,
The Camat has come, we don't know where he the bay.
BOOK-p
The district officer came here, but he doesn' 't know where we are.
24


Reviews:
[{"review_id": "y7WTTo8KDT", "reviewer": "Reviewer_nVmi", "summary": "The study presents a very thorough revision of the claim that prompting LLMs with grammar books of an unseen language enables the model to translate from/into that language. The main claim is that actually the translation examples in grammar books are the key support of handling the language, while grammatical explanations do not bring any improvement. For that purpose the work uses grammatical books for Kalamang and Nepali and presents contrastive experiments with several language models (Gemini, Llama 3.1, other more specific models), covering prompting, fine-tuning and LoRA. Besides splitting the grammar books into parallel examples and everything else the authors also include a list of words and other supplementary material. In addition to the task of translating, the grammatically judgement is also included, where grammatical descriptions are useful unlike with translation.\n\nAlthough the study is narrowly targeted, it presents very clearly and exhaustively verified conclusions on the capability of LLMs, showing once again that their behavior is more akin to elaborate pattern matching, rather than comprehension and thinking.", "questions": "* would you expect different conclusions if the prompted / tuned LMs were much smaller?, what about larger models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* wide coverage of materials, models and approaches\n* presented clearly (except for the complexity of results in Table 2) and convincingly", "weaknesses": "* minor weakness: there are so many different models and comparisons that Table 2 is hard to understand, the reader has to go back and forth between the table and section 4.1 to remind, what the abbreviations mean -- perhaps there is a way to improve the readability?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9IZuOEr7m9", "reviewer": "Reviewer_5eBi", "summary": "This paper evaluated whether LLMs can translate extremely low-resource languages using a grammar book. The findings reveal that parallel sentence examples are crucial for translation, while grammatical explanations offer little benefit. Tests on English-Kalamang and Nepali confirm that translation quality improves with parallel data rather than grammatical descriptions. For linguistic tasks like grammaticality judgment, a typological feature prompt is more helpful. The authors recommend focusing on parallel data collection over grammar descriptions for effective translation in low-resource languages.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.  The paper showed a novel study that whether grammar books alone support translation for extremely low-resource (XLR) languages or not. By isolating grammar book components (e.g. parallel sentences vs. grammatical explanations), they illustrated a comprehensive evaluation, complemented by introducing typological prompts for linguistic tasks, a creative addition for XLR NLP.\n\n2. By the controlled experiments across different closed-source (Gemini), open-source LLMs (Llama 3.2 8b), and NLLB NMT model, They found out that LLMs can't effectively learn from grammatical explanations for XLR MT in Kalamang and Nepali, relying instead on the parallel sentences. Also, We showed that fine-tuning NLLB model matches the performance of LLMs with long context. Additionally, LLMs can utilize grammatical information effectively when applied to suitable tasks, e.g. grammaticality judgment or IGT prediction, especially when enhanced with targeted grammatical data like our typological prompt, which achieves top performance on these linguistic tasks.\n\n3. The paper is well-written, coherent with comprehensive analysis.", "weaknesses": "1. While the paper effectively examines the Kalamang and Nepali languages, its findings may not generalize across the diversity of low-resource languages. XLR languages can vary significantly in structure, script, and available resources, and focusing only on these two could limit the applicability of the conclusions. It would be nice to do similar analysis on low-resource languages too.\n\n2. The paper introduces typological prompts as a promising way to improve linguistic tasks but does not compare this approach against other techniques that incorporate linguistic typology (e.g., embeddings based on typological features from databases like [WALS](https://arxiv.org/abs/2010.03920))", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QTzaqsonN5", "reviewer": "Reviewer_DQQ1", "summary": "This paper examines the contribution of a grammar book to building a machine translation system for an extremely low-resource language, Kalamang, that LLMs cannot have seen in training. In Kalamang, there is also parallel sentences and a dictionary. The authors evaluate LLMs by placing different mixtures of these language resources into a long context and evaluate the model's ability for in-context learning with the goal of translating test samples to and from English. They find that the additional grammatical explanations contribute little, if anything, on top of the parallel sentences, which themselves help the LLM learn to translate Kalamang. They similarly find that for Nepali, a much higher-resource language (but still low) where models have likely seen some text for, parallel sentences contribute to improving MT but not the grammar book. They also show this when finetuning NLLB, a translation-specific model, and show that grammar is also token inefficient in ICL in comparison to parallel sentences.", "questions": "1. Line 44: how does the quantity of data available make Kalamang not unique ? I would better explain why the findings on Kalamang are relevant to many other languages\n2. I would discuss the fact that you are using in-context learning (ICL) more clearly in the intro or even abstract\n3. I would add more references to explain/justify the methodology in 3.3 and 3.5\n4. Line 215: parallel data annotation is typically very rigorous; annotators have to follow specific instructions or for scraping, sentence alignment has to be in a particular way or else the sample is thrown out. How clean & consistent are the parallel samples that come from your heuristic filter ? How did you determine what was appropriately parallel examples vs an explanation when no Kalamang speakers were involved in the project ? I would add more details about this heuristic filtering process.\n5. Notation for tables is confusing and takes a while to digest. It might be worth including quick natural language names like instead of Book_p, putting \"Grammar Book, parallel subset\"\n6. Table2: how is Gemini able to do 11&13 ChrF when it has never seen Kalamang before ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper is very well organized and is very clear to follow\n2. The paper properly investigates a research question, with few, if any, holes in their experimental design as far as I can find\n3. The presentation of the experiments and results are clear and the findings align logically with the numerical results\n4. The authors incorporate novel methods (as well as new datasets) to understand the importance of grammatical explanations for low-resource MT", "weaknesses": "1. In general, the biggest weakness of the paper is the significance of its findings:\n\na. It is already established that parallel data, the most directly related samples to the MT task, are the most effective to training MT models. In many languages, parallel data is in fact the only form of labeled data that exists. For Nepali and Guarani, for example, it feels obvious that a grammar book would not help given that Gemini and Llama have likely implicitly seen many Nepali and Guarani texts in pretraining (or other highly related Indo-Aryan languages with similar grammar).\n\nb. I understand that, by nature of targeting extremely low-resource languages, there are very few available data sources for such an experiment. But generalizing these results past Kalamang seems questionable, especially given that there is a singular grammar book, presumably written by a very small number of people. If possible, this paper would be strengthened by a few more languages (more similar in task than Nepali). I would also suggest more context on grammar books themselves be provided and how frequent these are for extremely low-resource languages.\n\nc. Title is a bit misleading in that it asks whether using grammar books are useful, all to prove that not really. The title moreso implies that the paper would show that grammar books are useful.\n\n2. In comparison to ACL venues, ICLR attendees will typically have less linguistics background. I am concerned the discussion of more specific linguistics topics (e.g. interlinear gloss text, grammar & typology) might be not well suited for this audience. I will leave that to the AC to discern, but also advise authors modify the Related Work & Methodology section to take that into account.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)