Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?

## Abstract

Abstract Extremely low-resource (XLR) languages lack substantial corpora for training NLP models, motivating the use of all available resources such as dictionaries and grammar books.
Machine Translation from One Book (Tanzer et al., 2024 ) suggests prompting long-context LLMs with one grammar book enables English–Kalamang translation, an unseen XLR language—a noteworthy case of linguistic knowledge helping an NLP task.
We investigate whether the book’s grammatical explanations or its parallel examples are most effective for learning XLR translation, finding almost all improvement stems from the parallel examples.
Further, we find similar results for Nepali, a seen low-resource language, and achieve performance comparable to an LLM with a grammar book by simply fine-tuning an encoder-decoder translation model. We then investigate where grammar books help by testing two linguistic tasks, grammaticality judgment and gloss prediction, and we explore what kind of grammatical knowledge helps by introducing a typological feature prompt that achieves leading results on these more relevant tasks. We thus emphasise the importance of task-appropriate data for XLR languages: parallel examples for translation, and grammatical data for linguistic tasks.
As we find no evidence that long-context LLMs can make effective use of grammatical explanations for XLR translation, we suggest data collection for multilingual XLR tasks such as translation is best focused on parallel data over linguistic description.

## 1 Introduction

Most of the world’s languages are extremely low-resource (XLR), severely lacking in suitable corpora for NLP tasks (Joshi et al., ; Ranathunga & de Silva, ), such as parallel data for machine translation (MT). However, over 50% of languages have both a dictionary and a grammar (Nordhoff & Hammarström, ). While human-readable, grammar texts are less easily incorporated into standard NLP models due to their non-standard, unstructured format. Large language models (LLMs), which competently handle free-form textual instructions, provide a potential solution to this data mismatch. After pre-training on trillions of tokens (in mainly high-resource languages), LLMs can learn tasks from only a few in-context examples (Brown et al., ; Wei et al., ). For this reason, interest in grammar texts for NLP tasks is growing (Tanzer et al., ; Zhang et al., ; ).

Machine Translation from One Book (Tanzer et al., ) claims LLMs can learn to translate between Kalamang (ISO 639-3: kgv)—a newly-documented language unseen in LLM training data—and English (eng) using only a grammar book in-context. We note that kgv is not unique as an XLR language: it has over 3,000 parallel sentences, a dictionary with over 3,000 definitions (Visser, ), a 500-page grammar book (Visser, ) consisting of grammatical explanations and over 1000 parallel glossed examples, and nearly 100 typological feature specifications (Skirgård et al., ; ). These resources place kgv alongside thousands of XLR languages (OLAC, ), and we expect most are also minimally represented in LLMs’ pretraining data. Given this, finding methods to effectively exploit the available kgv resources could have wider implications for XLR NLP.

In this paper, we question the claimed utility of grammatical explanations for XLR MT with LLMs, then ask where and what kind of grammatical knowledge helps. We show that:

- Parallel examples are essential for translation  We disentangle grammar books’ parallel examples from grammatical explanations, finding explanations add no significant advantage over parallel data: adding $+$0.7 ChrF++ into kgv, while into eng the book lags $-$0.3 points against its parallel sentences; and quality drops up to 8 points with parallel data removed. Our findings generalise to Nepali (npi), a seen low-resource language, where the book’s parallel sentences outperform the full book by 4 ChrF++. LLMs fail to effectively exploit grammatical explanations for translation.
- Fine-tuning matches long-context LLMs  We fine-tune small translation models on the parallel data, achieving competitive results within 0.2 ChrF++ of the performance of Gemini with a grammar book into kgv, and beating Llama-3.1-8B settings with access to the same data by up to 20 points. Parallel examples (especially with glosses) are both more token-efficient and readily available than grammar books, and enable computationally cheaper methods than long-context LLMs.
- Typological prompting outperforms explanations and helps linguistic tasks  We introduce a novel typological feature prompt, and for kgv and npi translation we find our method is more effective than explanations into eng, but not into XLR languages. On kgv grammaticality judgment, our typological prompt improves up to 3% over the book’s 1.2k parallel sentences and 8% over the whole book. For gloss prediction, parallel sentences again beat the book by up to 5.3% morpheme accuracy, and adding typology achieves leading performance on this task. Therefore LLMs can exploit grammar for relevant linguistic tasks—if in a useful form—but not for translation.

Task appropriate-data is therefore essential. In the current paradigm, we suggest that data collection for XLR MT is thus better focused on parallel data over linguistic description, given the advantages in token efficiency, computational cost, and availability.

## 2 Related Work

#### Grammar for extremely low-resource machine translation

Translation of low-resource languages (meaning $<$100k parallel examples) with LLMs is currently of significant interest (Cahyawijaya et al., ; Court & Elsner, ; Iyer et al., ). Methods include fine-tuning (Xu et al., ), dictionary prompting (Ghazvininejad et al., ), and retrieval-augmented few-shot prompting (Merx et al., ). Alongside advances in long-context LLMs, recent work has introduced grammar information in-context for various tasks:  Guo et al. () test a textbook-style prompt with LLM-generated parses, seeing limited gains against parallel sentences; and Zhang et al. () add a singular syntactic rule to their prompt with small effects. Zhang et al. () chain morphological analysis, a dictionary and an LLM-summarised grammar book in-context, observing small gains from book passages over a dictionary-only setup. Others meanwhile use grammars with LLMs for data augmentation (Lucas et al., ) or as a hybrid rule-based translation system (Coleman et al., ).

The recent work Machine Translation from One Book (MTOB) (Tanzer et al., ) is central to the question of grammar for LLM-based XLR translation. MTOB introduces a translation test set for the newly documented XLR language Kalamang, plus a grammar book and additional parallel sentences. MTOB suggests long-context LLMs can exploit linguistic knowledge (a grammar book) for an NLP task (XLR translation). This represents a strong contribution to XLR NLP, leveraging underused linguistic resources with state-of-the-art LLMs.
However, several issues weaken its claims. The test sets of 50 short, easy examples are potentially too small for making wider generalisations, and the human baseline is somewhat flawed as they may learn from examples at test time; relatedly, Gemini Team et al. () make the uncommon step of using a non-fluent annotator to rate model outputs and their own kgv predictions. Furthermore, using ChrF is less appropriate than ChrF++, which factors in word order and is the de-facto standard in XLR translation (Maillard et al., ; Costa-jussà et al., ; Edman et al., ). The results would benefit from further ablations since the signal from parallel sentences and explanations is not disentangled, and we also miss a strong translation approach to the problem. Finally, we note that the grammar book is not designed for learning the language, but rather describes theoretical linguistic phenomena—which the authors note limits LLMs to a basic competence. MTOB therefore leaves open questions over LLMs’ ability to exploit linguistic information for XLR tasks. In this paper, we tackle these issues by combining the test sets, using automatic ChrF++ scores, disentangling the parallel/non-parallel signal, and testing two better aligned tasks for grammar books.

#### Linguistics in NLP

The incorporation of linguistic information into NLP models is a long-standing question with mixed results (Lakoff, ; Raskin, ; Uszkoreit, ; Opitz et al., ). Past work sees gains from incorporating syntactic knowledge into translation models using constituency parses (Currey & Heafield, ), grammar supertags (Nădejde et al., ), or tree-structured models (Sartran et al., ). Typological features are a useful form of linguistic description available for many languages in standardised databases (Dryer & Haspelmath, ; Skirgård et al., ; ); features describe language variation in terms of phenomena such as word order, verb tenses, and noun cases. Feature specifications are usually drawn from grammar books by linguists, condensing fine-grained, unprocessed textual descriptions into high-level features. Typological features have been incorporated into NLP models with some success in the form of embeddings (Malaviya et al., ; Östling & Tiedemann, ) but several studies find minimal positive effects on performance (Ponti et al., ; Üstün et al., ). To test whether LLMs follow this trend, we construct a novel prompt that uses readily available typological feature specifications for source and target languages, as an in-context and language-invariant method for bridging cross-lingual grammatical differences.

#### Interlinear gloss prediction

Interlinear Glossed Text (IGT) is a structured, standardised data format for linguistic analysis comprising a morphologically segmented transcription (where morphemes are the smallest units of meaning), an interlinear gloss containing lexical and grammatical information, and a translation (Comrie et al., ; Mortensen et al., ); a Kalamang example is shown in 2 (Visser, ). Glossing is an essential step in language documentation (Ginn et al., ) and is designed for trained linguists, rather than language learners. Glosses have been widely applied in NLP tasks, including as a pivot for translation (Zhou et al., ), dependency parsing (Georgi et al., ), grammar generation (Bender et al., ), morphological analysis (Moeller et al., ; Shandilya & Palmer, ), and linguistic resource development (Beermann et al., ; Buchholz et al., ). Predicting IGT is therefore a well-motivated task, and segmented IGT is a valuable linguistic resource. IGT prediction is most relevant for XLR languages where it is impactful in assisting annotators for documentation and preservation (Ginn et al., ). Prior methods include supervised neural models (Zhao et al., ; Girrbach, ), or adapting multilingual language models (He et al., ; Ginn et al., ; ). Since IGT is costly to generate, past work has scraped it from books (Nordhoff, ; Nordhoff & Krämer, ); we use this method to extract Kalamang glosses from the grammar book. In this paper, we test gloss prediction to better understand whether LLMs can understand and use grammatical knowledge from the grammar book in-context.

bal se sor=at na ma se nan=i koyet
Transcription
dog iam fish=obj consume 3sg iam consume=plnl finish
Interlinear Gloss
\glt‘The dog ate the fish, after he ate.’
Translation
\z

## 3 Methodology

### 3.1 Grammar Books for Translation

Our methods are guided by open questions over the use of grammar books for XLR translation. First, we manually filter the grammar books into parallel examples and word pairs, and explanatory, descriptive text, to disentangle the signal from translations and grammatical explanations (see Appendix A for a kgv book extract). This novel ablation is necessary to understand which specific aspects of grammar books are useful for XLR MT. We ask whether LLMs really learn effectively from the grammar explanations, or if most translation supervision stems only from the book’s parallel examples. We combine the directional test sets into a single 100 example test set to improve the generalisability of these results, and evaluate with ChrF++ (Popović, ) to take word order into account(^1^11We omit human evaluation (Gemini Team et al., , cf.) given the infeasibility of engaging proficient Kalamang speakers. See Appendix H for a small-scale qualitative analysis of several kgv–eng examples.). We also test eng–npi translation, a low-resource language with (we expect) a low data weight in LLMs and an established evaluation set, Flores (Costa-jussà et al., ); while not unseen, these experiments broaden our results to seen low-resource languages more generally.(^2^22Grammar book splits for both kgv and npi will be made available after review.)

### 3.2 Neural Machine Translation Approaches

To compare the LLM-based approach with a standard MT approach for learning to translate a language as yet unseen by the model, we run experiments fine-tuning NLLB-1.3B (Costa-jussà et al., ) on the parallel data sourced from the grammar book. We expect similar results to be achieved with the same resources using a small, specialist encoder-decoder model, which would confirm that the useful translation signal stems from the parallel sentences contained within grammar books—which constitute less than 20% of the kgv grammar book’s total tokens (see Table 1 for token counts).

### 3.3 Typological Feature Prompting

In asking what kind of grammatical knowledge can aid LLMs in XLR tasks, we introduce a method for incorporating typological information into prompts. We extract categorical typological feature specifications from Grambank Skirgård et al. () for kgv, npi, and eng, and use a rule-based template to construct a prompt containing features for each language and a short explanation. For an example of the prompt format, see Appendix B. Most languages with grammar books have some typological feature specification, since features are distilled by trained annotators from external resources. This method therefore isolates the high-level grammatical tendencies of a language from the specific instantiations of those features (i.e. parallel examples). We hypothesise this method, when combined with the grammar book’s parallel sentences, will at least match the performance of the grammar book. We expect that providing explicit features such as word order rules removes some reasoning requirements for the LLM. Conversely, typological features will not have relevant parallel examples, so some reasoning and retrieval is still required, potentially tempering the advantages.

### 3.4 Grammaticality Judgment

To test the LLM’s ability to acquire knowledge and understanding of Kalamang grammar from the book, we introduce a discriminative grammar judgment experiment. We ask the model to choose the original Kalamang test sentence against a modified example, with three successively easier settings: swapping two adjacent words (Swapadj), two random words (Swapran), and shuffling all words (Shuffle). We acknowledge that while we cannot guarantee all corruptions are ungrammatical (since no author speaks Kalamang), we assume the original examples are linguistically unmarked sentences. For all settings we expect a 0-shot model to achieve approximately 50% accuracy, while for high-resource languages we would expect near 100% accuracy. We expect the grammar book to have a greater positive impact in this setting where grammatical knowledge is explicitly rewarded.

### 3.5 Interlinear Glossed Text Prediction

To explore another more relevant task for exploiting grammar explanations, we test IGT prediction with the grammar book against few-shot and supervised baselines. This experiment tests whether LLMs can learn grammar from a book to the extent that we see a difference in performance on a grammar-focused task.
IGT requires both lexical translation and grammar analysis, without any generation in the language at hand. This makes IGT prediction a more appropriate task to perform from a descriptive, non-didactic grammar text. We argue that IGT prediction accelerates XLR documentation more than translation, and is likely to have more direct impact for both first language (L1) speakers and linguists, not to mention the potential downstream uses e.g. POS tagging and MT. IGT prediction is also a well defined task with strong baselines and clear evaluation metrics (Ginn et al., ). Finally, we argue grammar books are intuitively suited to IGT prediction more than translation, because their unique contribution is glossed text, rather than just parallel sentences. We use all available sentences with IGT from Dictionaria as our test set, and for our supervised baselines, we process the grammar book IGT examples into a training and development set. We expect the grammar book to provide marginal gains over raw parallel sentences because the grammar book explicitly explains the glossed examples therein.

## 4 Experimental Setup

### 4.1 Data

We use the preprocessed Kalamang (kgv) grammar book (Visser, ; Tanzer et al., ), with additional processing of irregularities (particularly for glossing) introduced in LaTeX conversion. We similarly preprocess a grammar text in Nepali (npi) (Bal, ). We prompt with the entire grammar, $\textsc{Book}_{all}$, in-context. We heuristically filter parallel glossed examples and bilingual word/phrase pairs from the book into a parallel subset, $\textsc{Book}_{p}$. The remainder of the book contains grammatical explanations without parallel signals, labelled $\textsc{Book}_{\neg p}$. Subset statistics are shown in Table 1. We preprocess kgv parallel examples from the grammar book into an unsegmented, structured format, giving $\textsc{P}_{book}$ (used for 5*-shot examples and in full as a prompt) and $\textsc{P}_{book}^{\textsc{IGT}}$ which includes glosses (1239 examples) – for excerpts of prompt types, see Appendix C. We additionally test prompts with $\textsc{P}_{train}$ (400 examples) and Wordlist (W) (3813 examples). Additionally, we sample 500 examples from Dictionaria(^3^33[https://dictionaria.clld.org/contributions/kalamang](https://dictionaria.clld.org/contributions/kalamang)) (Visser, ) as the development set for fine-tuning. In total there are 3.3k eng$\rightleftharpoons$kgv parallel examples(^4^44Data and code will be shared after review.); we focus on the 1.2k from $\textsc{P}_{book}$ for fair comparison with Book settings. For testing, we use our combined 100 example test set for Kalamang, and the Flores devtest for Nepali (1012 examples) (Guzmán et al., ), with few-shot examples from Flores dev.

For IGT prediction, we preprocess 1221 examples from the Grammar Book with complete glosses for the training (5623 words) and development (612 words) set (split 90:10% by sentences). Following Ginn et al. (), we introduce a test set of 97 glossed examples (447 words) from a different source, Dictionaria, which were sanity-checked manually for correct alignment.

**Table 1: Dataset statistics for grammar book subsets, in lines and space-separated tokens.**
| Language | Split | Lines | Tokens |
| --- | --- | --- | --- |
| kgv | $\textsc{Book}_{p}$ | 4489 | 17858 |
| $\textsc{Book}_{\neg p}$ | 2282 | 81268 |  |
| npi | $\textsc{Book}_{p}$ | 759 | 5333 |
| $\textsc{Book}_{\neg p}$ | 2896 | 23233 |  |

### 4.2 Models

In our experiments we use the API-only Gemini-1.5-Flash-001 (henceforth Gemini) (Gemini Team et al., ). We justify this choice due to Gemini’s context window of 1M tokens, significantly larger than other models, which can handle the entire grammar book, and use Flash over Pro due to the prohibitive cost differences. We also use the open-weight Llama-3.1-8B(^5^55[https://huggingface.co/meta-llama/Meta-Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B)) (Dubey et al., ) and its instruction-tuned variant, with a context of 128k tokens. This is insufficient for kgv $\textsc{Book}_{all}$, but fits $\textsc{Book}_{p}$ and $\textsc{Book}_{\neg p}$, plus the npi $\textsc{Book}_{all}$. Since this work is intended to help under-resourced language speakers, the minimisation of costs by using smaller, open-weight models is well-motivated. We test Llama-Instruct (Llama-I), and fine-tune Llama base with LoRA (Hu et al., ) on Pbook (Llama-ftp) with prompt masking for 5 epochs with a constant learning rate of 1e-4, batch size 4, and LoRA $\alpha=16$, $r=16$, targeting all linear projections. For our NMT baseline, we fine-tune NLLB-1.3B-Distilled (NLLB) (Costa-jussà et al., ) on Pbook.
For Kalamang grammaticality judgment and IGT prediction, we use the same Gemini model as above.

### 4.3 Evaluation

We evaluate translation automatically with ChrF++ (Popović, ). We favour ChrF++ over ChrF, used in Tanzer et al. (), since it takes into account word order, which is central to correct grammar. We report scores for trimmed responses after the first newline character to distinguish translation quality from overgeneration and chat explanations (Aycock & Bawden, ) and use a forceful prompt (detailed in Appendix C) to ensure the translation is produced on the first line.

### 4.4 Baselines

For translation experiments, we test the following baselines: 0-shot translation with a standard translation prompt; word-for-word translation with fuzzy dictionary lookup (W4W); 5 retrieved examples per word (5*-shot) based on longest common subsequences following Tanzer et al. (); prompting with the full dictionary (Wordlist, W); the processed set of book parallel examples ($\textsc{P}_{book}$) and with glosses ($\textsc{P}_{book}^{\textsc{igt}}$); and processed training set examples ($\textsc{P}_{train}$).

For IGT prediction, we use a baseline frequency-based classifier (Top-Class), a fine-tuned RoBERTa token classifier (Ginn et al., ) (SMP-Base); a hard-attention glossing model (TüCL-Morph) (Girrbach, ); and ByT5-ft and Glosslm-ft models (Ginn et al., ) fine-tuned on our kgv IGT training and development sets. We provide segmented input, and English translations to models which accept them.

### 4.5 Experiments

Our central research question investigates the contributions of grammatical explanations and parallel data to translation performance. We therefore prompt models with $\textsc{Book}_{all}$ and the complementary subsets $\textsc{Book}_{p}$ and $\textsc{Book}_{\neg p}$. We test our typological feature prompt, Typ, as a replacement for $\textsc{Book}_{\neg p}$. For npi, we repeat the book settings as above. We fine-tune translation models with the preprocessed Pbook parallel data for comparison with $\textsc{Book}_{p}$ settings. For grammaticality judgment and IGT prediction tasks, we similarly test Gemini with the kgv $\textsc{Book}_{\{all/p/\neg p\}}$ and Typ prompts.

## 5 Results & Analysis

**Table 2: Translation results for eng$\rightleftharpoons$kgv with Gemini, Llama-Instruct (L-I) and fine-tuned (L-ft), and prompt tokens counted with NLTK’s tokenizer (Bird et al., ). Highest Bookp scores are underlined, highest overall are bolded. Grey rows indicate settings with access to anything other than the book’s parallel data. Tests ruled out by context lengths are indicated by –. *W4W tests are not run with Gemini but are included for comparison. The book’s parallel sentences almost match or outperform the grammar book, while its grammatical explanations perform poorly.**
| Model |  | Morph Acc. | Word Acc. | Stem F1 | Gram F1 | ChrF++ |
| --- | --- | --- | --- | --- | --- | --- |
| Top-Class | (Ginn et al., ) | 44.0 | 39.7 | 40.6 | 57.8 | 34.5 |
| SMP-Base | (Ginn & Palmer, ) | 45.2 | 41.7 | 39.7 | 58.9 | 34.3 |
| TüCL-Morph | (Girrbach, ) | 43.6 | 38.8 | 40.0 | 50.7 | 35.4 |
| ByT5-FT | (Xue et al., ) | 40.8 | 48.6 | 40.9 | 45.4 | 49.0 |
| GlossLM-FT | (Ginn et al., ) | 43.8 | 47.7 | 41.5 | 50.4 | 49.1 |
| 10*-shot |  | 43.9 | 43.7 | 44.3 | 45.2 | 46.4 |
| $\textsc{Book}_{all}$ |  | 40.1 | 31.5 | 38.7 | 43.4 | 40.5 |
| $\textsc{Book}_{p}$ |  | 45.4 | 42.1 | 44.0 | 49.0 | 45.0 |
| $\textsc{Book}_{\neg p}$ |  | 21.0 | 8.8 | 23.9 | 15.6 | 26.0 |
| Typ + $\textsc{Book}_{p}$ |  | 46.1 | 40.9 | 44.2 | 50.5 | 44.8 |

### 5.1 Results

#### Grammar versus parallel sentences for translation

We disentangle the signal from grammar books’ explanations and parallel sentences for translation. Our kgv results in Table 5 show that most or all performance improvements stem from the book’s parallel sentences, with quality plummeting when parallel data is removed. With Gemini into eng, $\textsc{Book}_{p}$ marginally outperforms $\textsc{Book}_{all}$, and beats $\textsc{Book}_{\neg p}$ by 7 ChrF++, while into kgv, $\textsc{Book}_{p}$ outperforms $\textsc{Book}_{\neg p}$ by over 8 points, and $\textsc{Book}_{all}$ performs 3 points better than $\textsc{Book}_{p}$. However, we show statistically in Section 5.2 that this small improvement is modelled directly by an increase in test set vocabulary coverage, rather than from the grammatical explanations. Additionally, this gap closes with the $\textsc{P}_{book}^{\textsc{igt}}$ prompt, which preprocesses and structures the parallel data in $\textsc{Book}_{p}$ into kgv–gloss–eng triples. $\textsc{P}_{book}^{\textsc{igt}}$ performs particularly well for Llama-I, with over 10 points improvement over $\textsc{Book}_{p}$ into kgv. Due to context limits restricting kgv $\textsc{Book}_{all}$ tests, conclusions with Llama-I are limited, but we find again that $\textsc{Book}_{\neg p}$ performance lags far behind $\textsc{Book}_{p}$. We note that baselines including 0-shot show kgv translation is non-trivial. We also find that additional parallel data further improves translation quality, and note 5*-shot is generally competitive despite its short average prompt, achieving the best $\textsc{Book}_{p}$ into kgv with Gemini. Thus for kgv translation, both LLMs on test mainly learn from the book’s parallel sentences, failing to exploit the grammatical explanations.

We observe a similarly strong trend for npi, a seen low-resource language with a high-quality Flores test set, in Table 5.1. $\textsc{Book}_{p}$ settings either match or outperform $\textsc{Book}_{all}$ for both models and directions. We note few settings beat 0-shot and differences between Gemini settings are less pronounced than for kgv; perhaps the model’s prior competence and a shorter grammar book mean there is less to be gained. However, analysing Book settings in isolation shows that both $\textsc{Book}_{all}$ and $\textsc{Book}_{\neg p}$ have a detrimental effect of up to 4 points below 0-shot with Llama-I, while $\textsc{Book}_{p}$ has a neutral or small positive impact into and from npi respectively. Finally 5*-shot is again effective especially for Llama-I into npi, likely due to the greater vocabulary coverage of the example set. These results generalise our findings for kgv to seen low-resource languages: we find no evidence that LLMs can effectively exploit grammatical explanations for translation.

**Table 3: Translation results for eng$\rightleftharpoons$npi with Gemini and Llama-I. Best Bookp (white rows) scores are underlined, best overall are bolded. While Bookall and Book¬p decrease performance from 0-shot, Bookp has a neutral or positive effect into and from npi respectively.**
| Model |  | Morph Acc. | Word Acc. | Stem F1 | Gram F1 | ChrF++ |
| --- | --- | --- | --- | --- | --- | --- |
| Top-Class | (Ginn et al., ) | 44.0 | 39.7 | 40.6 | 57.8 | 34.5 |
| SMP-Base | (Ginn & Palmer, ) | 45.2 | 41.7 | 39.7 | 58.9 | 34.3 |
| TüCL-Morph | (Girrbach, ) | 43.6 | 38.8 | 40.0 | 50.7 | 35.4 |
| ByT5-FT | (Xue et al., ) | 40.8 | 48.6 | 40.9 | 45.4 | 49.0 |
| GlossLM-FT | (Ginn et al., ) | 43.8 | 47.7 | 41.5 | 50.4 | 49.1 |
| 10*-shot |  | 43.9 | 43.7 | 44.3 | 45.2 | 46.4 |
| $\textsc{Book}_{all}$ |  | 40.1 | 31.5 | 38.7 | 43.4 | 40.5 |
| $\textsc{Book}_{p}$ |  | 45.4 | 42.1 | 44.0 | 49.0 | 45.0 |
| $\textsc{Book}_{\neg p}$ |  | 21.0 | 8.8 | 23.9 | 15.6 | 26.0 |
| Typ + $\textsc{Book}_{p}$ |  | 46.1 | 40.9 | 44.2 | 50.5 | 44.8 |

#### Fine-tuning versus in-context learning

We test a standard MT approach for adding a new language by fine-tuning NLLB, a small MT model, on the book’s parallel data, shown in Table 5.1. NLLB achieves competitive or improved performance compared to prompting Gemini with the same preprocessed parallel data, $\textsc{P}_{book}$.
We also test backtranslation (BT), a standard method to boost performance in MT (Sennrich et al., ). A single BT iteration with $\textsc{P}_{train}$ has a negative impact into kgv, likely due to the poor quality of the initial model introducing excessive noise. However we see a boost of 3 ChrF++ into eng, we expect because of the strong English language modelling of NLLB. Further, adding a small 400 example parallel training set sees large gains of 4-8 points. These results suggest the MTOB benchmark can be adequately addressed as a standard XLR MT problem with simple data preprocessing, a small pre-trained model, and fine-tuning on a single GPU for 1 hour.

**Table 4: Translation results for eng$\rightleftharpoons$kgv with NLLB, an MT model, fine-tuned on book parallel data; equivalent in-context learning results with Gemini are shown for comparison. Fine-tuned NLLB achieves competitive results with an LLM given the same parallel data, especially into kgv.**
| Model |  | Morph Acc. | Word Acc. | Stem F1 | Gram F1 | ChrF++ |
| --- | --- | --- | --- | --- | --- | --- |
| Top-Class | (Ginn et al., ) | 44.0 | 39.7 | 40.6 | 57.8 | 34.5 |
| SMP-Base | (Ginn & Palmer, ) | 45.2 | 41.7 | 39.7 | 58.9 | 34.3 |
| TüCL-Morph | (Girrbach, ) | 43.6 | 38.8 | 40.0 | 50.7 | 35.4 |
| ByT5-FT | (Xue et al., ) | 40.8 | 48.6 | 40.9 | 45.4 | 49.0 |
| GlossLM-FT | (Ginn et al., ) | 43.8 | 47.7 | 41.5 | 50.4 | 49.1 |
| 10*-shot |  | 43.9 | 43.7 | 44.3 | 45.2 | 46.4 |
| $\textsc{Book}_{all}$ |  | 40.1 | 31.5 | 38.7 | 43.4 | 40.5 |
| $\textsc{Book}_{p}$ |  | 45.4 | 42.1 | 44.0 | 49.0 | 45.0 |
| $\textsc{Book}_{\neg p}$ |  | 21.0 | 8.8 | 23.9 | 15.6 | 26.0 |
| Typ + $\textsc{Book}_{p}$ |  | 46.1 | 40.9 | 44.2 | 50.5 | 44.8 |

#### Typological prompting for linguistic tasks

Given the limited contribution of grammatical explanations to translation performance, we introduce a novel prompting method summarising languages’ typological features. This prompt is intended to replace $\textsc{Book}_{\neg p}$, thus we are primarily focused on results when combined with $\textsc{Book}_{p}$ data. Our results for eng–kgv translation in Table 5 show expectedly poor 0-shot performance due to the lack of any Kalamang text. Into kgv, our prompt beats $\textsc{Book}_{p}$ but not $\textsc{Book}_{all}$; however into eng, our prompt with $\textsc{Book}_{p}$ achieves the best translation results for settings with book parallel data. Similarly for Nepali in Table 5.1, Typ + $\textsc{Book}_{p}$ is less effective than $\textsc{Book}_{all}$ into npi, and marginally outperforms it into eng up to 0.5 ChrF++, though $\textsc{Book}_{p}$ alone performs best. The performance of typological prompting for translation is therefore inconsistent, supporting the above finding that LLMs fail to effectively exploit grammatical information for MT.

Figure: Figure 1: Grammaticality judgment accuracy in kgv; for reference in eng tests, Gemini scores 100%, 99%, and 100% respectively. Our prompt Typ + $\textsc{Book}_{p}$ performs best overall suggesting grammar can help LLMs for linguistic tasks.
Refer to caption: x1.png

To determine whether grammar is not useful for MT or LLMs cannot exploit grammatical explanations more broadly, we test two more relevant tasks: grammaticality judgment and IGT prediction. In Figure 1, grammaticality judgment results in kgv with Gemini show all settings perform similarly poorly on $\textsc{Swap}_{adj}$, though improving on 0-shot by around 7%. Generally, 10*-shot is worse than prompts with $\textsc{Book}_{p}$, likely because diverse sentences may help here more than overlapping vocabulary, which helps more for MT. For Book settings we observe that $\textsc{Book}_{p}$ matches or outperforms $\textsc{Book}_{all}$ across all three tests by up to 5%, and consistently beats $\textsc{Book}_{\neg p}$, by up to 18% in Shuffle tests. So far, the LLM still fails to exploit grammatical explanations effectively and learns mainly from parallel examples. However, our Typ + $\textsc{Book}_{p}$ setting performs best over the three tests by up to 3% over $\textsc{Book}_{p}$. These positive results suggest that LLMs can learn from grammar, given the right kind of grammatical knowledge and a relevant task.

For kgv IGT prediction, we compare Gemini settings with supervised baselines in Table 5. The leading performer in morpheme accuracy, the key IGT metric, is again our typological prompt Typ + $\textsc{Book}_{p}$, scoring 6% above $\textsc{Book}_{all}$, 0.5% over $\textsc{Book}_{p}$, and 25% over $\textsc{Book}_{\neg p}$. Additionally, our prompt beats all supervised systems by 1-5%, suggesting in-context learning with typological knowledge and parallel glossed examples is a strong method for XLR IGT prediction. Results for other metrics show slightly differing trends, with supervised models showing stronger word accuracies and Gram F1 scores (since most are closed-set classifiers). Generally though, $\textsc{Book}_{\neg p}$ shows extremely poor performance, while $\textsc{Book}_{p}$, 10*-shot, and Typ + $\textsc{Book}_{p}$ settings perform consistently well, often beating supervised baselines. We note that Typ + $\textsc{Book}_{p}$ scores show competent performance for both grammatical (on morpheme accuracy and Gram F1) and lexical aspects (via Stem F1) of IGT prediction, suggesting all-round competence on this task. These results reinforce our findings that while parallel sentences still provide most of the useful signal, LLMs can exploit grammatical—specifically typological—information for linguistic tasks.

**Table 5: IGT prediction results in kgv for supervised baselines and Gemini settings. Our Typ + $\textsc{Book}_{p}$ prompt achieves the highest morpheme accuracy and high scores on other metrics, while $\textsc{Book}_{all}$ performs poorly overall.**
| Model |  | Morph Acc. | Word Acc. | Stem F1 | Gram F1 | ChrF++ |
| --- | --- | --- | --- | --- | --- | --- |
| Top-Class | (Ginn et al., ) | 44.0 | 39.7 | 40.6 | 57.8 | 34.5 |
| SMP-Base | (Ginn & Palmer, ) | 45.2 | 41.7 | 39.7 | 58.9 | 34.3 |
| TüCL-Morph | (Girrbach, ) | 43.6 | 38.8 | 40.0 | 50.7 | 35.4 |
| ByT5-FT | (Xue et al., ) | 40.8 | 48.6 | 40.9 | 45.4 | 49.0 |
| GlossLM-FT | (Ginn et al., ) | 43.8 | 47.7 | 41.5 | 50.4 | 49.1 |
| 10*-shot |  | 43.9 | 43.7 | 44.3 | 45.2 | 46.4 |
| $\textsc{Book}_{all}$ |  | 40.1 | 31.5 | 38.7 | 43.4 | 40.5 |
| $\textsc{Book}_{p}$ |  | 45.4 | 42.1 | 44.0 | 49.0 | 45.0 |
| $\textsc{Book}_{\neg p}$ |  | 21.0 | 8.8 | 23.9 | 15.6 | 26.0 |
| Typ + $\textsc{Book}_{p}$ |  | 46.1 | 40.9 | 44.2 | 50.5 | 44.8 |

### 5.2 Analysis

#### Type coverage and Token efficiency

We investigate whether any added performance from grammatical explanations is statistically significant or can instead be attributed to greater test set type coverage in the prompt. We distinguish between types, meaning unique words in a vocabulary, and tokens, i.e. individual occurrences of types in a text.
We fit univariate least squares regression models to ChrF++ scores with test set type coverage as the independent variable, for both directions, shown in Figure 2. No setting falls outside the confidence interval, and the models are significant in both directions ($p<0.005$)(^6^66For details of these and following statistical tests, see Appendix G.); the correlations are also significant ($p<0.005$). Thus maximising target vocabulary coverage (via parallel sentences) in-context is the most efficient method for improving LLM-based XLR translation. These linear regressions show that translation performance can be directly modelled by test set vocabulary coverage, and that the book’s grammar explanations provide no significant advantage over its parallel sentences. See Appendix D for full statistics on our prompts’ test set type coverage.

We then explore whether the improved translation scores can be attributed to a longer (or shorter) prompt, by testing for a relationship between prompts’ total tokens and translation quality in terms of ChrF++ for $\textsc{Book}_{\{all/p/\neg p\}}$ with Gemini. The resulting linear models are not significant in either direction ($p=0.997$, $p=0.78$ into and from kgv), with no correlations. The grammar book is therefore both a token-inefficient way to learn (with similar performance despite nearly 5x more tokens than kgv $\textsc{Book}_{p}$), and a cost-inefficient dataset to generate, compared to using its parallel sentences. The needle-in-a-haystack problem could partially explain this: with increasing context, retrieval of relevant information (i.e. similar parallel examples) becomes harder (Hsieh et al., ), so while $\textsc{Book}_{p}$ is a subset of $\textsc{Book}_{all}$, there is a greater ratio of relevant to irrelevant information in the prompt—assuming grammatical explanations cannot be effectively exploited for translation.

Figure: Figure 2: Regression models of ChrF++ score against test type coverage for eng–kgv and kgv–eng translation with Gemini. Prompt settings are labelled for clarity. The plots show that translation performance can be statistically modelled by test set vocabulary coverage.
Refer to caption: x2.png

#### Discussion

We note that our results do not indicate LLMs cannot understand books in general; rather, we find no quantitative evidence that the results here and in MTOB show LLMs can effectively exploit grammar books (or linguistic knowledge) for translation. Indeed, we show that LLMs can exploit grammatical information in the form of typology for more relevant, linguistically-focused tasks. More broadly, from an educational perspective, translation is a problem-solving task aiming to reach a goal state (translation) via a series of actions given an initial state (source) and optionally rules on applying actions. Humans tend to learn more efficiently via worked-examples (van Gog et al., ), i.e. with explicit explanations, rather than pure discovery learning, meaning without explicit guidance (Mayer, ). Our results however indicate that for translation, LLMs learn more effectively from unannotated parallel examples (i.e. discovery) than from grammar principles with explained examples (i.e. example-based). Our results thus tentatively support a divergence between learning strategies for translation between human learners and LLMs learning in-context.

## 6 Conclusion

We find no evidence that LLMs can effectively exploit grammatical explanations for XLR MT in Kalamang and Nepali, relying instead on the parallel sentences within the book. This runs counter to the claim of prior work including MTOB which use grammar books to enable XLR tasks. We show that fine-tuning small MT models matches the performance of costly long-context LLMs. Further, we show statistically that grammar adds no significant advantage above the increased type coverage it provides, and that grammar books are less token-efficient for prompting than parallel sentences. However, LLMs can exploit grammatical information, given an appropriate task—e.g. grammaticality judgment or IGT prediction—and more useful grammatical data in the form of our typological prompt, which achieves leading results on these linguistic tasks. We therefore emphasise the importance of task-appropriate data: parallel data for MT, and grammatical, preferably typological, knowledge for linguistic tasks. Moreover, we suggest data collection efforts for multilingual XLR tasks, at least for MT, are better focused on parallel data over linguistic description, which enables less costly, more token-efficient translation.

## Ethics Statement

We emphasise that this work does not aim to and does not solve social problems, and instead investigates the utility of grammar books as resources for XRL NLP. We operate on the assumption of continued consent of the Kalamang community to use their language in our research, as discussed in Tanzer et al. (). We discuss the utility of our work (specifically the IGT prediction task) for linguists and L1 speakers, in its capacity to improve language documentation processes.

## Appendix A Kalamang Grammar Book Extract

In Figure 3, we provide a brief extract from the Kalamang grammar book (Visser, ), where the first paragraph exemplifies $\textsc{Book}_{\neg p}$, and examples 17 and 18 show the format of $\textsc{Book}_{p}$.

Figure: Figure 3: A brief passage from Visser (), showing the format of $\textsc{Book}_{\neg p}$ (above) and $\textsc{Book}_{p}$
Refer to caption: x4.png

## Appendix B Typological Feature Prompt

We provide an extract of the kgv–eng typological feature summary constructed from Grambank (Skirgård et al., ) in Table 6.

**Table 6: An extract of our typological feature prompt Typ constructed from Grambank data, specifying features and descriptions for both source (kgv) and target (eng) languages where available.**
| The following typological features describe the grammatical features of Kalamang and English including word order, verbal tense, nominal case, and other language universals. Each feature is assigned a value that indicates the extent to which the language tends to exhibit that feature. |
| --- |
| Feature ID: GB020<br>Are there definite or specific articles? |
| Kalamang Value: absent, Code 0 |
| Kalamang is coded 0 for this feature, meaning the feature is absent. |
| This feature indicates Kalamang does not obligatorily encode the grammatical function of definite articles. |
| English Value: present, Code 1 |
| English is coded 1 for this feature, meaning the feature is present. |
| This feature indicates English obligatorily encodes the grammatical function of definite articles. |
| —<br>Below is a short summary of the grammatical feature, an explanation of the process for assigning the feature’s code, and examples of the feature from other languages including interlinear glossed text.<br>—<br>Are there definite or specific articles?<br>Summary<br>An article is a marker that accompanies the noun and expresses notions such as (non-)specificity and (in)definiteness. Sometimes these notions of specificity and definiteness are summed up in the term ’identifiability’. The formal expression is irrelevant; articles can be free, bound, or marked by suprasegmental markers such as tone.<br>Articles are different from demonstratives in that demonstratives occur in a paradigm of markers that have a clear spatial deictic function. As demonstratives can grammaticalize into definite or specific articles, they form a natural continuum, making it hard to define discrete categories, but to qualify as an article a marker should be used in some cases to express definiteness without also expressing a spatial deictic meaning.<br>Procedure<br>1. Code 1 if there is a morpheme that can mark definiteness or specificity without also conveying a spatial deictic meaning. |
| 2. Code 0 if the source does not mention a definite article and you cannot find one in examples or texts in an otherwise comprehensive grammar. |
| 3. Code ? if the grammar does not contain enough analysis to determine whether there is a definite article or not. |
| 4. If you have coded 1 for GB020 and 0 for GB021 and GB022, please write a comment explaining the position of the definite or specific article. |
| This is the end of the summary for feature GB020: "Are there definite or specific articles?". |
| — |
| Feature ID: […] |
| — |
| This is the end of the typological feature summary for Kalamang and English. |

## Appendix C Prompt Examples

To further clarify the difference between prompt settings, we provide brief excerpts in Table 7.

**Table 7: Excerpts from various prompt settings for kgv–eng translation. All prompts include the text from the 0-shot setting.**
| Setting | Example |
| --- | --- |
| 0-shot | Kalamang is a language spoken on the Karas Islands in West Papua. Translate the following sentence from Kalamang to English: [source]<br>Now write the translation. If you are not sure what the translation should be, then give your best guess.<br>Do not say that you do not speak Kalamang. Do not say you do not have enough information, you must make a guess. If your translation is wrong, that is fine, but you have to provide a translation.<br>Your translation must be on the first line of your response, with no other text before the translation. Only explain your reasoning after providing the translation.<br>It is crucial that you only give the translation on the first line of your response, otherwise you will fail. Now write the translation:<br>Kalamang: [source] English: |
| 5*-shot | To help with the translation, here is a translated sentence with words similar to [source word] in a list of translated Kalamang-English reference sentences:<br>Kalamang: [source].<br>English translation: [target]<br>To help with the translation, here is a translated sentence… |
| Wordlist | To help with the translation, here is a Kalamang-English word list:<br>Kalamang: =a = English: focus marker<br>Kalamang: a = English: filler<br>Kalamang: a’a = English: yes<br>Kalamang: adat = English: tradition<br>Kalamang: ade = English: pejorative interjection<br>Kalamang: adi = English: interjection of pain<br>Kalamang: adu = English: interjection of surprise or pain<br>Kalamang: afukat = English: avocado<br>Kalamang: ahat = English: sunday |
| Pbook | To help with the translation, here are some example Kalamang-English parallel sentences:<br>Kalamang: Bal se sorat koraru.<br>English translation: The dog has bitten the fish.<br>Kalamang: Mu kiem.<br>English translation: They run.<br>Kalamang: Ma reitkon purapi anat kamatet.<br>English translation: He sent me one hundred and fifty thousand rupiah. |
| P${}_{book}^{\textsc{igt}}$ | To help with the translation, here are some example Kalamang-English parallel sentences:<br>Kalamang: bal se sor=at koraru = Interlinear gloss: dog IAM fish=OBJ bite = English translation: The dog has bitten the fish.<br>Kalamang: mu kiem = Interlinear gloss: 3PL run = English translation: They run.<br>Kalamang: ma reitkon purap-i an=at kamat=et = Interlinear gloss: 3SG hundred fifty-OBJQNT 1SG=OBJ send=IRR = English translation: He sent me one hundred and fifty thousand rupiah. |
| Bookp | To help with the translation, here is the full text of a Kalamang-English grammar book:<br>—<br>bal se sor=at koraru<br>dog IAM fish=OBJ bite<br>‘The dog has bitten the fish.’<br>mu kiem<br>3PL run<br>‘They run.’ |
| Book¬p | To help with the translation, here is the full text of a Kalamang-English grammar book:<br>—<br>This is a description of Kalamang (ISO 639-3 code kgv, glottocode kara1499), a Papuan language of the Greater West Bomberai family. It is spoken by around 130 people in East Indonesia. The majority of speakers live on the biggest of the Karas Islands, which lie just off the coast of the Bomberai Peninsula in West Papua province. The language is known as Karas in older literature … |

## Appendix D Prompt Vocabulary Statistics

In Table D, we show test set out-of-vocabulary (OOV) type counts (i.e. unique words) and corresponding test set type coverage in the input prompt for each setting. If the prompt includes a word that is in the test set in the target language, we count that as an in-vocabulary type, and words which do not appear in the prompt as OOV; our denotation of OOV is therefore unrelated to the model’s vocabulary. We additionally include token counts (individual occurrences of types) for each prompt.

**Table 8: Test set OOV type counts and type coverage, plus token counts, for all prompt settings in eng$\rightleftharpoons$kgv translation.**
| Setting | Best | 0-shot |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Language | kgv | kgv | nld | deu | fra | spa |  |  |  |  |  |  |
| Direction | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ |
| ChrF++ | 43.7 | 46.6 | 11.0 | 12.7 | 80.5 | 73.7 | 80.1 | 67.1 | 87.8 | 72.2 | 84.0 | 73.1 |
| Bleu | 12.2 | 22.5 | 0.0 | 0.0 | 61.1 | 53.6 | 63.4 | 44.4 | 80.0 | 51.6 | 74.2 | 53.6 |

## Appendix E Test set analysis

To illustrate the weakness of the kgv test set, we generate eng--xxx test sets in: Dutch (nld), German (deu), French (fra), and Spanish (spa) using Google Translate(^7^77[https://cloud.google.com/translate](https://cloud.google.com/translate)), and test Gemini’s performance on these sets to find the upper bound. Table 9 shows the test set is weak and a score below 50 ChrF++ falls far below the observed high-resource upper bound. The 100 example set also falls well below standard translation test sets in size, usually 500-1000 examples (Costa-jussà et al., ). We addressed the issues of simplicity and size by testing on the Nepali Flores test set.

**Table 9: ChrF++ and Bleu scores of Gemini zero-shot tests on the translated 100-example kgv test set, plus our best kgv results.**
| Setting | Best | 0-shot |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Language | kgv | kgv | nld | deu | fra | spa |  |  |  |  |  |  |
| Direction | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ | $\rightarrow$ | $\leftarrow$ |
| ChrF++ | 43.7 | 46.6 | 11.0 | 12.7 | 80.5 | 73.7 | 80.1 | 67.1 | 87.8 | 72.2 | 84.0 | 73.1 |
| Bleu | 12.2 | 22.5 | 0.0 | 0.0 | 61.1 | 53.6 | 63.4 | 44.4 | 80.0 | 51.6 | 74.2 | 53.6 |

## Appendix F Additional Fine-tuning results

Table F shows translation results for fine-tuning the instruction-tuned Gemini on $\textsc{P}_{book}$ data, and tested in a 0-shot setting. Included are results with Llama-ft and fine-tuned NLLB. Fine-tuning the small MT model is more effective than tuning an LLM in this particular 0-shot setting.

**Table 10: Translation results for eng$\rightleftharpoons$kgv with Gemini, Llama base, and NLLB, fine-tuned on the preprocessed $\textsc{P}_{book}$ data. We observe tuning a translation model is more effective than tuning an LLM (whether pretrained or already instruction-tuned) in this setting.**
| Setting | Output |
| --- | --- |
| kgvsrc | Fiber eira metko, kon tama Dian taraunkin? Kon kerkapten. |
| eng tgt | There are two fibre boats there, which one is Dian’s grandfather’s? The red one. |
| 0-shot | The fiber is wet, is it Dian’s fault? It’s broken. |
| 5*-shot | Is there a fibre boat there, where is Dian’s grandfather? It’s getting dark.              (1) |
| $\textsc{Book}_{all}$ | The fiber boat is there, which one is Dian’s grandfather’s? The one that is red. |
| $\textsc{Book}_{p}$ | Those two fiber boats are over there, which one is Dian’s grandfather’s? The red one. |
| $\textsc{Book}_{\neg p}$ | Is there fiber over there, which one is Dian’s? It’s the one with the cover. |
| kgvsrc | Wa me kariak kaia kon untuk emumur kalo tumun amkeiret mu wara nanet. |
| eng tgt | This is blood medicine for mothers, when they have given birth they drink this. |
| 0-shot | You should be careful when you go out at night. |
| 5*-shot | This one is a kind of food for women if a child is born they eat it.                       (2) |
| $\textsc{Book}_{all}$ | This is the blood of a woman who has given birth to a child. |
| $\textsc{Book}_{p}$ | This is a blood sacrifice for the women so that the children will be born. |
| $\textsc{Book}_{\neg p}$ | This is a story about a child who was born on the island. |
| kgvsrc | Desili koyet an koi panggutki tenaunat tawara. |
| eng tgt | After planing, I use the panggut to chop the keel. |
| 0-shot | The young man is very strong. |
| 5*-shot | Desili is diligent and she is planting cassava.                                                (3) |
| $\textsc{Book}_{all}$ | After I planed, I went to cut the keel. |
| $\textsc{Book}_{p}$ | After planing, I again cut the keel. |
| $\textsc{Book}_{\neg p}$ | Desi is already here and she is asking for a drink. |
| kgvsrc | Kalamang esun mu walorkawat reitkon. |
| eng tgt | Kalamang’s father’s family have one hundred coconut leaf stems. |
| 0-shot | The Kalamang people are very strong. |
| 5*-shot | Kalamang’s father is waiting for the hundredth day.                                       (4) |
| $\textsc{Book}_{all}$ | The Kalamang people are fishing for lobsters. |
| $\textsc{Book}_{p}$ | The Kalamang man has many coconut leaves. |
| $\textsc{Book}_{\neg p}$ | The Kalamang people have one hundred walorkawat. |
| kgvsrc | Ma kaleuna ning. |
| eng tgt | He has a kidney-disease. |
| 0-shot | You are very kind. |
| 5*-shot | He is sick.                                                                                        (5) |
| $\textsc{Book}_{all}$ | He/She is not sick. |
| $\textsc{Book}_{p}$ | He is sick. |
| $\textsc{Book}_{\neg p}$ | He/She is not going. |
| kgvsrc | Os Kamburkadok me kahen. |
| eng tgt | The beach at Kambur is long. |
| 0-shot | The Kamburkadok are coming. |
| 5*-shot | The one on the beach is tall.                                                                   (6) |
| $\textsc{Book}_{all}$ | The sand on Kamburkadok is far away. |
| $\textsc{Book}_{p}$ | The sand on the beach is far away. |
| $\textsc{Book}_{\neg p}$ | The Kamburkadok is far away. |
| kgvsrc | Camat mu lukta, in arep neko komahal. |
| eng tgt | Camat and family came, we in the bay didn’t know. |
| 0-shot | The chief is sick, and the people are worried. |
| 5*-shot | They came first, we don’t know where they went.                                          (7) |
| $\textsc{Book}_{all}$ | The District Officer came to us, we don’t know what’s inside the bay. |
| $\textsc{Book}_{p}$ | The Camat has come, we don’t know where he is in the bay. |
| $\textsc{Book}_{\neg p}$ | The district officer came here, but he doesn’t know where we are. |

## Appendix G Statistical Tests

As discussed in Section 5.2, we fit linear regression models to ChrF++ score with test set type coverage as the independent variable for Gemini eng$\rightleftharpoons$kgv translation experiments. We find the models are significantly useful in both directions, ($p<<0.005$). For eng–kgv: $F(1,15)=96.0,R^{2}=0.87,p=6.5\times 10^{-8}$, and for kgv–eng: $F(1,15)=90.9,R^{2}=0.86,p=9.3\times 10^{-8}$. The correlations of these results are also significant, where for eng–kgv: $r=0.93,p=3.3\times 10^{-8}$, and for kgv–eng: $r=0.93,p=4.7\times 10^{-8}$.

Finally, in modelling ChrF++ with prompt tokens as the independent variable for $\textsc{Book}_{\{all/p/\neg p\}}$, we find the resulting linear models are not significant. For eng–kgv, $p=0.997,F(1,1)=0.00,R^{2}=0.00$; and for kgv–eng, $p=0.78,F(1,1)=0.13,R^{2}=0.11$. There is no correlation between the number of tokens and the observed ChrF++ score.

## Appendix H Qualitative Evaluation

Table 11 shows 7 test set examples of Kalamang to English translation with various Gemini prompting settings. We note again that a qualitative evaluation of English to Kalamang translation is not possible without a Kalamang speaker among the authors. We also note that the test set examples have been available online from Dictionaria(^8^88[https://dictionaria.clld.org/contributions/kalamang#texamples](https://dictionaria.clld.org/contributions/kalamang#texamples)) (Visser, ) and its related Github repository(^9^99[https://github.com/dictionaria/kalamang/tree/v1.0](https://github.com/dictionaria/kalamang/tree/v1.0)) since November 2020. We argue this does not compromise our results, since we always compare performance with the book to 0-shot settings; whether or not the model has already seen the test set is less relevant if the 0-shot performance is extremely poor, as is the case for kgv. Let us now qualitatively discuss each one in turn.

In Example 1, 0-shot only translates the borrowed word ‘fiber’ and the name (visible to due capitalisation), but is otherwise irrelevant. 5*-shot gets some vocabulary correct such as ‘boat’ and ‘grandfather’, but misses the overall meaning. While $\textsc{Book}_{\neg p}$ manages some correct lexical translation, many words are incorrect and the overall meaning is lost. Both $\textsc{Book}_{all}$ and $\textsc{Book}_{p}$ get the general meaning correct, but $\textsc{Book}_{p}$ is more accurate, correctly generating ‘two’ and ‘are’ over ‘is’, and more naturally predicting ‘the red one’. $\textsc{Book}_{p}$ is therefore marginally more grammatically correct and fluent, in relation to the reference target.

**Table 11: Examples of source, target, and predicted outputs for 0-shot, 5*-shot, $\textsc{Book}_{all/p/\neg p}$ settings, in kgv–eng translation with Gemini.**
| Setting | Output |
| --- | --- |
| kgvsrc | Fiber eira metko, kon tama Dian taraunkin? Kon kerkapten. |
| eng tgt | There are two fibre boats there, which one is Dian’s grandfather’s? The red one. |
| 0-shot | The fiber is wet, is it Dian’s fault? It’s broken. |
| 5*-shot | Is there a fibre boat there, where is Dian’s grandfather? It’s getting dark.              (1) |
| $\textsc{Book}_{all}$ | The fiber boat is there, which one is Dian’s grandfather’s? The one that is red. |
| $\textsc{Book}_{p}$ | Those two fiber boats are over there, which one is Dian’s grandfather’s? The red one. |
| $\textsc{Book}_{\neg p}$ | Is there fiber over there, which one is Dian’s? It’s the one with the cover. |
| kgvsrc | Wa me kariak kaia kon untuk emumur kalo tumun amkeiret mu wara nanet. |
| eng tgt | This is blood medicine for mothers, when they have given birth they drink this. |
| 0-shot | You should be careful when you go out at night. |
| 5*-shot | This one is a kind of food for women if a child is born they eat it.                       (2) |
| $\textsc{Book}_{all}$ | This is the blood of a woman who has given birth to a child. |
| $\textsc{Book}_{p}$ | This is a blood sacrifice for the women so that the children will be born. |
| $\textsc{Book}_{\neg p}$ | This is a story about a child who was born on the island. |
| kgvsrc | Desili koyet an koi panggutki tenaunat tawara. |
| eng tgt | After planing, I use the panggut to chop the keel. |
| 0-shot | The young man is very strong. |
| 5*-shot | Desili is diligent and she is planting cassava.                                                (3) |
| $\textsc{Book}_{all}$ | After I planed, I went to cut the keel. |
| $\textsc{Book}_{p}$ | After planing, I again cut the keel. |
| $\textsc{Book}_{\neg p}$ | Desi is already here and she is asking for a drink. |
| kgvsrc | Kalamang esun mu walorkawat reitkon. |
| eng tgt | Kalamang’s father’s family have one hundred coconut leaf stems. |
| 0-shot | The Kalamang people are very strong. |
| 5*-shot | Kalamang’s father is waiting for the hundredth day.                                       (4) |
| $\textsc{Book}_{all}$ | The Kalamang people are fishing for lobsters. |
| $\textsc{Book}_{p}$ | The Kalamang man has many coconut leaves. |
| $\textsc{Book}_{\neg p}$ | The Kalamang people have one hundred walorkawat. |
| kgvsrc | Ma kaleuna ning. |
| eng tgt | He has a kidney-disease. |
| 0-shot | You are very kind. |
| 5*-shot | He is sick.                                                                                        (5) |
| $\textsc{Book}_{all}$ | He/She is not sick. |
| $\textsc{Book}_{p}$ | He is sick. |
| $\textsc{Book}_{\neg p}$ | He/She is not going. |
| kgvsrc | Os Kamburkadok me kahen. |
| eng tgt | The beach at Kambur is long. |
| 0-shot | The Kamburkadok are coming. |
| 5*-shot | The one on the beach is tall.                                                                   (6) |
| $\textsc{Book}_{all}$ | The sand on Kamburkadok is far away. |
| $\textsc{Book}_{p}$ | The sand on the beach is far away. |
| $\textsc{Book}_{\neg p}$ | The Kamburkadok is far away. |
| kgvsrc | Camat mu lukta, in arep neko komahal. |
| eng tgt | Camat and family came, we in the bay didn’t know. |
| 0-shot | The chief is sick, and the people are worried. |
| 5*-shot | They came first, we don’t know where they went.                                          (7) |
| $\textsc{Book}_{all}$ | The District Officer came to us, we don’t know what’s inside the bay. |
| $\textsc{Book}_{p}$ | The Camat has come, we don’t know where he is in the bay. |
| $\textsc{Book}_{\neg p}$ | The district officer came here, but he doesn’t know where we are. |

Example 2 shows predictably poor performance in the 0-shot setting. For 5*-shot, the model manages some correct lexical translations but the sentence-level meaning is lost. $\textsc{Book}_{all}$ and $\textsc{Book}_{p}$ get most of the meaning; however they both miss some lexical translation (e.g. ‘sacrifice’ rather than ‘medicine’) and incorrectly predict verb tenses. $\textsc{Book}_{\neg p}$ only correctly translates a few words (including ‘child’ and ‘born’), and generates an unrelated sentence.

In Example 3, 0-shot is again an inadequate translation (despite being fluent). Here, the 5*-shot setting is also off-target in meaning, being unable to find a translation for ‘Desili’ and instead using it as a name. $\textsc{Book}_{\neg p}$ also fails to translate this word, and the output is irrelevant. $\textsc{Book}_{all}$ and $\textsc{Book}_{p}$ predict a similar meaning, close to the target; however, $\textsc{Book}_{p}$ correctly generates the tenses of past continuous ‘planing’ and the present simple ‘cut’, instead of the simple past ‘planed’ and ‘went to cut’. Therefore here the parallel, glossed examples in $\textsc{Book}_{p}$ help to predict correct grammar moreso than the grammatical explanations in $\textsc{Book}_{all}$ and $\textsc{Book}_{\neg p}$.

Example 4 shows a largely irrelevant 0-shot translation, with the correct proper noun. 5*-shot gets the possessive ‘father’, and the meaning of ‘one hundred’, but the overall meaning is lost. The Book settings are similar and get different aspects of lexical and sentence-level meaning correct. $\textsc{Book}_{all}$ is the worst among them, predicting an inadequate output. $\textsc{Book}_{\neg p}$ correctly translates the meaning of ‘one hundred’, but fails to translate ‘walorkawat’; and $\textsc{Book}_{p}$ is the only setting to mostly correctly translate ’coconut leaves’, but misses the meaning of ‘father’s family’ and ‘one hundred’.

In Example 5, 0-shot is completely wrong. 5*-shot and $\textsc{Book}_{p}$ outputs are identical, and close to the meaning but lack the reference’s specificity. $\textsc{Book}_{all}$ and $\textsc{Book}_{\neg p}$ however both predict negation, and output a grammar book-style sentence showing the indeterminate gender of the pronoun with ‘He/She’, which is penalised against the reference; $\textsc{Book}_{\neg p}$ also misses the meaning of sickness.

Example 6 again illustrates the largely inadequate 0-shot performance. Here, the 5*-shot setting is fairly lexically accurate with ‘beach’ and ‘tall’ (against ‘long’), but misses the sentence-level meaning. $\textsc{Book}_{\neg p}$ fails to translate ‘beach’ but gets some of the meaning; while $\textsc{Book}_{all}$ and $\textsc{Book}_{p}$ both get some aspects correct: the former keeps the beach’s name but misses the word ‘beach’, and the latter misses the name but predicts ‘beach’.

Finally, in Example 7 we see another failure of the 0-shot setting. With 5*-shot, the model gets the verbs correct but misses some vocabulary (i.e. ‘the bay’) and the general meaning. Both $\textsc{Book}_{all}$ and $\textsc{Book}_{\neg p}$ predict ‘District Officer’ for ‘Camat’, which is the Indonesian translation, while the reference denotes this as a given name, showing the model relying on previously observed but unrelated vocabulary when lacking a translation in the prompt. $\textsc{Book}_{p}$ is closer to the reference meaning for the first clause, though with the present perfect ‘has come’ instead of the simple past ‘came’, and misses some meaning in the second clause. $\textsc{Book}_{all}$ is further away from the reference in the second clause referring to ‘what’ rather than ‘Camat/him’, and $\textsc{Book}_{\neg p}$ misses ‘the bay’ and has an incorrect subject for ‘know’.

In summary, 0-shot is predictably irrelevant but fluent; 5*-shot tends to give correct lexical translations with incorrect sentence-level meaning; $\textsc{Book}_{\neg p}$ predicts some higher-level meaning but lacks lexical translation adequacy; and $\textsc{Book}_{all}$ and $\textsc{Book}_{p}$ produce the best translations which are largely complete at the sentence-level, with $\textsc{Book}_{p}$ sometimes generating more precise grammar and lexical translations.

## Appendix I Limitations

In addition to those noted in the main paper, we acknowledge the following limitations of this work. While we combine the Kalamang test sets to give a 100 example set, this is still far below a standard test set for MT, often 1-2k sentences. In Kalamang, we are limited by the availability of additional data. However we do test Nepali with the Flores devtest set of 1012 examples which provides a more realistic low-resource translation setting. Nepali experiments also address generalisation issues of focusing only on one XLR language. Regarding evaluation, we note that many differences in ChrF++ score were fairly small, and as reported in Kocmi et al. () a difference in ChrF (note, not ChrF++) of 3.05 is required for more than 90% of humans to agree that a system is better than another in practice; this emphasises the need for future experiments and qualitative analyses (see Appendix H for a small scale qualitative analysis).

Further, the majority of our translation experiments are run on Gemini-1.5-Flash, an API-only LLM. Given the nature of our long-context experiments, we are necessarily limited in our choice of model—at the time of running experiments and to our knowledge, no other model family can handle context lengths over 200k tokens which is necessary for the entire Kalamang book. We run selected short-context experiments with the open-weight Llama-3.1-8B model to improve the generalisation of our results, and we leave tests with other long-context models to future work. We finally note that while ideally we would have a larger kgv test set, running long-context inference of paid API models for >1k examples becomes prohibitively expensive. This limitation applies to the entire method of long-context LLM prompting, justifying the fine-tuning of smaller, open-weight, local models for XLR translation instead - especially for members of these language communities who are unlikely to have access to large API models but may have access to free GPUs through services such as Google Colab(^10^1010[https://colab.research.google.com/](https://colab.research.google.com/)) and Kaggle(^11^1111[https://www.kaggle.com/code](https://www.kaggle.com/code)).

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: * would you expect different conclusions if the prompted / tuned LMs were much smaller?, what about larger models?
soundness: 4
strengths: * wide coverage of materials, models and approaches
* presented clearly (except for the complexity of results in Table 2) and convincingly
summary: The study presents a very thorough revision of the claim that prompting LLMs with grammar books of an unseen language enables the model to translate from/into that language. The main claim is that actually the translation examples in grammar books are the key support of handling the language, while grammatical explanations do not bring any improvement. For that purpose the work uses grammatical books for Kalamang and Nepali and presents contrastive experiments with several language models (Gemini, Llama 3.1, other more specific models), covering prompting, fine-tuning and LoRA. Besides splitting the grammar books into parallel examples and everything else the authors also include a list of words and other supplementary material. In addition to the task of translating, the grammatically judgement is also included, where grammatical descriptions are useful unlike with translation.

Although the study is narrowly targeted, it presents very clearly and exhaustively verified conclusions on the capability of LLMs, showing once again that their behavior is more akin to elaborate pattern matching, rather than comprehension and thinking.
weaknesses: * minor weakness: there are so many different models and comparisons that Table 2 is hard to understand, the reader has to go back and forth between the table and section 4.1 to remind, what the abbreviations mean -- perhaps there is a way to improve the readability?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Please refer to the weaknesses.
soundness: 4
strengths: 1.  The paper showed a novel study that whether grammar books alone support translation for extremely low-resource (XLR) languages or not. By isolating grammar book components (e.g. parallel sentences vs. grammatical explanations), they illustrated a comprehensive evaluation, complemented by introducing typological prompts for linguistic tasks, a creative addition for XLR NLP.

2. By the controlled experiments across different closed-source (Gemini), open-source LLMs (Llama 3.2 8b), and NLLB NMT model, They found out that LLMs can't effectively learn from grammatical explanations for XLR MT in Kalamang and Nepali, relying instead on the parallel sentences. Also, We showed that fine-tuning NLLB model matches the performance of LLMs with long context. Additionally, LLMs can utilize grammatical information effectively when applied to suitable tasks, e.g. grammaticality judgment or IGT prediction, especially when enhanced with targeted grammatical data like our typological prompt, which achieves top performance on these linguistic tasks.

3. The paper is well-written, coherent with comprehensive analysis.
summary: This paper evaluated whether LLMs can translate extremely low-resource languages using a grammar book. The findings reveal that parallel sentence examples are crucial for translation, while grammatical explanations offer little benefit. Tests on English-Kalamang and Nepali confirm that translation quality improves with parallel data rather than grammatical descriptions. For linguistic tasks like grammaticality judgment, a typological feature prompt is more helpful. The authors recommend focusing on parallel data collection over grammar descriptions for effective translation in low-resource languages.
weaknesses: 1. While the paper effectively examines the Kalamang and Nepali languages, its findings may not generalize across the diversity of low-resource languages. XLR languages can vary significantly in structure, script, and available resources, and focusing only on these two could limit the applicability of the conclusions. It would be nice to do similar analysis on low-resource languages too.

2. The paper introduces typological prompts as a promising way to improve linguistic tasks but does not compare this approach against other techniques that incorporate linguistic typology (e.g., embeddings based on typological features from databases like [WALS](https://arxiv.org/abs/2010.03920))

---

confidence: 4
contribution: 2
initial_rating: 8
presentation: 4
questions: 1. Line 44: how does the quantity of data available make Kalamang not unique ? I would better explain why the findings on Kalamang are relevant to many other languages
2. I would discuss the fact that you are using in-context learning (ICL) more clearly in the intro or even abstract
3. I would add more references to explain/justify the methodology in 3.3 and 3.5
4. Line 215: parallel data annotation is typically very rigorous; annotators have to follow specific instructions or for scraping, sentence alignment has to be in a particular way or else the sample is thrown out. How clean & consistent are the parallel samples that come from your heuristic filter ? How did you determine what was appropriately parallel examples vs an explanation when no Kalamang speakers were involved in the project ? I would add more details about this heuristic filtering process.
5. Notation for tables is confusing and takes a while to digest. It might be worth including quick natural language names like instead of Book_p, putting "Grammar Book, parallel subset"
6. Table2: how is Gemini able to do 11&13 ChrF when it has never seen Kalamang before ?
soundness: 4
strengths: 1. The paper is very well organized and is very clear to follow
2. The paper properly investigates a research question, with few, if any, holes in their experimental design as far as I can find
3. The presentation of the experiments and results are clear and the findings align logically with the numerical results
4. The authors incorporate novel methods (as well as new datasets) to understand the importance of grammatical explanations for low-resource MT
summary: This paper examines the contribution of a grammar book to building a machine translation system for an extremely low-resource language, Kalamang, that LLMs cannot have seen in training. In Kalamang, there is also parallel sentences and a dictionary. The authors evaluate LLMs by placing different mixtures of these language resources into a long context and evaluate the model's ability for in-context learning with the goal of translating test samples to and from English. They find that the additional grammatical explanations contribute little, if anything, on top of the parallel sentences, which themselves help the LLM learn to translate Kalamang. They similarly find that for Nepali, a much higher-resource language (but still low) where models have likely seen some text for, parallel sentences contribute to improving MT but not the grammar book. They also show this when finetuning NLLB, a translation-specific model, and show that grammar is also token inefficient in ICL in comparison to parallel sentences.
weaknesses: 1. In general, the biggest weakness of the paper is the significance of its findings:

a. It is already established that parallel data, the most directly related samples to the MT task, are the most effective to training MT models. In many languages, parallel data is in fact the only form of labeled data that exists. For Nepali and Guarani, for example, it feels obvious that a grammar book would not help given that Gemini and Llama have likely implicitly seen many Nepali and Guarani texts in pretraining (or other highly related Indo-Aryan languages with similar grammar).

b. I understand that, by nature of targeting extremely low-resource languages, there are very few available data sources for such an experiment. But generalizing these results past Kalamang seems questionable, especially given that there is a singular grammar book, presumably written by a very small number of people. If possible, this paper would be strengthened by a few more languages (more similar in task than Nepali). I would also suggest more context on grammar books themselves be provided and how frequent these are for extremely low-resource languages.

c. Title is a bit misleading in that it asks whether using grammar books are useful, all to prove that not really. The title moreso implies that the paper would show that grammar books are useful.

2. In comparison to ACL venues, ICLR attendees will typically have less linguistics background. I am concerned the discussion of more specific linguistics topics (e.g. interlinear gloss text, grammar & typology) might be not well suited for this audience. I will leave that to the AC to discern, but also advise authors modify the Related Work & Methodology section to take that into account.

Meta-review: This is a nice paper that challenges recent claims about LLMs' ability to learn translation from grammar books alone, using as the use case Kalamang (extremely low-resource, XLR) and Nepali (low-resource but seen). Through experiments with different models (Gemini, Llama, and a fine-tuned NLLB MT system), the authors show that parallel sentence examples from the text  rather than the book’s grammatical explanations drive translation performance. The authors also check whether prompting the model with typological information about an XLR language can help in linguistic tasks, and find improvements inconsistent, casting more doubt on LLMs’ ability to effectively use grammatical information for machine translation. 
Reviewer DQQ1 had some concerns regarding limited generalizability beyond the studied languages, particularly since only Kalamang is truly unseen XLR language. The authors address this in part by providing additional experimental results from Guarani. While reviewer DQQ1 maintains some reservations about the paper’s contribution, both reviewers nVmi and 5eBi strongly support acceptance. 

Overall, despite the concerns about generalizability, I believe this work provides a useful data point w.r.t previous claims from Tanzer et al. (2024). The paper presents strong empirical evidence that parallel examples, not grammatical explanations, drive translation performance and it gives clear practical guidance on how to allocate resources for XLR language tasks. I recommend acceptance.

Decision: Accept (Spotlight)