An Incomplete Loop: Instruction Inference, Instruction Following, and In-Context Learning in Language Models
Authors: Emmy Liu
Graham Neubig
Jacob Andreas
Venue: COLM (2024)
Keywords: inductive reasoning,deductive reasoning,abductive reasoning,self-guided reasoning,hypothesis proposal
TL;DR: LMs can use self-hypothesized instructions to tackle problems and differentiate reasoning types. However they may identify correct rules but not always apply them accordingly, or vice versa.

Abstract: Modern language models (LMs) can learn to perform new tasks in different ways: in instruction following, the target task is described explicitly in natural language; in few-shot prompting, the task is specified implicitly with a small number of examples; in instruction inference, LMs are presented with in-context examples and are then prompted to generate a natural language task description before making predictions. Each of these procedures may be thought of as invoking a different form of reasoning: instruction following involves deductive reasoning, few-shot prompting involves inductive reasoning, and instruction inference is abductive reasoning. How do these different capabilities relate? Across four LMs (from the gpt and llama families) and two learning problems (involving arithmetic functions and machine translation) we find a strong dissociation between the different types of reasoning: LMs can sometimes learn effectively from few-shot prompts even when they are unable to explain their own prediction rules; conversely, they sometimes infer useful task descriptions while completely failing to learn from human-generated descriptions of the same task. Our results highlight the non-systematic nature of reasoning even in some of today's largest LMs, and underscore the fact that very different learning mechanisms may be invoked by seemingly similar prompting procedures.

Published as a conference paper at COLM 2024
Instruction Inference, Instruction Following, and In-context Learning in Language Models
An Incomplete Loop:
Emmy Liu & Graham Neubig
Jacob Andreas
Language Technologies Institute Carnegie Mellon University emmy@cmu. edu, gneubigêcs, cmu edu
CSAIL Massachusetts Institute of Technology jdaêmit.
Abstract
Modern language models (LMs) can learn to perform new tasks different ways: in instruction following, the target task is described explicitly in nat- ural language; in few-shot prompting, the task is specified implicitly with a small number of examples; in instruction inference, LMs presented with in-context examples and are then prompted to generate a natural language task description before making predictions. Each of these procedures may be thought of as invoking different form of reasoning: instruction follow- ing involves deductive reasoning, few-shot prompting involves inductive reasoning, and instruction inference involves abductive reasoning. How do these different capabilities relate? Across four LMs (from the gpt and 1lama families) and two learning problems (involving arithmetic functions and machine translation) we find strong dissociation between the different types of reasoning: LMs can sometimes learn effectively from few-shot prompts even when they are unable to explain their own prediction rules; conversely, they sometimes infer useful task descriptions while completely failing to learn from human-generated descriptions of the same task. Our results highlight the non-systematic nature of reasoning even in some of today's largest LMs, and underscore the fact that very different learning mechanisms may be invoked by seemingly similar prompting procedures
Rrpothosie
1 Introduction
In-Gonbart Exarples
Hypothesen y- Sx
Inpak Galput
B-N
Suppose friend is teaching you cook. You watch them place pan on the stove and heat olive oil low heat, adding minced garlic and chili flakes to the olive oil once gets hot. Later, you decide to make the recipe yourself, but you are out of olive oil. You hypothesize that the olive oil served to cook the garlic without burning it, and so substitute butter for olive oil, as it should fulfill the same function. Here, you learned a generalizable cooking procedure by reasoning Abductively-findii that best explains your experience (Frank- furt 1958 Peirce, 1965). This is only one way of learning: the triend could have in- stead provided recipe, from which you could have reasoned deductively about how to apply your kitchen. you had remembered other times when you put but-
you 30
B-S
idbetirou
thescisatt Impuc:
12/tuol
Af,, Hypothmata Selection
Output:-12
Prediction
Figure 1: Diagram of abductive reasoning for an LM. Red arrows show data flow in induc- tive reasoning (few-shot prompting), while blue arrows show data flow in deductive rea- soning (instruction following). Black arrows indicate data flow unique to abductive reason- ing (instruction induction). Instruction infer- ence generally improves on few-shot prompt- ing and zero-shot chain of thought. However, success at inductive reasoning and success at instruction inference are not related.
the rule
!Code and data, including generated hypotheses may be found at nightingal3/rule induction
1

---

Published as a conference paper at COLM 2024
Reasoning type
Analogue
Citations
Deductive
Instruction following
Wei ct al.
2022a
Sanh et
2022
Inductive
In-context learning
Brown et al.
2020
Abductive
thain-of-thought (one hypothesis)
Wei al
2022b
Kojima al
2022
Abductive
Externalyy aided instruction inference (many hypotheses)
Qiu
Wang
2024
Zhu et al
2022
Yang et al. 2024
Abductive
IM-guided instraction inference (many hypotheses)
This paper
Table 1: Summary of reasoning types and analogues in language models. Citations to instruction following, in-context learning, and of-thoht- are limited to the original paper due to the high number of papers on these topics, while we have tried to list all currently available papers on instruction inference (for abductive reasoning) with LMs.
ter on hot pan before cooking, without explicitly reasoning about why, you could have cooked the same meal inductively.
Each form of reasoning has a close analogue in current procedures for steering language models (LMs). In order to induce LMs to perform explicit commands (instruction following: Wei et al. 2022a Sanh et al. 2022), or collection of examples from the task of interest (few-shot prompting; Brown et al. 2020); recently, several methods have been proposed that prompt LMs to generate textual commands from examples before conditioning on commands during prediction (instruction inference; Andreas et al.) 2018: Honovich et al. 2023). But it is often unclear when to prefer one of these procedures over the other, and more generally how these different capabilities relate in current LMs. Does the ability to learn effectively from few-shot prompts imply the ability to perform instruction inference, or vice-versa? Are there tasks that can be learned from few-shot prompts, but not instructions? In this paper, we examine these questions through two tasks: learning numerical functions (4.1) and translation models (.4.2 two questions:
tasks, may condition them
We ask
RO1 When does (abductive) instruction inference improve LM performance over ordi- nary (inductive) few-shot prompting?
Finding:
Instruction inference improves over few-shot prompting in simple cases (linear function learning and simple artificial language learning), but suffers from in- correct hypotheses in more complex case (low-resource machine translation).
RQ2 How does the ability to learn from instructions (deductively) relate to the ability to learn from in-context examples (inductively abductively)?
Finding:
The ability to learn through abduction (proposing hypotheses) is generally not related to learning through induction (few-shot learning). Deductive rea- soning is generally strong, with large performance gains over using inductive reasoning alone when the provided hypothesis is correct.
Our results highlight the diverse capabilities (and diversity of different reasoning mech- anisms) triggered by different prompts and examples in current LMs; future work may investigate how these reasoning types can be combined or made consistent to enhance problem solving in LMs.
2 Three Types of Reasoning in Language Models
We first give a formal definition of instruction following, in-context learning, and instruc- tion inference, relating these processes to deductive reasoning, inductive reasoning, and abductive reasoning respectively. (Other ways of interacting with LMs may also evoke one or more of these forms of reasoning, but we focus on important and widely used prompting strategies.) Figure Ilives schematic of the reasoning loop, while Table giives examples of
2

---

Published as a conference paper at COLM 2024
work falling in each category Throughout this section, assume that we have an input x, and we want to produce an output y with an autoregressive LM conditioned on some additional piece of data D that specifies the target task: PLM(Y x, D). We use in-context learning of linear functions as a running example.
2.1 Instruction Following
In instruction following, we want to map each input x to a according to general instruc- tion or prediction rule D that is specified in the input. This may be viewed as a kind of deductive reasoning, which in begins with one more premises, and applies logically valid rules to reach conclusion (Shapiro & Kouri Kissel 2024). For instance, the instruction D might name a general function, such as Apply this function to the imput: y = 5x followed by specific query: Imput: -3. This is illustrated in Figure
Few-Shot Prompting
Few-shot prompting, by contrast, specifies the target task implicitly through examples. For each input the task specification D consists of a set of k training examples D = We then sample from PLM (y x, D). Few-shot learning requires LMs to perform inductive reasoning (Hawthorne) 2021). Unlike deductive reasoning, there is no explicit premise stated, but the model must complete the task in a similar way to the examples. In Figure this is pairs of inputs and outputs, i.e. Imput: 5, Output: 28. These examples help specity the task as a numeric prediction task, as well as the identity of the specific target function.
2.3 Instruction Inference
Instruction inference connects instruction following and in-context learning: given examples yx), ) and input x, we can instruct the model to generate a hypothesis h about the identity of the task, i.e. instruction describing the task associated with D. We may sample one hypothesis and immediately condition on it (a form of chain-of-thoughtt prompting), or sample several and select the most promising one.
D (
Compared to chain-of-thought, fewer studies have explored multi-hypothesis instruction induction; those that do typically rely on an external validation model rather than using the LM itself to evaluate. Our approach (in $3) has the following high-level form: after sampling n hypotheses h,,,,,, h, from PLM x), we evaluate each hypothesis by assigning score
score !) based on the hypothesis and in-context examples. We choose the best one
h+ arg max score(h;; D), and feed it back into the context as instruction
Abductive reasoning is often called "inference to the best explanation" (Lipton 2001 Dou- ven 2021). Suppose that the model generates the hypotheses shown in Figure We may parse hypotheses and apply them back to the in-context examples, then ranking them by prediction error. Then the model simply has to follow the instruction in Section 2.1
3 Methods
We describe the implementation of instruction inference with multiple hypotheses in this section. As the other settings (few-shot, zero-shot hfain-of-thought etc) are already well un- derstood, do not explain them further, but include the exact prompts used to implement methods in Appendix
and
We note that there is some disagreement in the philosophy literature about the exact distinction about inductive and abductive reasoning. Here we commonly-cited definitions, which happen to distinguish in-context learning from the more explicit process of verbalizing and testing hypotheses.
SSeveaal recent papers have recently proposed similar processes, but called this inductive reasoning. We believe that abductive reasoning may be a more apt term in any process that includes hypothesis evaluation and selection. See Section 6 for more discussion.
3

---

Published as a conference paper at COLM 2024
Domain
In-context examples (D,)
Query ample (xi)
Expected (j:)
Hypothesis example (h)
Functions
(-10, -213), (9, 167), (4,67),..
15
287
f(x) 20x 13
Colours
(lug dax, blue green), (lug zup, blue yellow), (lug bluf, blue blue), ...
lug walm
blue blue blue green
lug + blue
dax bluf
green
Kalamang vocab (ek)
is their place, Tompat muin), (I'm getting pandandus, want to make a mat.)...
Sakina pouching
Sakina sarimara lawat.
guava sarim
guavas.
Kalamang grammar
subject
verb: SV
(ek)
Table 2: Examples of in-context examples, queries, and hypotheses in each domain.
Instruction inference [Abduction with many hypotheses] After generating и hypotheses H (II,,.., h,,,, , explore methods for reranking them. For all experiments, r 5. Gen- erally, these reranking methods capture "fit" to training data. External validator reranking was used only in the functions domain (see Section 4.2). The other reranking methods used a language model. Given scores of each hypothesis, we choose:
h² mag max score D;).
licH
Settings with instruction inference are referred to collectively as instruction.inferemee We detail the score functions below:
Verbalized confidence We directly prompt the model to estimate the probability of the hypothesis given Dj. scoreverbal (h; D;) is set to the model's probability estimation.
P(data) We use separate LM, to generate log probabilities for in-context examples given the hypothesis. scorep(data) (h; D;) is the sum of log-probabilitenn of tokens of D,-
P(answer) This is similar to P (ddata) and uses the same template, but scorep/answen) (hi, Dj) only sums prgpabbbabilities of tokens in answers yj in the in-context examples.
External validator For structured hypotheses, it is possible to parse them and apply them back to in-context examples, with the score as the negative error. may not be possible to parse all hypotheses, if they are in natural language or inconsistent formats.
Domains and Evaluation
We investigate LM behavior in three domains: linear function inference, an artificial lan- guage learning task, and vocabulary + typological feature learning in the Kalamang lan- guage. We refer the underlying task we would like to improve (output prediction, translation) as the base task, and the task of inferring explicit natural language hypoth- esis from few-shot examples as the abductive task. Examples of each task are in Table We also evaluate hypotheses themselves in each domain. For the exact prompts used in each domain, refer to Appendix family models (gpt-3. 5-turbo; Brown et al. 2020 gpt-4-turbo OpenAI et al. 2024) and
B and Models used across all domains consisted of gpt
1lama family models (11 1alaama--7---tt llama- 2-7 700b-chat, Touvron
T et al. 2023
4.1 Linear Functions
Following investigations of language models' in-context learning abilities (Garg et al. 2022 Akyürek al. 2023), we construct dataset of 40 linear functions f(x) ax + b, where a and b are uniformly sampled from the integers 2- 20), along with 5 test examples for each function, yielding 200 test examples. For each test example, we randomly generate
was used the logprobs-generating LM for all models.
4

---

Published as a conference paper at COLM 2024
in-context examples of the function (xi, y:) and one test example. The test example and inputs for in-context examples are also uniformly sampled from the integers refer to this as the functions domain. Prompts used are given in Appendix 200 We
Hypotheses In this domain, hypotheses are proposed using all in-context examples for each test example. The model was presented with functions (and instructed to write them) in the form y ax + b. For external validator reranking, we parse the generated hypothesis, and score by its negative mean-squared error (MSE) when applied to the in-context examples If (xx) represent executing the parsed hypothesis represented by h; on example Xik:
$ground Srruhh (ht D,) E -(xx)))(((x))))
Evaluation To evaluate the base task, we use 0-1 accuracy, well as median squared errors. To evaluate hypotheses, we examine the accuracy of mooel-proposed a and b coefficients, as well as the Spearman correlation between proposed coefficients and the ground truth.
4.2 Simple Artificial Languages
Inspired by compositional instruction-following datasets, we generate simple dataset where the inputs are nonce words such as lug, and the outputs are colour terms such as blue (Lake & Baroni 2018: Lake et al.) 2019). We call the expanded dataset the colours domain. The ruleset for this domain can be found in Appendix We generate 200 test examples and 800 training examples Prompts used are in Appendix F
Hypotheses Following the original miniscan, we create fixed minimal set of in-context examples that contains each nonce word at least once, and from which the meaning of each nonce word can be reasonably inferred. During instruction inference, we isolate word at a time from the sentence, and have the model try to induce the vocabulary mapping for that word from retrieved in-context examples containing that word. The ground truth grammar was written by the author, and consisted of production rules for each nonce term. Prompts used in the abductive and base tasks can be found in Amppenddxx F
nonce
Evaluation We also use 0-1 accuracy to evaluate the base task of nonce word translation, as well as corpus-level chrF (Popović, 2015). To evaluate the quality of the hypotheses themselves, we extract the production rules proposed by models and compare 0-1 accuracy against the ground truth. As lenient evaluation on the "repeat" terms (see Asee/Appendit for nonce terms and meanings), we marked a hypothesis for "repeat" terms as correct if it contained the term repeat or the numerals and respectively.
4.3 Kalamang Translation
For low-resource translation, we use the Machine Translation with One Book (MTOB) dataset, an English-KKllmann dataset with grammar book (Tanzer et al. 2024), Kalamang an extremely low-resource language with fewer than 200 speakers, and virtually no text on the web. The base task is to perform sentence-level translation in both directions, while the abductive task is infer correct grammar features and vocabulary mappings. The dataset consisted of 100 test sentences (50 in each direction) and 400 train sentences. A ground-truth bilingual dictionary was provided (Visser) 22020). A grammar book was included as well, but to check correctness of high-level grammar inferences,
compiled high level grammar
sketch instead from WALS and GramBank features (Dryer & Haspelmath, 2013 Skirgård et al. 2023). More details about the grammar sketch can be found Appendix Kl Otherwise, we use the same experimental settings as the baseline, summarized in Appendix
SIf a model produced unparsable hypothesis (for instance, J don't loww, or generic answer like y = ax h), that hypothesis was assigned score of oo. WW generate this dataset instead of using the original miniscan to mitigate memorization of the original nonce words, well as gain more test examples, as the original dataset has only 10.
5

---

Published as a conference paper at COLM 2024
Domain
Deductive reasoning works on average
Abductive reasoning works on average
Hypothesis proposal works on average
Abductive reasoning related to inductive reasoning
Functions
X
Colours
X
Kalamang
X
X
Table
Summary of results. A checkmark indicates that the property held for all or almost all language models, a half-checkmark indicates partial success for all almost all language models, while an X-mark represents lack of success for most language models.
Hypotheses We split hypotheses into vocabulary and grammar. The ground truth instruc- pe tion included retrieved examples from the wordlist and the grammar sketch. To induce the grammar sketch, we posed each grammar feature as question, for instance: What is the order of subject and verb in Kalamang?, and sampled 5 sentence pairs at a time with the requisite parts of speech until the model proposed answer the question. If the model responded that it was unclear, this would repeat up to maximum of 10 iterations. We only performed this process once, for UPT-3.5-t8 5-turbo and GPT-4-turbo respectively. Each model used its self-induced grammar sketch at test time For vocabulary induction, we followed a similar process as in the colours domain
Evaluation We use corpus-level chrF. To evaluate the grammar sketch and vocabulary induction, we respectively compare to the ground truth wordlist and grammar sketch.
5 Results
In this section, we first evaluate models' concrete performance across different domains (RQ1). We highlight the significant improvement instruction inference offers in some cases in synthetic tasks, yet despite this, improvements are not uniform across tasks, and not attested challenging domains like Kalamang. Our analysis also highlights an inconsistent correlation between the quality of generated hypotheses and few-shot learning success RQQ2), meaning that the ability to generate or follow instructions doesn't tt reliably predict task mastery vice versa. These results (Talle 3) suggest that while structured instructions can boost performance in simpler scenarios, their impact is less predictable in complex settings. Furthermore, the relationship between instruction induction, instruction following, and in-context learning is complex, and each capability may rely on separate unknown aspects of model architecture, training procedures, data.
5.1 When Does Instruction Inference Improve Over In-Context Learning?
How effective is using the true instruction? In the domains we study, the ground-truie instruction tends to yield accurate results. Figure 3 displays mean accuracy in the linear functions and colours domains over six trials, sampled at different temperatures (see Ap- pendix A) for details. Notably, in linear functions A), GPT-4-turbo's ' accuracy increases to 96% from a baseline of 30%, with GPT-3. also notably improving. In the colours domain, (3),, we see the true instruction also helps all models except for Llama-2-7b. However, this trend does not extend to the Kalamang task, where most models struggled to leverage the provided wordlist and grammar sketch effectively, indicated by chrF scores in Figure 4
How effective are models' induced instructions? Self-generated instructions also improve the baseline in many cases, with variations by domain. In linear functions, models' hypothesis induction markedly surpasses the few-shot baseline, with both the verbalized
7 "Llama-2 models were found be unable to propose grammar features with our prompts, so used the GPT-3. 5-turbo induced grammar sketch for these models. For Kalamang, due to computational costs, we cache the first parseable hypothesis proposed for each word and reuse it subsequent sentences containing that word.

---

Published as a conference paper at COLM 2024
confidence and log-probability based reranking methods yielding comparable improve- ments (see (aeeFFiguree 3/A again). For the colours domain (3B), instruction inference benefits performance, though not as strongly as in linear functions. chrF scores follow similar trend, and are depicted in Appendix G Interestingly, for gpt-4-turbo and 2amm---7b, using the models' self-proposed hypotheses benefits performance more than using the ground-truti grammar for the colours language, despite the fact that self-proposed are not always correct. Unlike the two synthetic domains, inducing grammar and vocabulary items for Kalamang does not improve translation metrics in most cases.
5.2 How Does the Ability to Induce Instructions Relate to In-Context Learning?
How accurate are model-generated hypotheses? In assessing accuracy of models' self-generated hypotheses across different domains, our findings reveal significant variations in accuracy. In the linear functions domain, we plot hypotheses generated by two models in Figure 2. and list the Spearman p
GPT-3.5-turbo
Predicted coefficietss
20
a coeff
20
b coeff
Spearman I: 27
5 Spearmnn r 0.56
10
10
-10
- 110
20
-20
-20
20
-20
0
20
(Seerrman 1904) as well as p-values of model-predicted coefficients in Table
Rea coefficlents
4 5PT-3.5-turle and GPT-4-turbos proposed coefficients are positively cor- related with the real coefficients, and this statistically significant. How-
GPT-4-turbo
Predicted coefficients
20
coeff
20
coeff
Spearman 0.87
Spearman
ever, this level of accuracy is not ob-
10
10
served with Llama-2 models, indicat-
ing disparity in model capabilities. In the colours domain, most mod-
-10
800
els, except GPT-4-turbo, tend to gen-
-2
-20
0
20
erate inaccurate hypotheses. The ex- act accuracy of proposed hypotheses for colours is shown in Appendix H Mappings of simple vocabulary items such as lug tend to more accurate, with GPT-4-turbo achieving an 87% mean accuracy for hypotheses about this word. On the other hand, the diffi- culty increases with the terms which involved repeats, with only achieving 15% mean accuracy on bluf, the "repeat twice" term.
Rea coefficlents
Figure 2: Real coefficients of linear functions and relationship to hypothesized coefficients for GPT- 3.5-turbo and GPT-4-turbo. Remaining models can be found in in/Appendix D The x-axis has been truncated for visualization purposes (as there are purposes
some large outlier hypotheses). GPT-4-turbo is able to induce a reasonable function in-context, but other models struggle.
However, when extending these anal- yses to Kalamang, all models' perfor- mance in predicting grammar features is relatively poor, with GPT-3. 3.35-tttrb predicting 5/18 and 4/18 features cor- rect respectively. AAppendix L shows the correctness of each grammar fea- ture. Vocabulary induction accuracy also generally low, falling between 10-20% for most models in both the En- glish to Kalamang as well as Kalamang to English directions. See Appenddx Mor details, as well averaged segment-level chrF for the vocabulary hypotheses.
Model
corr.
b p-val
cor
A ap-vval
GPI--.5-turbo
0.27
5.0 10
0.56
2.7 x 10 57
GPT-4-urbo
0.85
2.2 x 10
0.91
0.0
Llama-2-7b
4..069
0.90
0.14
0.0081
Llama-2-70b
0.14
0.060
Table 4: Spearman correlation and p-values of true vs predicted coefficients for each model in the func- tions domain.
Is abductive reasoning related to in-context learning ability? In our final analysis, we examine whether the ability to induce correct instructions correlates with success in in- context learning. Specifically, we focus on the final hypotheses selected by the gpt model family, given that many hypotheses generated by 1 1lama models were incoherent or not
7

---

Published as a conference paper at COLM 2024
Prompt Type
3.6
0.0
few shot
true-instre inteiiitttctionn
0.06
Instruction, inferenceeestemmaa.oo
0,030
0.04
Instruction inference.vertal confidence instruction inferencerpléatall Instruction Infernccu.lansween
0.2
0.025
0.0
0.0
0,000
0.00
gpt 5-rurr
apt Nurbo
Llama 70b chat ht
Uama 2 7b cnat hf
Accurscy (baau toare 80071
0.5
015
Prompt Type
0.8
few-shot cot
0.6
0.6
0:10
ins-iinttuution
04
0.7
instruction in'erence-estermal vel instraction, viferenceeerball confidence instruction interencespddata)
Instraction, lnfonenceepionswer
Figure 3: Accuracy of models in synthetic domains with and without hypothesis generation. Error bars indicate standard error. The top row shows results for the functions domain, while the bottom row shows results for the colours domain. Results are aggregated across runs, and zero values are marked with '0'.
formatted correctly. We use the point-biserial correlation (Pearson, 1895) to assess the relationship between the accuracy of model's final hypothesis and its success in in-context learning.
In linear functions, we examine GPT-3.5 --r-bbb- because of GPT-4-turbo's consistent accuracy in selecting correct hypotheses The analysis reveals a chance-level agreement (0.0013), suggesting GPT-3. NPT-3.5-turbo may be able to predict outputs without identifying the underlying function well, or vice versa. This reveals dissociation between prediction accuracy and instruction induction.
In the colours domain, we examine both gpt models and conduct a similar analysis for each nonce word. We divide the test examples into examples containing each word, and examine the point-biserial correlation between accuracy of induced word meanings and correct translations in the few-shot context. This correlation is generally low, and few p-values are significant after correcting for multiple comparisons with false discovery rate (FDR) (Benjamini & Hochberg 1995). See Appendix H for details.
In Kalamang, we repeat the process of computing point-biserial correlation between vocab- ulary induction correctness with segment-level chrl' in the few-shot translations. Unlike the other domains, there is small positive correlation between correct vocabulary hypotheses and chrF in gpt models. See Appendix M for details. We note that chrF is a more fine- grained measure than accuracy, and the initial scores were low enough that copying some correct vocabulary items may have had slight impact on otherwise completely incorrect translations.
6 Related Work
Hypothesis Proposal with Language Models Recent work explores hypothesis propos- alto improve language model performance in synthetic tasks, namely ACRE, the original MiniSCAN, ListOps, and versions of the ARC dataset (Qiu et al. 22024] Wang et al. 2024). These methods often rely on domain-specific interpreters or code generation by I.Ms, akin to our functions domain's ground-truth truth reranker. We additionally explores probability- based reranking for rule selection across different domains and assesses the accuracy of model-induced rules prior to reranking.
9hhis means agreement cannot be computed with the point-biserial correlation.
8

---

Published as a conference paper at COLM 2024
Language models have also been used to automate hypothesis discovery end in itself, to discover distributional differences in text (Zhong et al., 2022 2023). In this case, the hypothesis proposer LM is paired with validator trained to filter out irrelevant hypotheses. We similarly find that hypotheses generated by language models themselves may not be very accurate inherently.
Abductive Reasoning in Lan- guage Models Datasets for ab- ductive commonsense reasoning
lang ang_direction
(Wang al. 2019 Bhagavatula et al. 20020] Zhang et al.) 22000) and logical reasoning (Sinha al. 2019 Yang et al., 2024) have been proposed. Bhhaavatula et al. (2020) focuses on the ability for model to select the more plausi- ble hypothesis from pairs of hy- potheses, while Yang et al. (2024) focuses on proposing natural lan-
20
Promipt Type
few shat coue-inttruction
Instraction on, inference www.eeaeooollooooeeeoooooooo confidence
Instrart indeencerptoonn
instruction, inferenceelleeeeee
lang,
guage rules and evaluating them
against ground-truth human rules, Zhao et al. (2023) use contrastive explanations tune a model to recognize fluent ones. Compara- tively, we do not focus on com- monsense reasoning, and examine relationships between reasoning types on the same task.
30
20
1.0
924-3. 5-bubo
go6-4-1128 presie
Llama-2/00
Llama-2775
model
Figure 4: chrF scores for Kalamang under different methods, in English to Kalamang direction (top row) and Kalamang to English direction (bottom row)
Program Synthesis and Library Learning The general approach we outline for abductive reason- ing be considered soft form of program synthesis with language models. Library learning, in which a set of reusable tools is learned from examples, is related (Ellis et al.. 2023). A similar approach a similar two-step process to learn a library of rules for an arithmetic domain as well as previous synthetic dataset (Sinha et al. 2019). We find similar gains on synthetic domains, and further examine challenges in applying abductive reasoning in complex domains.
ours using LLMs is proposed by Zhu et al. (2023). who use
LM-generated Instructions Instruction backtranslation is a related concept (Honovich al., 2023 Li et al., 2024), in which an LM generates instructions for textual data. However, it difters that the proposed instructions are not directly used at the same time that they are proposed, and it does not focus generating rules.
7 Conclusion
We have examined the interplay between deductive, inductive, and abductive reasoning in LMs through the tasks of hypothesis proposal, in-context learning, and self-generated instruction following. Across three domains (linear function learning, artificial language translation, and Kalamang translation), we show that instruction inference is able to improve over few-shot prompting in simple synthetic domains, but that the relationship between these types of reasoning is complex, and they may not work together as expected presently when models are solving complex tasks. As abductive reasoning seems to be relatively weaker capability in current language models as compared to instruction following, fu- ture work could develop more advanced mechanisms for natural-languaae hypothesis verification and correction. The use of hypothesis proposal during training remains under- explored, and joint training of models on question answering and hypothesis proposal with enforced consistency may help models display more consistent behaviour. Enhancements in these areas could accelerate progress towards models capable of autonomous learning and self-improvement.

---

Published as a conference paper at COLM 2024
Acknowledgements
Thank you Kiril Gashteovski, Ekin Akyürek, Shuyan Zhou, and Linlu Qiu for helpful discussions on this project. Additional thanks to David Mortensen for suggestions on gram- mar features and linguistic consultation. Thank you to NEC Labs Europe for supporting this project through the Student Research Fellowship program. We also acknowledge the support of the National Sciences and Engineering Research Council of Canada (NSERC) through postgraduate fellowship given to EL (PGSD). We also appreciate support from Intel and the National Science Foundation through grants CCF-2217064 and IIS-2238240.
References
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023.
Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2166-2179, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v1 N18-1197.
1197
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series (Methodological), 57(11:289-33) 1995.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abduc- tive commonsense reasoning. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreviiee httts://wooeeneeeeeeeeee/er/orrs 1y
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan- Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. 42F644-
wewpiiinnnnnnuurinn meurips. cc/p2ppr/2020/
Igor Douven. Abduction. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2021 edition, 2021.
Matthew S. Dryer and Martin Haspelmath (eds.). WALS Online (v2020.3). Zenodo, 2013. doi: 10.5281/2eno007785533 URL httpt:tttoioooooo wrw/ http://......1//11 000552211222dd 7385533
Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sablé-Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 381 381 (20220050), 2023. doi: 10.1098 10.1098/rsta.20220 2022.00000 1098/rsta. 2022. 0050
URL https:://di.orgg..
H. Frankfurt. Peirce's notion of abduction. Journal of Philosophy, 55:593 596, 1958.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022.
10

---

Published as a conference paper at COLM 2024
James Hawthorne. Inductive Logic. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Spring 2021 edition, 2021.
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. In Proceedings of the 61st Anmul Meeting of the Association for Computational Linguistics (ACL), pp. 1935-1952, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systemss, 35:22199- 222119,, 2022.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the
compositional skills of sequence-to--seqeence recurrent networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüsssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2879-2888. PMLR, 2018. URL http //proceedings. mlr. press//v0/lakk1888 html
Brenden M. Lake, Tal Linzen, and Marco Baroni. Human few-shot learning of compositional instructions. In Annual Meeting of the Cognitive Science Society, 2019. URL httpss//api
semanticscholar wttp://ooroo11::0000555505
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In Proceedings of the Twelffh International Conference on Learning Representations (ICLR), 2024.
Peter Lipton. Inference to the Best Explanation. Routledge, 2001.
OpenAT, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamai Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila
11

---

Published as a conference paper at COLM 2024
Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Karl Pearson. Notes regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240-242, 1895.
Charles Sanders Peirce. Collected Papers of Charles Sanders Peirce, Volume 5, volume 5. Harvard University Press, 1965. http: http:://ww..hpp.harrtaa// hww.ar..................m... cattlaaalooooaoooooooooooooooooe phaptaalllogpppppppopppp shnni
9780674138001
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, 2015. Association for Computational Linguistics. doi:
URL
3004
Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnau Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In Proceedings of the Tenth International Conference on Learning Representations (ICLR), 2022. URL 088207
Stewart Shapiro and Teresa Kouri Kissel. Classical Logic. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Spring 2024 edition, 2024.
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4506-4515, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ 10.18653/v1/D19-1111 v1/D19- URL https::/aallnntiooogy.rrrr
1458
Hedvig Skirgård, Hannah J. Haynie, Damián E. Blasi, Harald Hammarström, Jeremy Collins, Jay J. Latarche, Jakob Lesage, Tobias Weber, Alena Bitzlack-Makareviii Sam Passmore, Angela Chira, Luke Maurits, Russell Dinnage, Michael Dunn, Ger Reesink, Ruth Singer, Claire Bowern, Patience Epps, Jane Hill, Outi Vesakoski, Martine Robbeets, Noor Karolin Abbas, Daniel Auer, Nancy A. Bakker, Giulia Barbos, Robert D. Borges, Swintha Danielsen, Luise Dorenbusch, Ella Dorn, John Elliott, Giada Falcone, Jana Fischer, Yustinus Ghanggo Ate, Hannah Gibson, Hans-Philipe Göbel, Jemima A. Goodall, Victoria
12

---

Published as a conference paper at COLM 2024
Gruner, Andrew Harvey, Rebekah Hayes, Leonard Heer, Roberto E. Herrera Miranda, Nataliia Hübler, Biu Saintington-Raneyy. Jessica K. Ivani, Marilen Johns, Erika Just, Eri Kashima, Carolina Kipf, Janina V. Klingenberg, Nikita König, Aikaterina Koti, Richard G. A. Kowalik, Olga Krasnoukhova, Nora L.M. Lindvall, Mandy Lorenzen, Hannah Lutzenberger, Tônia R.A. Martins, Celia Mata German, Suzanne
der Meer, Jaime Montoya Samamé, Michael Müller, Saliha Muradoglu, Kelsey Neely, Johanna Nickel, Miina Norvik, Cheryl Akinyi Oluoch, Jesse Peacock, India O.C. Pearey, Naomi Peck, Stephanie Petit, Sören Pieper, Mariana Poblete, Daniel Prestipino, Linda Raabe, Amna Raja, Janis Reimringer, Sydney C. Rey, Julia Rizaew, Eloisa Ruppert, Kim K. Salmon, Jill Sammet, Rhiannon Schembri, Lars Schlabbach, Frederick W.P. Schmidt, Amalia Skilton, Wikaliler Daniel Smith, Hilário de Sousa, Kristin Sverredal, Daniel Valle, Javier Vera, Judith VoB, Tim Witte, Henry Stephanii Yam, Jingting Ye, Maisie Yong, Tessa Yuditha, Roberto Zariquiey, Robert Forkel, Nicholas Evans, Stephen C. Levinson, Martin Haspel- math, Simon J. Greenhill, Quentin D. Atkinson, and Russell D. Gray, Grambank reveals global patterns in the structural diversity of the world's 's languages. Science Advances, 9, 2023. doi: 10.1126/ sciadvadg61155
Charles Spearman. The proof and measurement of association between two things. American Journal of Psychology, 15:72-101, 1904.
Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi A benchmark for learning to translate new language from one grammar book, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Eline Visser. Kalamang dictionary. Dictionaria, (13):1 2020. URL WRRLhttts://diciiooarii http:///oonnrttiions/ooooo
Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. Does make sense? and why? a pilot study for sense making and explanation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4020-4026, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 /t///////P1/1/9/ URL httpss//acclanthlooo.oorg///////
Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. Hypothesis search: Inductive reasoning with language models. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In Proceedings of the Tenth International Conference Learning Representations (ICLR), 2022a. /tLLhttss://arxi wttpaaawwww..rrrra////aa//////// 20660
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. ohain-of-thought prompting elicits reasoning in large lan- guage models. In Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS), 2022b.
13

---

Published as a conference paper at COLM 2024
Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. Language models as inductive reasoners. Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 209-202, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL httptss///claaathllllgg/o 0r2/2024 2024 eacl-long. 13
Hongming Zhang, Xinran Zhao, and Yangqiu Song. WinoWhy: A deep diagnosis of essential commonsense knowledge for answering Winograd schema challenge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5736-5745, Online, 2020, Association for Computational Linguistics. doi: 20118653/v122220ad0 URL B.KLihttps://acaanthollee
org/ 2020 acl macl-main.56s
Wenting Zhao, Justin Chiu, Claire Cardie, and Alexander Rush. Abductive commonsense
reasoning exploiting mutually exclusive explanations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 14883- 14896, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. wwtl///.....llooooo..ooao 881. URL httpp:///ee..aatooooo..o.//////. acl.
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. Describing differences be- tween text distributions with natural language. In Kamalika Chaudhuri, Stefanie Jegelka, Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 27099-27116.. PMLR, 2022. URL html
https://proceednng..tttt wta.ggs.mmr..eeeee http:///.w....ee/////////o///220
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery of distributional differences via language descriptions. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023.
Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models can learn rules, 2023.
14

---

Published as a conference paper at COLM 2024
Appendix
A
Model Inference Settings
Generation settings, as well as other details, used the three domains are detailed here. Each model was tested 6 times for each setting (few-shot, zs-cot.... at different tempera- tures, and the aggregate results are shown in figures and tables throughout the paper. The ex- act model versions used for gpt models were
5-ttbbooooottt and 4pt-4-1106-previer
When answering questions in the base task, gpt models were tested times each at temperature T = {0, 1 (note that T 0 is nondeterministic in gpt models because of hardware). No max number of tokens was set for the generation. For 1lama models, T= (0.1, 1} was used, also with no max number of tokens.
Functions
When generating hypotheses, we always used the same model as performed the base task. That is to say, 3pt-3.5-ture would generate hypotheses used by 3pt-3.5-turbo and on. Hypotheses were always generated with temperature above 0 to encourage generation of diverse hypotheses. For gpt models, hypotheses were generated with T and for 1lama models, with 7' = 1.0625. The specific value for 11ama models was because 11ama would usually generate the same hypothesis 5 times at T but higher values greatly increased the number of nonsensical and badly formatted hypotheses.
When self-evaluating hypotheses, the verbalized confidence score was generated at T When using the log- rog-probabilities from text-davinci-eee to rerank hypotheses, the model was also set
Colours As in the functions domain, gpt models and 1lama models were respectively tested 3 times each at temperature = {0,1} } and = {0.1, 1] with no max number of tokens.
When generating hypotheses, T = 1 was used for all models.
When self-evaluating hypotheses, settings were the same as in the functions domain.
Kalamang For the base translation task, the same settings were used for generation as in Tanzer et al. (2024). Due to cost constraints, we ran only once in each translation direction on each setting with T 0.05.
Vocabulary hypotheses for all models were generated with T = 1. Grammar feature hypotheses were generated independently of translations, and the first non-null hypothesis was chosen due to cost constraints. T 0.7 was used for gpt grammar hypotheses, and T ==1 1 was used for 1lama grammar hypotheses.
When self-evaluating hypotheses, T was once again used for all models.
15

---

Published as a conference paper at COLM 2024
B Prompts for Linear Functions Domain
Table S: Prompts for the functions domain. Newlines are depicted visually for case reading. Variables that are substituted depending on the question are marked like (this). The wording for the "prompt with self-induced hypothesis" and "zero-shot chain-of chain-of-thought prompt" slightly changed from the few-shot examples prompt because models sometimes confused by long-winded hypotheses, and responded with failures to parse their answers. In comparison, models mostly returned outputs in correct formats in other settings.
answer return, causing
Prompt Type
Usage
Prompt Text
Base system prompt
For reasoning
You are problem solving systen. Your job to use the input-output pairs to solve the problen well can.
with in-context
examples
Hypothesis proposal system
Fur proposing hypotheses based
You are pattern recognition system. Your jub is to cone with function that describes the data well you
prumpt
examples
Instruction following system prompt
applying proposed hypothesis or ground-truth hypothesis the input
are problen solving systen. jab to apply function to the data in order to produce an answer.
Return the output preceded by 'Output:"
Input: (input1}
Output: {outout]}
Few-shot examples prompt
For reasoning with in-context examples only
Input: (input2)
Output: (output2)
Input: (query input]
Function: {The real function]
Examples:
Input: (input1)
Output: {outout
Prompt with ground-truth hypothesis
Used when prompting model directly apply the correct hypothesis to the input. In-context examples also included.
Input: (input2)
Output: (output2)
Return the output preceded by "Output:"
Input: (query input)
Mrite the function that captures the relationship between inputs and outputs.
You should write it in the form y 2x'6 bx"1.
Input: (imput1)
Output: {outout1}
Prompt hypothesis induction
Used model propose single hypothesis for the function based in-context examples
Input: (input2)
Output: (outout2)
Function (please ease write explicitly exact forn" 'Output: y = ax'O bx'1)":
16

---

Published as a conference paper at COLM 2024
reading. Table Prompts functions dumain. Newlines depicted visually Variables that are substituted depending on the question are marked like (this). The wording for the "prompt with self-induced hypothesis" and "zero-hoot chain-of-thought prompt" were slightly changed from the few-shot examples prompt because models were sometimes confused by long-windee hypotheses, and responded with long-winded failures to parse answers. comparison, models mostly returned outputs correct formats in other settings.
return, causing
Use this functian apply to the input exapple to get the correct output.
model's hypothesis
However, just write the output 1ike what's shown in these examples.
Prompt with self-induced hypothesis
Used similarly the "prompt with ground- truth hypothesis", exceptt with self ref--geeerted hypothesis. The wording slightly changed.
Input: (input1)
Output: {output
Input: (input2)
Output: (outoutz)
Return the output preceded by "Output:"
Input: (query input]
Return the output preceded by 'Final Output:""
Input: (input1)
Output: {outout
Input: (imput2)
Prompt for sero-shot chair-of-thoughh thouught
Used to encourage the model to generate chain thought.
Output: (outputz)
Let's think by about what the function be. write "Final Output:
before your Final answer.
Input: (query input]
How likely 15 hypothesis
about the function to be true given the data?
Examples:
Input: (input1)
Output: {output1)
Input: (input2)
Prompt for hypothesis probability estimate
Used to prompt a language model directly for reranking hypotheses
Output: (output2)
Function explanation: [nodel 's hypothesis]
give probability and only answer with number.
inclusive.
Probability:
These are exampees of applying this function: {model's hypothesis]
Examples:
Input: (input1)
Prompt for data logprohs given hyp estimate
Used rerank hypotheses
Output: (output1)
on logprobs.
Input: (input2)
Output: (output2)
17

---

Published as a conference paper at COLM 2024
C Prediction Fit of Other Models on Linear Functions
Figure 5 shows predictions made by each model when using few-shot prompt, true instruction, or self-induced instruction.
few-shot
instruction induction
instruction
200
GPT-3.5-turr 3.5-turbo answer
100
answer
100
100
-100
-100
-100
-200
-300
-400
-400
-400
-400
-200
200
400
4000
200
200
400
-400
-200
0 outputs
200
400
outputs
outputs
300
200
200
200
GPT-4-uurbo
100
100
100
answer
answer
-200
-300
-300
-300
-400
400
-400
-400
-200
200
-400
200
200
600
4400
-200
200
400
outputs
outputs
autputs
400
300
300
300
200
200
Llama-2-7b
answer
100
answer
100
100
100
-200
-200
-300
-300
2000
-400
4000
-300
-400
-200
outputs
outputs
outputs
400
400
400
300
300
300
200
200
Llamma---700 ---00
anower
answer
-100
-100
-100
-200
-200
-200
300
300
330
-400
-400
-400
-400
-200
-400
200
200
400
-200
200
outputs
outputs
Figure 5: Model predictions plotted against true function output for all models. Range is restricted to the [-400, 400] range for visualization purposes, although there are large outlier values for all models.
18

---

Published as a conference paper at COLM 2024
D Predicted Coefficients Compared to Real Coefficients in Linear Functions
Figure plots model hypotheses about the coefficients of x² and x in a linear function against the actual coefficients
A
B
Eptammnn
15
15
Proa coethre
-13
-20
cortorrss
Tacl craficarta
D
20-
SRANTOR TEEE
Heddicoo Do0 corfice
-15
-20
cseffiderss
Figure 6: Model predictions plotted against true function output for all models. Range is restricted to the |-400, 400] range for visualization purposes, although there are large outlier values for all models.
Grammar and Details of Colours Domain
The rules of the colours domain are expressed through production rules below. Models were found to respond better to verbal instructions than formal ones in initial testing (e.g. verbal statement "repeat twice" rather than [[x]]bluf + [[x[[[[]]]
Listing 1: Grammar of the colours language, as presented LMs.
lug -> blue
dax -> green
wif -> red
zup -> yellow
bluf -> repeat the last action twice
walm repeat the last action three times
Training and test data was automatically generated by generating sentences of up to 5 nonce words on the source side, with shorter sentences being more likely (the respective probabilities for sentence lengths from 1 to 5 are [0.4, 0.3, 0.15, 0.1, 0.05)). Each colour could also repeated with the repeat actions, and whether repeat nonce words were inserted was also random, though skewed toward repetition (the probabilities were respectively (0.8,
19

---

Published as a conference paper at COLM 2024
0.1, 0.1] for repeats, one repeat, or two repeats). A repeat term never followed another repeat term, and the same colour word never appeared twice consecutively to make easier to learn the repeat terms.
Additionally, we provided a fixed set of few-shot examples that covered all the nonce terms:
Listing 2: Few-shot examples in the colours translation task
Input:
lug dax
Output: blue green
Input: wif zup
Output: red yellow
Input: lug bluf
Output: blue blue
Input: wif walm
Output: red red red
Input: lug walm dax bluf
Output: blue blue blue green green
Prompts for Colours Domain
Table 6: Prompts for colours clmainn Newlines are depicted visually for case of reading. Variables that are substituted depending on the question are marked like (this). The "prumpt with a self-induced hypothesis" was slightly modified from the base prompt order to encour- age models follow the correct formatting, while the "prompt zero-shot thain-of-thoughh" was slightly modified encourage models generate concrete chain thought, and also follow the correct formatting.
Prompt Type
Usage
Prompt Text
Base system prompt
For reasoning with in-context examples
problem solving systen. input-output pairs to solve the problem as well as you can."*
Hypothesis proposal system
For proposing hypotheses based
You are rule induction systen. Your job is figure out the rules underlying problen and report on then. Use the examples to guide your thinking.
prompt
in-context examples
Instruction following system prompt
For applying pruposed hypothesis or ground-trutt trutt hypothesis to the input
You
Carefully the gramar inputs deternine correct output.
Return the output preceded by 'Output:"
Input: (input1)
Output: {outputh)
Few-shot examples prompt
For resssoning with in-context examples only
Input: (input2)
Output: (output2)
Input: (query input)
Remreber start your answer with "Cutput:"
20

---

Published as a conference paper at COLM 2024
Table Prompts the colours dumain. Newlines depicted visually for of reading Variables that are substituted depending the question marked like (this). "prompt with a self-induced hypothesis" was slightly modified from the base prompt in order to encour- age models to follow the correct formatting, while the "prompt for zero-shot chain-of-thought" slightly modified encourage models generate concrete chain thought, and also follow the correct formatting.
gramer parse input example to get the correct output.
Gramar: {colours domain gramnar)
Examples:
Input: (input1)
Prompt with ground-truth hypothesis
Used when prompting model to directly apply the correct hyttheesis to the imput. In-context examples are also included.
Output: (output1)
Input: (imput2)
Output: (output2)
Return the output preceded by "Output:" Input: (query input]
Prompt for hypothesis induction
Used to have the model propose single hypothesis the translation a word.
The below examples contain the word (word).
Using the examples, deduce what {word} neons.
Input: (input]]
Output: {output]]
Input: (input2}
Output: {output2)
Write your like this: word meaning. Meaning can or general dependent context.
Rule:
Use this gramnar to parse the input example to get the correct output.
nodel's 's hypothesis gramar
However, just write the outout like what's shown in these examples.
Prompt with self-induced hypothesis
Used similarly "prompt ground-truth hypothesis", except with self-generated hypothesis. The wording slightly changed.
Input: (input1)
Output: (output1)
Input: (input2)
Output: (outout2)
Return the output preceded by 'Output:" Input: (query input)
21

---

Published as a conference paper at COLM 2024
Table Prompts the colours dumain. Newlines depicted visually for of reading Variables that are substituted depending the question marked like (this). The "prompt with a self-induced hypothesis" was slightly modified from the base prompt in order to encour- age models to follow the correct formatting, while the "prompt for zero-shot chain-of-thought" slightly modified encourage models generate concrete chain thought, and also follow the correct formatting.
output preceded by 'Final Output:"
Input: (input1)
Output: {output1}
Input: {input2)
Output: (outout2)
Prompt for zero-shot chain-of-thought
Used encourage the model to generate chain thought.
Let's think step by step about what the translation could be. Work through your step step and show your work. 'Final Output:"
before your Final answer.
Input: {query input]
exaples translation the word (word).
Exanples:
Input: (input1)
Output: {output1}
Input: (imput2)
Output: (output2)
Prompt for hypothesis probability estimate
Used to prompt language model directly for reranking hypotheses
Given these examples, how likely is this hypothesis about the neaning of (word)? (nodel's word hypothesis}
Please give probability between and inclusive, and only with number.
Probability:
exapple applying this function: (nadel's 's ward hypothesis)
Examoles:
Input: {input1}
Prompt for data logprobs given hyp estimate
Used to rerank hypotheses based on logprobs.
Output: {outout]}
Input: (input2)
Output: (output2).
22

---

Published as a conference paper at COLM 2024
G chrF for Colours Domain
Figure 7 depicts the mean chrl' score over trials for each model in each setting. Reranking methods are averaged for instruction-induction
100
80
chri
40
method
few-shot
instruction, Instractio_idduction Induction
true-instruction
0
gpt-3.5 Opt-3.5-tunp 5-turbo
gpt-4-turbo
Liama-2 Jlam-2--b-ee 2-7b-chat-hi
Liama 2- 102-70b-chha 70b-hhhtttht
Figure 7: chrF scores for the colours domain under different methods. All reranking methods are averaged.
H Vocabulary Induction Accuracy for Colours Domain
We show in Table 7 the mean accuracy for each colour term for gpt models, computed over all selected (highest-ranked) word hypotheses. Point-biserial correlation between hypothesis correctness (in the Instruction-indcttee shown. P-values are corrected with FDR in the trials for each model. We performed the correction per model rather than across all models because we were separately interested in the behaviour of each model. We parsed the production rule generated by the model in order determine hypothesis correctness.
case) and few-shot correctness is also
23

---

Published as a conference paper at COLM 2024
Table Vocabulary induction correctness and relation few-shot translation correctness for gpt models. Significant correlations after correction for multiple comparisons are bolded.
Model
method
Correctness
Correlation
p-value
p-value
g-t-3.5-turbo lug
verbal..onf
0.60
-0.07
0.30
0.53
gpt-3.5-turbo
dax verbal_conf
0.23
0.067
0.064
0.30
gpt-3.5-turbo
wif
verbal_conf
0.33
0.036
0.65
0.82
gpt-3.5-turbo
zup
verbal.conf
0.61
0.15
0.029
0.20
gpt-3.5-turbo
bluf
verbal conf
0
gpt-3.5-turbo
walm
verbal_conf
gpt-4-turbo
lug
verbal conf
0.88
0.044
0.17
3.58e-1
gpt-4-turbo
dax verbal_conf verbaL.conf
0.92
-0.042
0.19
3.58e-1
gpt-t-tubo
wif
0,86
0.0445
0.57
7,32e-1
gpt-A-turbo
zup verbal conf
0.058
gpt-4-turbo
bluf vert
0.15
0.13
0.26
3.83e-1
gpt-d-turbo
walm verbalcoonf
0.17
0.049
0.69
7.55e-1
gpt-3.5-tutte
lug
p-data
0.29
0.059
0.098
0.30
gpt-3.5-turbo
dax
p-data
0.010
0.78
8pt-3.5-turbo
wif
data
0,20
0.24
0.0016
0.023
gpt-3.5-turbo
zup
p p.data
-0.11
0.10
gpt-3.5-turbo
bluf
p_data
0.077
-0.17
0.14
0.33
gpt-3.5-turbo
walm
p data
gpt-4-turo
lug
p.data
0.73
-0.080
0.012
5.31e-2
gpt-4-turbo
dax
P
3.58e-1
gpt-4-turbo
wif
p_data
0.77
-0.026
0.74
7. 7,55e-1
gpt-4-turbo
zup
P_data
0,76
0.022
0.76
7 7,55e-1
gpt-l-turbo
bluf
p.data
0.10
-0.14
0.22
3.58e-
gpt-4-turbo
walm
data
0.14
-0.26
0.027
9 89c-
3pt-3.--turbo
lug
planswer
0.44
-0.013
0.72
0.84
gpt-3.--turbo
planswer
0.91
gpt-3.5-turbo
wif
p.answer
0.35
-0.074
0.34
0.53
g-t-3.5-turbo
zup
P answer
-0.088
0.21
0.43
gpt-3.5-turbo
bluf
P
0.051
+0.05
0.63
0.82
gpt-3.5-turbo
walm
p.answer
gpt-4-turbo
lug
answer
0.70
-0.066
0.038
1.15e-1
gpt-4-turbo
p paanswer
0.71
3.67e-9
6.61e-8
gpt-4-turbo
wif
panswer
0.82
-0.25
0.0013
1.13e-2
gpt-4-turbo
zup
p paasswrr
0.68
-0.14
0.045
1.15e-1
gpt-4-turbo
bluf
Paanswer
0.23
-0.29
0.0089
5.3ee-2
gpt-4-turbo walm
p.answer
0.057
-0.058
0.63
7.55e-1
24

---

Published as a conference paper at COLM 2024
I MTOB Experimental Settings
We generally used the same experimental setup as in mtob. For the few-shot condition, we used two reference sentences for each word on the source side, selected via longest subsequence. For the true iiveeinnstuuctioo setting, we used reference sentences, in addition to the wordlist and true grammar sketch. For each word in the source sentence, the most similar word from the wordlist was also retrieved based on the longest common substring. In the instruction-infed
settings, the model's self-induced grammar sketch was substituted for the true grammar sketch, and the model was also prompted to first create hypotheses about the translation of each word the source sentence.
At the time that we conducted experiments, the Kalamang to English training set was not available, so we created this training set by reversing the source and target in the English to Kalamang training set. This may have slightly harmed translations in this direction, as this caused the training set to be different from the one reported in the benchmark results.
J MТOB Prompts
Table 8: Prompts for Kalamang translation with MTOB. We follow the same prompt formatting as in Tanzer al. (2024), with the exception of replacing retrieved grammar book passages with the grammar sketch Appendix K] Newlines depicted visually for of reading.
Variables that substituted depending the question marked like (this). The English
Kalamang direction depicted in examples.
indicates that the prompt is the same as
in Appendix
Prompt Type
Usage
Prompt Text
Base system prompt
For reasoning
with in-context examples
Hypothesis proposal system
For propasing grammar feature hypotheses based
prompt (Grammar)
examples
Hypothesis proposal system prompt (Vocab)
proposing
vocabulary
mappings based in-context
examples
Instruction following system prompt
Fur applying proposed hypothesis or ground-truth hypothesis to the input
25

---

Published as a conference paper at COLM 2024
Table Prompts for Kalamang translation with MTOB. We follow same prompt formatting as in Tanzer all [2024) with exception replacing retrieved grammar book passages with the grammar s sketch in Appendix KJ Newlines are depicted visually for ease of reading. Variables that are substituted depending on the question are marked like (this). The English Kalamang direction depicted in the examples. ... indicates that the prompt is the
Appendix
Few-shot
reasoning with in-context examples only
examples prompt
Human: Kalanang is language spaken the Karas Islands in West Papua. Translate the following Englishh to Kalanang: {query input)
To help with translation, here translated sentence with words similar "(word)' list translated Kalanang-English reference sentences:
English sentence: {eng sentence) anann translation: [kgy sentence] <Repeated for all words> New write the translation.
English: {query inout) Kalanang translation on:
Prumpt with ground-truth hypothesis
Used when prompting the model to directly apply the correct hypothesis the imput. In-context examples are also included.
Human: Kalanang is language spaken the Karas Islands in West Papua. Translate the following Englishh Kalanang: {query input)
To help with the translation, here of the closest
entries to "[word]" Kalamang=English bilingual dictionary: English word or phrase: '[word)"
Kalanang translation: "(translation)
Repeated far all words>
To help with the translation, here a translated sentence with words sinilar to '[word)' in list of translated Kelanang-English reference sentences:
English sentence: (eng sentence) Kalanang translation: (kgy sentence)
<epeated for all words>
help translation,
grannar
Kulanang:
(sramar sketch).
Now write the translation. English: {query input]
Kalanong translation:
Prompt for hypothesis induction
Used to have the model propose single hypothesis the translation of a word.
The following sentences contain the word '(word)' " in English.
Exarples:
English sentence: {eng sentence) Kalanang translation: (kgv sentence]
What is the Kalanang translation of the word '(word?'?
Write your answer like this: (English word] >> (Kalanang translation).
26

---

Published as a conference paper at COLM 2024
Table Prompts for Kalamang translation with MTOB. We follow same prompt formatting as in Tanzer all [2024) with exception replacing retrieved grammar book passages with the grammar s sketch in Appendix K] Newlines are depicted visually for ease of reading. Variables that are substituted depending on the question are marked like (this). The English Kalamang direction depicted in the examples. ... indicates that the prompt is the Appendix
Prompt with self-induced hypothesis
Used similarly the "prompt with ground-truth hypothesis", except with gelf-generated hypothesis. The wording slightly changed.
Human: Kalanang is language spaken the Karas Islands in West Papua. Translate the following Englishh Kalanang: {query input)
To help with the translation, here
closest
entries to "[word}" Kalamang=Englishh bilingual dictionary; English word or phrase: "[word)"
Kalanang translation: "(hypathesis translation)"
Repeated far all words>
To help with the translation, here translated sentence with words
similar to '[wordy' in 1ist of translated Kelaneng-English reference sentences:
English sentence: {eng sentence) Kalanang translation: (kgv sentence)
<Rpeated for all words>
help translation,
gramnr
Kulanang:
(hypothesis gramar sketch)
Now write the translation. English: {query inout] Kalanang translation:
Prompt for zero-shot chain-of-thought
Used encourage the model to generate chain thought.
Human: Kalanang is Language spaken the Karas Islands Papua. Translate following from English to Kalanang: {query input)
help translation, here translated sentence sinilar to "(word)' list of translated Kolanang-Eng reference sentences:
English sentence: feng sentence) Kalanang translation: (kgv sentence)
<Repette for all words> Let's think step step about what the translation could be. translation.
inghish: (query input) Kalanang translation:
27

---

Published as a conference paper at COLM 2024
Table Prompts for Kalamang translation with MTOB. We follow same prompt formatting as in Tanzer all (2044 with exception replacing retrieved grammar book passages with the grammar sketch in Appendix KJ Newlines are depicted visually for ease of reading. Variables that are substituted depending on the question are marked like (this). The English Kalamang direction depicted in the examples. ... indicates that the prompt is the Appendix
exapples
translation the word (word).
Exanples: (example translation pairs)
Given these examples, how likely is this hypothesis about the neaning of (wordj? (nodel's word hypothesis)
Prompt hypothesis probubility estimate
to prompt a language model directly fur reranking hypothuses
Please give probability between and inclusive, and only with nunber.
Probability:
These are examples of sentences that contain the word (word): This is the translation of the word: (nodel's word hypothesis)
Prompt logprobs given hyp estimate
rerank hypotheses based on logprobs.
Examples: (example translation pairs)
K
Kalamang Grammar Sketch
Listing 3: Grammar sketch for Kalamang, compiled from WALS and Grambank features.
Start of grammar sketch
Basic Word Order: SV (Subject-Verb), ov (Object-Verb)
Noun Phrase Construction: Postpositional, Genitive-Noun, Noun-Adjective, Noun- Demonstrative, Noun-Num, Possessed-Possesson
Articles: No definite/specific or indefinite articles
Morphological Marking: No productive singular, dual, or plural marking on nouns
Possession marked by suffix on the possessed noun; Tense marked by auxiliary particle
Syntactic Alignment: Accusative
Negation: Standard negation marked clause-finally; Distinct imperative negation Reduplication: Both verbs and nouns can be reduplicated == End of grammar sketch
L Predicted Kalamang Grammar Sketches and Accuracy
Listing Grammar sketch for Kalamang, generated by GPT-3.5-turbo. Each feature was predicted individually based on randomly retrieved examples from the training data.
Start of grammar sketch
Basic Word Order: SV (Subject-Verb), ov (Object-Verb)
Noun Phrase Construction: Adposition-Nont Phrase, Ginitive-Noun, Adjective-Noun Noun-Demonstrative Noun-Num, Possessor-Possessed Articles: Unsure there are definite/specific or indefinite articles Morphological Marking: Unsure if singular, dual, plural marking nouns; Unsure if possession marked by suffix on the possessed noun; Unsure if tense marked by auxiliary particle
28

---

Published as a conference paper at COLM 2024
Syntactic Alignment: Unsure
Negation: Standard negation marked clause-initially; Unsure if distinct imperative negation
Reduplication: Unsure if verbs and nouns can be reduplicated == End of grammar sketch
Listing 5: Grammar sketch for Kalamang, generated by GPT-4-turbo. Each feature was predicted individually based on randomly retrieved examples from the training data.
Start of grammar sketch
Basic Word Order: SV (Subject-Verb), vo (Verb-Object)
Noun Phrase Construction: Unsure on order of adposition and noun phrase, Ginitive-Noun, Adjective-Noun, Unsure on order of noun and demonstrative, Num-Noun, Possessor-Possesseed
Articles: No definite/sppcifii or indefinite articles Morphological Marking: Unsure if productive singular, dual, or plural marking on nouns; Unsure if Possession marked by suffix on the possessed noun; Unsure if tense marked by auxiliary particle
Syntactic Alignment: Unsure
Negation: Standard negation marked clause-initially; Unsure if distinct imperative negation
Reduplication: Unsure if verbs and nouns can be reduplicated
End of grammar sketch
Model
Grammar feature accuracy
GPT-3.5-turbo
27.78%
GPT-44tturbo
22.22%
Llama-2-7B
0%
Llama-2-70B
0%
Table 9: Overall grammar sketch accuracy of models. Unsure answers marked as incorrect.
M Vocabulary Induction Accuracy for MТOB
The overall vocabulary induction accuracy for gpt models is shown in Table 10, Accuracy and chrF were calculated overall when exclude morphology marked No, and morpholog- ically rich words in the dictionary (marked with or with - to represent suffixes /prefixes) were excluded when this value was marked Yes. For words with multiple translations in the dictionary, the translation was marked correct if it matched any of the possible translations, Words without an entry in the dictionary were skipped. a null hypothesis such don't tt know was proposed, it was marked as incorrect.
Due hypotheses for this task being more difficult automatically evaluate, an author also annotated the correctness of results for single setting conf (nnicde order to calculate the association between translation quality and hypothesis correctness more accurately. Results can be found in Table 11
29

---

Published as a conference paper at COLM 2024
Table 10: Accuracy and segment-level chrF for vocabulary hypotheses in Kalamang transla- tion. If morphology was excluded, all words with + or symbol in the translation (corre- sponding to prefixes/ prefixes/sufixes) suffixes) were excluded, whereas otherwise we matched whether or not the model's hypothesis began/ /nded with the correct characters.
model
method
exclude morphology
acc
ck gpt-4-turbo
p. p.data_given weannnivveeeeeeuee hyp-guess
No
0.1528
20.33
gpt-4-turbo
Yes
0.0813
11.86
ek
gpt-4-turbo
p.answer htp.hhhyolooooooo
No
0.16
20.69
gpt-4-turbo
answer given hyp Ingprobs
0.08621
ek
gpt-4-turbo
p.data.giver www.ggiiee.hpp. hypploggpobbs
No
0.166
20.79
gpt-4-turbo
p.data. www.ggivpeeooopeee.....
Yes
0.1015
13.21
ke
8pt-3.5-turbo
p-answer--g ..answer-given.h.e hyven.hypppppppp..... hy..ohhpppplooboe
No
0.1033
15.33
gpt-3.5-turbo
p.answer wirgginiye__ooggege..e_ logprobs
Yes
0.8244
13.11
gpt-3.5-turbo
p_data.give given_hyp-guee
No
0.11
16.42
ke
gpt-3.5-turbo
data .iven. guess
Yes
0.09018
14.33
gpt-3.5-turbo
data given hyp logprobs
12.34
ke
gpt-3.5-turbo
p_data.given_hyy. hyp hhpeehhppooogppp logprobs
No
0.1014
15.28
Llama-2-7Ub
p p.data ddaaa.ivee..i given_hyvepppp
Yes
0.1344
21.69
Llama-2-70b
p-data -iv.ggvvee.yy. hyp-guess
No
0.155
23.62
ck
gpt-3.5-turbo
pudata p_data_given__
Yes
0.0922
15.05
8pt-3.5-turbo
p.data p.vata-given.. sivee._hy-gguee.
No
0.1217
19.29
ke
gpt-4-turbo
answer given hyp logprobs P answer given hyp Jugprobs
Yes
0.06127
13.54
gpt-4-turbo
No
15.25
ke
gpt-4-turbo
p.data-given_hyp htw.hhyyppoolbbee.
Yes
0.07641
12.82
gpt-4-turbo
p.lata. hivven.hyp.logrree
No
0.0924
14.46
Llama-2-70b
answer given hyp Jogprobs
Yes 0.1339
21.92
ke
Llama-2-70b
p.answer hiven_hyp_logrroom
No 0.1447
22.92
Llama-2-70b
d.daa. given_hyp Jogprobs data given hyp logprobs
Yes
21.36
ke
Llama-2-7Ub
No
0.1442
22.38
Model
Correlation
p-val
GPT-3.5-turbo
0.17
0.0063
GPT-4-turbo
0.12
0.051
Table 11: Point-biserial correlation between segment-level chrF of each translation with the correctness of vocabulary hypotheses induced for that question (in the intructionniffen
verbalized conf confidence setting). No correction for multiple com- parisons was done because we examined hypotheses at the model level (see Appendix H for how this was done in the colours domain)
N Example Kalamang Translations
Table shows example translations made by GPT-3.5-turbo in both directions.
30

---

Published as a conference paper at COLM 2024
Table 12: Example translations by the GPT-3.5-turbo model in different settings.
Source
Reference
Translation (few- shot)
Translation (instruction-
Translation (true- instruction)
inductionwerrbalized confidence)
Who you?
asked
Namana kat ger- ket?
mua kiemara kowam?
Naman
Naman era?
Naman mua
kiemara
paruo?
The rope Karek wrapped around kokarap.
mat
Kainasu esun kiwat semda bo dalangdalang.
Sar tok et dalangdalang na.
Karek mambon guanggarien ma me.
Before, they said Japan and The Netherlands war.
Wiseme mu toni Japangbon Be- ladarbon nausair.
Wiseme mu Ja- faris emun Nabil emun kaden kiesun ning.
Wise me mu salai Japan an Sar Be- ladargara din jam- bat mara.
Tikninda, termus siram panci tubak Beladar tamatko pang.
Mu bo They going (on return trip) to bury someone in Antalisa.
They go to Tamisengga and return.
They go to An- talisa, someone died, and they return.
They will return to Tamisengga un- til they bury the person.
Tamisen- gga som
dande yecie.
Suo me kene arkon metko.
At Suo is a kene tree.
washed my body the river.
Back me gives birth canoe.
Wash your face near the Narkon tree.
An istrat kahendengoa
I'm walking on long wide road.
I put the long stick on the ground.
have long dry- ing rack but am walking.
walked far down the street to my uncle.
ba temun.
31


Reviews:
[{"review_id": "GCTgvtQ3FJ", "reviewer": "Reviewer_WMHL", "summary": "The paper investigates how modern large language models (LMs) utilize deductive, inductive, and abductive reasoning to learn tasks. The study explores three learning methodologies: instruction following, where tasks are explicitly described; few-shot prompting, where tasks are implied through examples; and instruction inference, where models generate task descriptions from examples before making predictions. Analyzing four LMs from the GPT and Llama families across tasks involving arithmetic functions and machine translation, they find an interesting dissociation in reasoning capabilities. LMs can effectively learn from few-shot examples without being able to articulate the underlying rules, and conversely, they can generate useful task descriptions yet fail to learn from explicit human-generated instructions. The findings suggest that while instruction inference can enhance performance in simpler synthetic setups, the integration of various reasoning types remains challenging in more complex tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, we're glad you appreciate our contributions and writing! See our responses below:\n\n>  The paper primarily focuses on specific simple tasks, linear function The scope of tasks examined in the study—linear function learning, artificial language translation, and Kalamang translation—is somewhat narrow, limiting the broader applicability of the findings. Future studies could benefit from incorporating a wider array of tasks to strengthen the arguments presented.\n\nThanks for the comment! We agree that there are many potential future applications that are not explored here. As this was a first study exploring the relationship between these types of reasoning in language models, we thought it was important to use clear-cut domains such as the linear functions and colours domains, in which correctness of rules is unambiguous, easing verification. We attempted to show the differences between simple domains and a real-life task by incorporating the Kalamang translation results, in which instruction inference does not provide a large increase in performance, because of the breakdown in grammar rule proposal. We will explore more diverse domains as well as more avenues for improving instruction inference in future work.\n\n> There is room for deeper exploration regarding the limitations of the current methodologies. Discussing potential new domains for future research and outlining prospective directions would provide valuable insights.\n\nThis is a good point! We can certainly add more discussion of potential research directions and applications to the conclusion as part of future work. For instance, although the set of tasks we examined is more related to translation, one could imagine that similar methods could be used to help LMs reason about any domain with underlying rules or principles. For instance, LMs following a similar strategy could be useful to aid researchers in proposing scientific hypotheses to test given a body of existing literature.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-organized, and the results are presented clearly.\n\nThis paper provides a meaningful first step to understanding the three learning methodologies of LLM.", "reasons_to_reject": "The paper primarily focuses on specific simple tasks, linear function The scope of tasks examined in the study—linear function learning, artificial language translation, and Kalamang translation—is somewhat narrow, limiting the broader applicability of the findings. Future studies could benefit from incorporating a wider array of tasks to strengthen the arguments presented.\n\nThere is room for deeper exploration regarding the limitations of the current methodologies. Discussing potential new domains for future research and outlining prospective directions would provide valuable insights.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yV5itEQPEW", "reviewer": "Reviewer_z3T2", "summary": "This paper examines the relationship between instruction following, few-shot prompting, and instruction inference, linking them to deductive reasoning, inductive reasoning, and abductive reasoning, respectively. It explores these concepts across three domains: linear function learning, artificial language translation, and Kalamang translation. The findings suggest that instruction inference can surpass few-shot prompting in simpler tasks; however, its efficacy diminishes in more complex tasks such as low-resource machine translation. Additionally, the paper observes that a model's capability to generate hypotheses is not correlated with its ability to learn from few-shot examples.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments! See below for response:\n\n> While the paper concludes that the ability to generate hypotheses is unrelated to learning from few-shot examples, it fails to suggest future directions or solutions to address this gap. This omission may limit the paper's impact on advancing the field's understanding of integrating these capabilities.\n\n\nWe definitely agree with the need to further investigate the results in more domains and examine the broader applicability of the findings. However, we also emphasize that this is the first paper to examine the relationship between these different avenues of reasoning, and thus we emphasized two controlled domains, linear functions and simple translation, in order to obtain more clear-cut and verifiable results. This is akin to the papers examining the expressivity of in-context learning starting with examining linear functions, rather than complex and varied domains, in which results may be unclear for a variety of reasons related to the domain – we did include a real-life task of Kalamang translation, in which we did not see benefits to instruction inference, unlike the first two domains, likely due to a breakdown in hypothesis proposal. Since we have now examined simple domains and have clear results for these simple domains, this makes it easier for future work to build on these results and examine more varied domains while keeping in mind the pieces of the reasoning loop as individual components.\n\nOne potential idea is self-consistency training, which could be a form of instruction tuning. For instance, we could use few-shot examples with underlying rules which can be expressed through natural language or code, and have the LM generate both an answer and explanatory rule separately. We can then have a combined loss, a generative loss which is the usual cross-entropy on the answer tokens (with multiple x_is tested for a single rule), and a consistency loss which penalizes the loss incurred by applying the generated rule to the answers generated by the model. We can certainly add more discussion of potential research directions and applications to the conclusion as part of future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study conducts an extensive examination of the interplay between three prevalent reasoning paradigms, providing empirical insights that could be valuable for future research. This encourages the development of innovative methods for hypothesis verification and correction in natural language processing.", "reasons_to_reject": "While the paper concludes that the ability to generate hypotheses is unrelated to learning from few-shot examples, it fails to suggest future directions or solutions to address this gap. This omission may limit the paper's impact on advancing the field's understanding of integrating these capabilities.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YAKzntCxpv", "reviewer": "Reviewer_zPDQ", "summary": "This paper evaluates the performance of three styles of prompting, namely, instruction following, few-shot learning, and instruction inference, on four tasks.\nThe main findings are:\n- instruction inference, which asks an LLM to generate hypotheses given examples and apply the best hypothesis to the input, is effective on simple tasks but not on complex tasks (low-resource translation and grammatical knowledge acquisision), and,\n- the relationship among the three styles of prompting is not observed.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, we’re glad that you see potential for future research! Please see responses below:\n\n> As described above, the experimental results are meaningful but no generalized knowledge is obtained from these results. What we can know is that the effectiveness of different prompting styles is mixed, and this is kind of expectable.\n\nWe agree with the need to further investigate in more domains and examine the broader applicability of the findings. However, technically a lot of results related to prompting can be summarized as “the effectiveness of different prompting styles is mixed”. For instance, chain-of-thought reasoning can be summarized this way, since CoT sometimes degrades performance as well. However, knowing this would still (1) provide an easy and practical way for people to get more performance out of their models (2) reveal future avenues for investigating the inner workings of LMs. In our case, instruction inference can be used relatively easily to guide LMs, at least on simple tasks. Furthermore, the dissociation between hypothesis proposal and in-context reasoning is interesting, and may merit further investigation. For instance, perhaps by training to make proposed hypotheses amenable with LM outputs without the ground truth label, we can increase reliability of models. \n\nMore broadly, we would like to point out that many highly impactful papers identify problems without proposing a solution in the same paper (such as the original papers on adversarial examples in NLP/vision, which are arguably still not fixed). However, knowing that this problem exists can guide the community in best practices for using and developing models. \n\n> The main focus of the paper is on three styles of prompting, i.e., instruction following, few-shot learning, and instruction inference, rather than generic logical inference like deductive, inductive, and abductive reasoning. The paper describes these three methods as deductive, inductive, and abductive learning, while they are just a specific instance [...]\n\n\nThank you for bringing this up, we plan to clarify the terminology more in the final version. We can clarify more in Section 2 that we are not making claims about these capabilities in general, but rather on common instantiations of these types of reasoning in LMs (hence the use of “learning” rather than “reasoning” in the title).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental results shown in the paper are useful and meaningful to see which style of prompting is effective on which tasks.\nHowever, it is difficult to obtain any general insights from the results and to generalize these results to other tasks.", "reasons_to_reject": "As described above, the experimental results are meaningful but no generalized knowledge is obtained from these results.\nWhat we can know is that the effectiveness of different prompting styles is mixed, and this is kind of expectable.\n\nThe main focus of the paper is on three styles of prompting, i.e., instruction following, few-shot learning, and instruction inference, rather than generic logical inference like deductive, inductive, and abductive reasoning.\nThe paper describes these three methods as deductive, inductive, and abductive learning, while they are just a specific instance of these logical inference patterns.\nIt sounds overgeneralization to use these logical terms in the title and the main description of the work.\nIn other words, the paper does not provide any generic observations for deductive/inductive/abductive learning ability of LLMs.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JzjaCSOYn4", "reviewer": "Reviewer_Ugsb", "summary": "This paper investigates whether LMs possess the types of reasoning necessary to robustly learn to perform and describe a task in-context. Specifically, the authors propose a reasoning loop consisting of deductive, inductive, and abductive reasoning, and then test whether LMs can perform each type of reasoning. Using coefficient prediction, nonce word translation, and machine translation tasks, the authors test the ability of models to follow instructions (deductive reasoning), generalize from few-shot prompts (inductive reasoning), and induce instructions given labeled examples.\n\nThe contribution of this paper is clear and original, and it yields a more fine-grained understanding of what kinds of in-context reasoning, exactly, will be more difficult for LLMs to accomplish.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’re very glad to hear that you enjoyed the paper, and thank you for your detailed suggestions! We have incorporated your suggestions into the current draft, and please see below for responses to your comments: \n\n> R1. Table 3 is opaque. What were the criteria for deciding whether a cell got a checkmark, partial check, or X? Were these qualitative judgments, or based on statistical significance tests?\n\nThis was qualitative, based on the actual accuracy for “Deductive/abductive/hypothesis proposal works”, and the checkmark/partial check, or X was based on whether respectively ground truth rules and instruction inference improved performance for the first two columns, and whether the hypotheses were actually mostly correct for the third column. \nThe purpose of Table 3 was mainly to summarize the takeaways for readers, rather than to provide detailed results. We will clarify the meanings of the symbols in the caption for the camera-ready version.\n\n> R2. The Kalamang translation results are weak, and don’t seem to provide especially strong evidence for either RQ. That said, this is a very difficult task, and I appreciate its inclusion; I therefore don’t think this is a significant weakness.\n\nThank you for appreciating the inclusion of Kalamang, we are aware that the results are rather mixed in terms of not following the trends in the functions and colours domains, but we thought it was important to include a real-world task which was more difficult, especially since similar papers have also been tested on more synthetic domains. It is possible that with more work in the low-resource translation domain, instruction inference may work better and similar trends may be seen in this domain as well, but we leave this to future work.\n\n\n> Typos/suggestions\n\nThank you for your detailed reading in making these suggestions and finding all these typos! We have corrected all typos in our draft. In addition, we will also address the suggestions in the camera-ready draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1. The hypothesized reasoning structure is specific, falsifiable, and broken down into testable components.\n\nA2. The results provide evidence that these reasoning mechanisms are separate in LLMs. It is interesting that prediction accuracy is not closely related to the ability to perform instruction induction.\n\nA3. Reproducibility seems good. Detailed experimental setup details (including all prompts and hyperparameters) are provided in the Appendix.\n\nA4. The writing style and figures are clear. (The paper is also just a joy to read. I especially appreciate the frequent pre-21st-century citations.)", "reasons_to_reject": "R1. Table 3 is opaque. What were the criteria for deciding whether a cell got a checkmark, partial check, or X? Were these qualitative judgments, or based on statistical significance tests?\n\nR2. The Kalamang translation results are weak, and don’t seem to provide especially strong evidence for either RQ. That said, this is a very difficult task, and I appreciate its inclusion; I therefore don’t think this is a significant weakness.", "questions_to_authors": "Suggestions:\n- The Kalamang grammar row of Table 2 looks funny. This task setup is very different from the others, so maybe it makes more sense to simply put this in a separate table where the task setup would be better described using different columns.\n- In Figure 3, could you reorder the legend rows to match with the order of the bars? Also, I didn’t notice at first that the y-axis ranges were different across subfigures, which made me think that the Llama scores were better than they actually were. Maybe you could use the same [0, 1] range for all subfigures?\n- When linking to RQ1 and RQ2 (as on p.6), it might look better to exclude the colon.\n- Inconsistent references to subfigures within Figure 3. First it’s “3A” and “3B”, but in the next paragraph, it’s “3(A)” and “3(B)”.’\n\nTypos:\n- Add period to end of Footnotes 4 and 10.\n- Why are “few-shot”, “zero-shot”, and “chain-of-thought” in \\texttt{} format on p.3 but nowhere else in the paper?\n- Add period after “Appendix F” at end of \"$\\textbf{Hypotheses}$” on p.5.\n- Remove space before “Settings” at top of p.4.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received two lukewarm and two strong reviews. Based on my reading of the reviews and paper, among the strengths and weaknnesses, I believe the following are relevant:

Strengths:
- The reviewers agreed the problem is well-motivated, and found the paper innovative, interesting, and well-written.

Weaknesses:
- The title is an overstatement (WMHL, z3T2; cf also statement by zPDQ). This is easy to fix: The authors are urged to consider more accurate titles, such as the suggestion by reviewer z3T2.
- While the tasks are diverse (linear functions, artificial languages, Kalamang translation) are quite diverse, the set of tasks is still limited (WMHL).
- No generalizable knowledge (zPDQ) or meaningful takeaways for future work (WMHL, z3T2) are derived from the results. The authors counter that identifying empirical results itself is valuable.

Overall, I concur with the reviewers in an overall positive assessment, and believe the paper meets the high standards of this conference.

[comment from PCs] Please consider the title suggestions of the AC and reviewers.
Decision: Accept