MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations
Authors: Arkil Patel
Satwik Bhattamishra
Siva Reddy
Dzmitry Bahdanau
Venue: EMNLP 2023 Main (2023)
Keywords: Benchmarking LLMs
Compositional Generalization
In-Context Learning
TL;DR: Analysing the in-context learning ability of LLMs to acquire novel interpretations of English words and phrases.

Abstract: Humans possess a remarkable ability to assign novel interpretations to linguistic expressions, enabling them to learn new words and understand community-specific connotations. However, Large Language Models (LLMs) have a knowledge cutoff and are costly to finetune repeatedly. Therefore, it is crucial for LLMs to learn novel interpretations in-context. In this paper, we systematically analyse the ability of LLMs to acquire novel interpretations using in-context learning. To facilitate our study, we introduce MAGNIFICo, an evaluation suite implemented within a text-to-SQL semantic parsing framework that incorporates diverse tokens and prompt settings to simulate real-world complexity. Experimental results on MAGNIFICo demonstrate that LLMs exhibit a surprisingly robust capacity for comprehending novel interpretations from natural language descriptions as well as from discussions within long conversations. Nevertheless, our findings also highlight the need for further improvements, particularly when interpreting unfamiliar words or when composing multiple novel interpretations simultaneously in the same example. Additionally, our analysis uncovers the semantic predispositions in LLMs and reveals the impact of recency bias for information presented in long contexts.

MAGNIFICO:
Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations
Arkil Patel
Satwik Bhattamishra
Siva Reddy Ww
Dzmitry Bahdanau #
Mila and McGill University
ServiceNow Research
University of Oxford
Facebook CIFAR Al Chair
Canada CIFAR AI Chair
Barkil.patel, siva.reddy.
bahdanau]@mil la. quebec
satwik, shra@cs. ox. ac. uk
Abstract
How many players with
training?
Humans possess remarkable ability to assign novel interpretations linguistic expressions, enabling them to learn new words and under- stand community-specific connotations. How- ever, Large Language Models (LLMs) have knowledge cutoff and are costly to finetune re- peatedly. Therefore, it is crucial for LLMs to learn novel interpretations in-context. In this paper, we systematically analyse the ability of L.MSs acquire novel interpretations using in- context learning. To facilitate study, we in- troduce MAGNIFICO, an evaluation suite im- plemented within text-to-SQL semantic pars- ing framework that incorporates diverse tokens and prompt settings to simulate real-world com- plexity. Experimental results MAGNIFICO demonstrate that L.MSs exhibit a surprisingly robust capacity for comprehending novel inter- pretations from natural language descriptions as well as from discussions within long conver- sations. Nevertheless, our findings also high- light the need for further improvements, partic- ularly when interpreting unfamiliar words or when composing multiple novel interpretations simultaneously in the same example. Addition- ally, our analysis uncovers semantic predis- positions in LLMs and reveals the impact of recency bias for information presented in long contexts.
Novel Interpretction
manry sgaresshe players
"Player who has received a yellow card"
foregy
mery gludi players more than 1000 craining??
adverzurial
How mony must alayers have more 1030 Usiinn?
Datput
Saueez COUNZ(") PAGH playes MUSSAL
1002 AND yeard "yea"
MODEL
Figure 1: An example from MAGNIFICO illustrating novel interpretation denoted by plausible, foreign, and adversarial forms. The corresponding base example is also provided.
language models are similarly capable of assign- ing novel interpretations to words and phrases is therefore interesting and requires investigation.
The task of learning novel interpretations has predominantly been studied from the perspective of finetuning a model, particularly the word em- beddings, to acquire novel words during training (Lampinen and McClelland, 2017; Pham et al.. 2018; Schick and Schütze, 2019). Prior studies on compositional generalization (Lake and Baroni, 2018; Kim and Linzen, 2020) also attempt to eval- uate novel word learning using specially crafted train-test splits in which certain combinations of words are systematically held out from the test set. recent years, however, contemporary Large Language Models (LLMs) have brought about a paradigm shift away from the classical train-test setup with their incredible capacity to learn new tasks in-context (Brown et al., 2020). With this study, seek to understand how well can LLMs acquire novel interpretations in-context. Compared to previous setups, in-context learning (ICL. is also more practical since it is difficult to train models every time a new interpretation is encountered.
1 Introduction
Humans can assign new interpretations to words or phrases a language and consequently use them compositionally in utterances. For instance, the word 'zoom' is increasingly used to refer to a vir- tual calling service in the context of the COVID-19 pandemic. Similarly, as our society progresses, new words such as "binge-watching' and 'selfie" keep getting coined frequently and become part of our daily usage. Moreover, in regular conversa- tions, people might assign custom interpretations to words or phrases (c.g., see the interpretation of "underpaid" in Figure 2). The question of whether

---

account il with runnerap number burnackerss
The wndd "aenure' refers messured empllsmmmnt cakoratadd by sekrng the cfferance belewen the and cate
depariment underpaid
SKLECT acosunt, FROM (aKLECT accsunt, id couneçe]
oRdice av couar(-) seedd LENET s3 AS cab Dance by __tab.*__rop GENER
somaome undarpaid??
Which webeaor has rumber of saaderia"
Wel.
sSUECT adviloo rane (eriect advitor COUNEC-) M 1_prpp FROM student axoup advisoe dsama cocus(e) ausc
underpard If Ihay sam Isas then 100
amployme's hestory when tenure a more tran 3 and aberi cata was alar Jumuary 1996.
t_prop
Whu are the underpedd inshuctoss Computer Science departrinn??
Joh, history ллкК
Whihh
number studerts?
end_
seuace mater Pack csilact ralor CouNt(+)
stare_date '1996-01-01"
cosnei+) assic sasus a] All t. tab ossiia av t_teb t_ptop GENCR
vicaaa deps nane Doap.
AND sallery
Language Description
Few-Shot Examples
Dialogue
Figure 2: We consider several scenarios of how novel interpretation can be introduced in the prompt. Left: A natural language description of the novel interpretation 'tenure' is provided with the question. Centre: Few-shot examples illustrating usage of the novel interpretation 'runner-up" are provided with the question. Right: Dialogue context describing the novel interpretation 'underpaid' is provided with the question.
In this work, we systematically analyse the abil- ity of LLMs in-context learn novel interpreta- tions. We summarize our contributions below.
find it difficult to learn interpretations presented earlier in the context.
Related Work
Evaluation Suite. To facilitate our analysis, we create an evaluation suite, MAGNIFICO: Measuring Adaptability and Generalization to Novel Interpretations For In-Context Learning. Each example in MAGNIFICO is text-to-SQL. semantic parsing problem that requires models to understand one or more novel interpretations used in the input text to generate the correct SQL query. To simulate real-world diversity and complexity. we experiment with different ways of introducing new interpretations to the model (see Figure 2).
Word Learning. Previous works (Wang et al., 2017; Herbelot and Vecchi, 2016; Lampinen and McClelland, 2017; Pham et al., 2018; Schick and Schütze, 2019) have developed task- or model- specific approaches for learning the embeddings of novel words. However, these methods cannot be applied in diverse scenarios with contemporary Large Language Models (LLMs). In this work, we take a more practical approach by evaluating how well do LLMs understand novel interpretations of words and phrases in-context on top of a grounded NLP task, text-to-SQL semantic parsing.
Capabilities of LLMs. We extensively experi- ment with 11 LLMs to understand their ability for learning novel interpretations in-context. Experi- ments on MAGNIFICO reveal that LLMs show a high degree of capability in learning novel in- terpretations even from a brief natural language description of the interpretation or from a long- form conversation. For larger LMs, learning from a description is competitive to providing explicit few-shot examples.
There are limited number of works that analyse the novel word learning abilities of LLMs. Haley (2020) and Thrush et al. (2020) analysed novel word learning with BERT (Devlin et al., 2019) us- ing synthetic tests. However, it is unclear how their findings relate to autoregressive LLMs. Brown et al. (2020) qualitatively tested GPT-3's ability to use a novel word in sentence after seeing its definition. Eisenschlos et al. (2023) analyse the in-context word learning abilities of LLMs using a synthetic co-reference resolution task. In this paper, however, we work on a more practical task and take a broader view of the problem by studying the acquisition of novel interpretations, which can arise even from
Challenges for LLMs. We find that LLMs severely fail learning multiple novel interpre- tations simultaneously. Moreover, we observed that L.LMs find more challenging to learn inter- pretations for unfamiliar words. Our analysis also shows that LLMs have a recency bias and might

---

CATEGORY
INTERPRETATION
EXAMPLES
Input: namo, and station the baseline Intitde?
Operations
Minimum
Output: SELECT name, lat, city FROM station ORDER BY lat LIMIT
Input: Display and first
prevaleni major.
Subquery-bssed Operations
Most-frequent
Output: SELECT Sex, Fnane FROM Student WHERE Major IN (SELECT Mejor FROM (HUNT(N) LIHIT 1)
Student GROUP Major
Input: List the names of all heavy courses ordered by their tites and credits.
Value- buueb Fillering
credit
Output:
FROM course WHERE credits ORDER title, credits
Concatenation of last and
Input: How many students are there with 'gE' in allns?
Column Operations
first name
Output: SELECT COUNT(+) FROM student WHERE Inane II
Thane LIKE
'NgEK"
Table
Examples of novel interpretations in MAGNIFICo. Illustrated examples use a plausible English form.
existing words and phrases in the vocabulary. We also study compositional generalization of multiple novel interpretations simultaneously.
3 MAGNIFICO
We choose the text-to-SQL task to test LLMs' abil- ity to handle novel interpretations because of its rel- evance to real-world applications. Moreover, con- temporary LLMs already achieve good zero/few- shot in-context performance the task. We cre- ate MAGNIFICO by modifying and re-tasking examples from an existing text-to-SQL. benchmark, Spider (Yu al., 2018). Below, we describe the procedure in detail.
Compositional Generalization. Recent works (Lake and Baroni, 2018; Kim and Linzen, 2020; Keysers et al., 2020) proposed benchmarks with a systematic difference between the train and test sets: novel combinations of certain words are held out from the train set. However, such evaluation setups susceptible to fairness issues (Sikarwar al., 2022) arising from the dependence train set. Moreover, indepedende factors in the train set can influence generalization performance (Patel et al., 2022). Our evaluation framework is set within the paradigm of in-context learning (ICL), which does not require the creation of an explicit train set. Note that even in ICL. settings, LLMS have saturated existing compositional generaliza- tion benchmarks (Drozdov et al., 2023). More re- cently, An al. (2023) proposed a new benchmark, CoFF, for in-context compositional generalization. However, their focus was on understanding the fac- tors affecting better selection of in-context exam- ples for compositional generalization. Moreover, the examples in COFE are based on another syn- thetic benchmark. while we focus on more realistic settings using grounded text-to-SQL task.
3.1 Novel Interpretations
We define a set of 24 interpretations that are ci- ther already being used or can be introduced in the examples of Spider:
* Basic operations: Standard column operations frequently used in SQL queries.
Subquery- -bsed operations: Complex opera- tions using nested subqueries.
* Value-based filtering: Particular subset of val- ues for specific columns.
Column operations: Operations performed over specific columns.
Knowledge-intensive text-to-SQL. Works on knowledge-intensiin text-to-SQL. (Li et al., 2023a; Lee et al., 2021: Zhao et al., 2022; Dou et al., 2022) have some similarities with our work that they as- sign schema-specific external knowledge to words or phrases in the input query. However, our in- terpretations are much more dynamic and do not have pre-defined formal definitions. Moreover, the focus of these works is on domain generalization for text-to-SQL semantic parsing. We only use text-to-SQL- as a testbed since allows us more formally ground meanings of interpretations and has real-world applicability.
Table I provides examples of some of the inter- pretations that we defined. We will refer to the word or phrase used to denote the novel interpreta- tion on the source side of the example as its form. We defined 18 interpretations denoted by sin- gle word form and 6 interpretations denoted by phrase form (see Figure 3 for an illustration). The full list of all interpretations can be found in Tables 4 and in the Appendix. For the 18 interpretations denoted by a single word, we experiment with three types of forms that vary in their pre-existing seman- tic meaning: (1) plausible forms are words that can

---

Phrase form
Example from Spider
Ust the names of ceart-certified and licensed Physiclans who never took any appointment.
Show de for akkrrgonee who don't have certif
EICEPT SULECT THDN cortiflcate
SRraao name FROX physician tocoee SERReT t2 FRON
appoictzent 2018 phyeiciands ON t1 phyeician ta. expleyeddd WHBRE position sor I8 ("Statr Internist")
Example MAGNIFICo
Show for exarpad employees who don't have a certifiate.
ees essuect eld MaoN exployee Ansie caliry sos0n Eacterr SELEC7
ceze lficaae
Multiple novel interpretations
Which the pure-sclende department with the prevalent major?
Figure 4: Illustration the 'Regex-based pattern match- ing' procedure used for creating examples in MAG- NIFICo.
SEtIEO dept, name FROX student simen por IM (smaee major FROM student WHERE dept_nase INl ("Paysios", "Biology") GROOP bY majox ORDER BY COUNz(+) DESC Limit
Figure 3: Illustrations of examples with phrase form and multiple novel interpretations in MAGNIFICO.
queries of the seed example to include the corre- sponding form in a suitable manner. In such cases, we prompt GPT-4 (OpenAI, 2023) with the query of the seed example and instruct to paraphrase the query so as to include the form of the novel in- terpretation. We then manually examine the model- generated paraphrase for grammatical correctness and consistency of intended meaning. Similar to the previous method, modify the target SQL query using hand-crafted rules, if required.
reasonably be expected to represent the novel in- terpretation in realistic scenarios, (2) foreign forms are novel words without any pre-defined meaning that are generated using random permutation of English characters, and (3) adversarial forms words with an existing meaning that is contrary the intent expressed by the novel interpretation. Figure illustrates the three types of forms in an example from MAGNIFICO.
(3) Synchronous Context-Free Grammar. For many interpretations, there are either none or very few examples already existing in Spider. It is also difficult to find such examples automatically based on regex patterns. In such cases, if we have obtained limited number of examples from Spider, we extract an SCFG representing those examples by abstracting away specific column and table names and data values similar to the method used by Yu et al. (2021). If there are not enough examples in Spider, we define an SCFG ourselves that represents the interpretation. We then automatically generate examples from the SCFG by filling with randomly sampled column and table names and values from the set of databases in Spider. We only keep the examples for which the generated SQL query correctly executes and does not give a NULL output. An illustration of this process is provided in Figure 5.
3.2 Generating Data
We create MAGNIFICO examples by modifying examples in the Spider dataset such that understand- ing a novel interpretation' used in the input is nec- essary successfully generate the corresponding SQL query. We will refer to the original examples from Spider as seed examples. For each interpre- tation, we generate data using one or more of the following methods:
(1) Regex-based pattern matching. Some inter- pretations such as 'the minimum value" (see Table 1) already have examples existing in Spider. For such interpretations,
find the relevant seed ex- amples using regex- based pattern matching, either on the source side by conditioning on the presence certain keywords such as minimum or lowest or on the target side by conditioning on operations such as min(). We then modify the seed examples to include the form of the interpretation in the input and inculcate the corresponding logic in the target SQL query using specific rules, if required. An illustration of this process is shown in Figure 4. (2) M.M--sisstted constrained paraphrasing. For many interpretations, is not possible to manu- ally devise rules for modifying the natural language
Multiple novel interpretations in same example. From the data created using the procedures above, we select pairs of examples that have different novel interpretations but use the same database schema. We devise set of rules using which, given such a pair, can automatically create new example that requires understanding both novel interpretations (one from each of the examples in the pair) simultancously. Figure illustrates such an example. We created a
We experiment with different types of prompt contexts for explaining the novel interpretation, detailed in $4.

---

Example from Spider
TYPE
UNIQUE TEMPLATES
TYPES
NUM EXAMPLES
ShowrCareeSte for whldrraattswtth Cittanee than average
adversarial.
Single-wond
1150
4600
Avngieiaan FROM Sirorariy
plausible Toreign
Phrase
279
hase, plansible
558
Multiple novel
base. adversarial,
Extract SCFG
interpretations
94
plausible, Foreign
376
Show columnó for al abted with colunnn more than average.
ExLace coluzno FROM tabluo WEsss nolimen > (SRLECT
TOTAL
1523
5534
Avo(solunat) EROM tabled)
Table Dataset statistics for MAGNIFICO.
Introduce Novel Interpretation
columno with satistactiry coummnt.
columnd
prompt types them. An illustration of base example is shown in Figure 1.
Example MAGNIFICo
Summary. Overall, we created 1523 unique exam- ples across 24 interpretations. The forms of inter- pretations in these examples can be automatically replaced to generate more data at scale. Dataset statistics for MAGNIFICO are provided in Table 2. Additional details creating MAGNIFICO provided in Appendix B. Note that each example in MAGNIFICO was manually reviewed by least one of the authors to ensure correctness.
Show narnes for all students with salislattory grades.
SEGROD ERON stadeet grade (SELECT AVG(crade) FROM ecudent)
Figure 5: Illustration of the 'Synchronous Context- Free Grammar' procedure used for creating examples in MAGNIFICO.
total of 376 such examples spanning 24 unique combinations of interpretations. We manually reviewed each example to ensure correctness.
Experimental Setup
Generating Conversations. We generate long-form dialogues for subset of examples in MAGNIFICO. For each database schema used in these examples, we prompt GPT-42 to generate long conversation between two users of that database. We instruct GPT-4 to introduce the corresponding novel interpretation and its description in manner that makes it blend naturally into the flow of the conversation at the beginning. We generated total of 125 unique dialogues, each at least 2000 tokens long. We manually reviewed all generated dialogues to ensure correctness.
In this section, we will discuss the setup for our experiments on MAGNIFICO."
We experiment with OpenAI 5-T-bo (v0301) (Brown et al., 2020; Ouyang et al., 2022), StarCoder (Li et al., 2023b), LLaMA-7B, 30B (Touvron et al., 2023a), Alpaca-7B (Taori et al., 2023), MPT-7B+, MPT-7B-Instruct, RedPajama-7B$, and RWKV-14B (Bo, 2021). We additionally experimented with GPT-4 (OpenAI, 2023) and LLaMA-2 (Touvron et al., 2023b), results for which are provided in Appendix C.4. For all models, we decode greedily for a maximum of 128 tokens. To take stock of the basic text-to-SQL semantic parsing capabilities of these models, show their execution accuracies over the base examples in MAGNIFICO averaged over all interpretations in Figure 6. Some of the results for RWKV-14B and the RedPajama-78 models can be found in Appendix C.
Models.
Base Examples. We are only interested in measuring the ability of models generalize to novel interpretations and not how well they perform on the text-to-SQL semantic parsing task. Hence, for every example in MAGNIFICO with a novel interpretation, we also maintain an example that does not include any novel interpretation form and instead directly states the interpretation as part of the input query. These examples serve as comparative reference point for our evaluation. We refer these examples as base examples and measure the performance of models across all
Prompt. All our experiments are in the in-context learning experimental setup. Our prompt structure
*We make code and data available at http://jhhhlcccocccccclllAAAAAnt hhnp://wwwww..ossiimmm.c/oooo/oo https:/wwwww.ooooo...yyy///opppa
Prompt provided in Figure 19 in the Appendix.

---

Execution Accuracy on Base Examples
Relative Parformance Plausible
Direct
Exerut on Accuracy
Pelsdive 40
LL399-18
Appica
Nnt-re not LaM ste Sartodar CPIS
Models
DE
TAN Shot
Prompt Setting
Figure 6: Average execution accuracy (1) of all mod- els on base examples in MAGNIFICO across various prompt settings.
Figure 7: Relative performance (f) of all models on MAGNIFICO across various prompt settings when the TOKEN is plausible English word.
largely follows the Create Table + Select 3 prompt format from Rajkumar et al. (2022) which resulted in the best performance on Spider with OpenAT Codex (Chen et al., 2021). This format provides the CREATE TABLE commands for each table in the schema and displays the values for the top three rows for each table. We experiment with three different prompt settings: (1) 'Direct' is exactly the zero-shot Create Table Select setting which includes information about how to interpret the form of the novel interpretation, (2) 'Deseription' additionally includes a brief, one-line natural language description of the novel interpretation(s), input-output examples of the novel interpretation instead of the description. We hold out 5 examples for each interpretation from the test sets to maintain consistency in testing across various experimental settings. For experiments with a dialogue in the context, the dialogue is prepended to the Create Table Select 3 prompt. Examples for each type of prompt are provided in Appendix E.
the corresponding base examples. 8 Hence, the higher the Relative Performance, the lower the model's drop in performance on examples with novel interpretation(s) (relative to base examples), and consequently, the higher its ability learn novel interpretations.
Results and Discussion
5.1 Impact of Description and Few-shot Examples
Question: How well can LLMs learn novel interpre- lations when the interpretation is simply described in an English sentence? And how does it compare against the case when we provide few-shot exam- ples of usage of that interpretation?
and (3) 'Few-shot' includes
We compare providing a natural language de- scription of the novel interpretation ("Description' prompt type) against providing examples of us- age of the novel interpretation ("Few-shot' prompt type). Figure provides the results when the form used represent the novel interpretation is a plau- sible English word. The results for foreign and adversarial forms can found in Figures 16 and 17 in the Appendix.
Metric. We use a metric called Relative Perfor- mance measure the generalization ability of mod- els towards acquiring novel interpretations. Our metric provides measure that is relative to the per- formance of the model on the corresponding base examples:
Most LLMs exhibit a surprisingly high capabil- ity to understand and generalize to novel interpreta- tions from simply the natural language descriptions. This capability seems to increase with model size as GPT-3 5-5-brbo and LLaMA-30G outperform all other models while the smallest model, LLaMA-7B, struggles to generalize from just the description. It is also interesting to see the benefit of instruction finetuning (Wang et al., 2023) in learning novel in- terpretations in-context just from natural language descriptions: the instruction-finetume models out- perform their corresponding base models, often by
EX
Relative Performance min
F. baxaase 1
100
where NX is the execution accuracy? on the examples with novel interpretations from MAG- NIFICO and EX base is the execution accuracy on
Forr multiple novel interpretations the same example. we include 3 support examples for each novel interpretation. 'Mcasure of equivalence hetween output obtained from ex- ocuting the generated SQL. query and the ground truth output.
"We only consider interpretations for which the execution accuracy on hase examples is at least 5%.

---

Effect of pre-existing semantic interpretation
103
StarCoder
GPT-3.5 5-Turbo
Apecs M7776
Nothaama dedagas re
Stacese 6PT
Average Relatuve Pertormanca
Prompt
Plausible
Foreign
Plausible
Foreign
Description
79.40
80,74
91.46
85.95
Dialogue
68.91
80.55
84.87
87.63
Table Relative performance (1) of StarCoder and GPT-3. GPT-3.5-Turbe on examples in MAGNIFICo when the description of the novel interpretation provided long form dialogue.
Hus
Advassoiall
Pre-oxisting Semantic interpratation
Figure 8: Relative performance (t) of all models across forms of different types.
on a dialogue history between multiple users. An example is provided in Figure 2.
Question: How well can LLMs learn novel in- terpretation from its description mentioned briefly within long-form dialogue?
large margins. All models generalize well when few examples of usage of the novel interpretation are provided.
We select 8 interpretationss covering total of 583 examples from MAGNIFICO, encompassing all four categorics. We generate long conversa- tion contexts for each of these examples as de- scribed in $3.2. An example of the prompt struc- ture is provided in Figure 22. We experiment with StarCoder and 5-Tubbo since they pable of processing more than 2000 tokens of text in-context. The results are provided in Table 3. For ease of comparison, we also state the results with the 'Description' prompt-type for the 8 interpreta- tions considered.
5.2 Impact of Pre-existing Semantic Interpretation
Question: How much does the existing semantic interpretation of the form denoting the novel inter- pretation inftuence the generalization capability of LLMs?
As mentioned in $3.1, we experiment with three types of form. We plot the relative performance averaged over the 'Description' and 'Few-shot' prompt types" in Figure 8.
We see a trend decrease in generalization abil- ity when the pre-existing semantic interpretation of the form steers farther from the intended meaning of the novel interpretation. This shows that L.MMs have strong semantic priors that may require tar- geted approaches to be overcome. Moreover, the fact that LLMs easily understand novel inter-
For both models, using a foreign form to repre- sent the novel interpretation does not result in much performance difference when the description of the novel interpretation is blended inside a long form dialogue instead of directly stating it. However, when the form is plausible english word, we see a clear decrease in generalization ability for both pretations when presented in familiar form (as opposed completely foreign words) an inter- esting finding for potential applications requiring acquisition novel interpretations in the wild (e.g., conversational agents).
models. The decrease is much more significant for Star compared to 5.T-..-Turbb This in- dicates that LLMs may find it difficult to associate case-specific interpretation with tokens that they already familiar with when used in long conver- sations. It possible that the models do not pay much attention to that aspect of the conversation might seem more 'normal' compared to the case where a foreign form is used.
5.3 Acquiring Novel Interpretations From Long Form Dialogue
We envision real-life scenario requiring compo- sitional generalization: acquiring novel interpreta- tions introduced long form conversation. This may arise in situations such as having a conver- sation with an Al personal assistant or when we want to condition the outputs of an Al system based
5.4 Impact of Position of Description in Context Window
Question: How sensitive are LLMs to the location in the context window that the novel interpretation is described in?
We average over the prompt types to improve readability of the figure. The complete figure can be seen in Figure 18 in the Appendix.
We experiment with placing the description at the beginning, middle, or the end of the prompt

---

We notice that LLaMA, StarCoder, and APT-3.5-Turbee models show surprisingly high ability learn the novel interpretation from just the description. It is even more surprising to see both MPT-7B models struggle since they compar- atively excelled for single-word form interpreta- tions (see Figure 7). This shows that the task of learning novel interpretations denoted by multi- word phrases is not simply extension of lcarning single-word form interpretations, but separate task that presents its own of challenges. Lastly, interesting to see that contrary to expectations, in both Sensitivity to location of description directly Plausible
Multiple novel interpretations learned simultaneously
Foreign
BtarCeder
74:26
93.37
100.0
51.32
180.0
SuuCoder
Model
50
91.17
1.0.00
79:35
100.0
star Position of Description
stat Position of Description
Modt:
trai
Relative
(a)
Sensitivity to location of description In dialogue Piausible
Foreign
89.04
100.0
91.56
100.0
Fareig
Model
Fre-existing Semanti Interpretation
90.8
100.07
100.0
61:78
64:33
100.0
Figure 10: Relative performance (t) of all models across all settings when there multiple novel in- terpretations in the same example.
Position Description
Position Description
Figure 9: Relative performance (1) of StarCoder and GPT-3.5- 5-Turbo on MAGNIFICO across different lo- cations of the description of the novel interpretation in the prompt when the description is directly stated (top) and when the description is mentioned in dialogue (bottom). The numbers are relative to the performance for the end position.
'Few-shot' prompt types provided in Figure 10. We notice that all models struggle learning multiple novel interpretations in the same example compared to learning just one novel interpretation. UPT-3.5-Turbee is the best performing model, sig- nificantly outperforming StarCoder while the rest of the models show nearly trivial performance. The difference in performance between 'description' and 'few-shot' prompt types for for foreign form sug- gests that models have a comparatively harder time composing interpretations when they are presented individually in separate examples in the prompt.
when using the 'Description' prompt type. We also experiment with the 'Dialogue' setting by placing the turns of conversation describing the novel in- terpretation at the beginning, middle, or the end of the dialogue. The results for both experiments are provided in Figure 9. Note that we measure performance relative to the performance when the novel interpretation is described in the end to better characterize recency bias.
5.6 Learning Novel Interpretations of Phrases
Question: Are LLMs able to learn novel interpreta- tions when they are denoted by more than a single word?
We observe a clear trend of recency bias in both LLMs, where the generalization increases when the interpretation is described nearer to the end of the context window. StarCoder suf- fers much more variation in performance com- pared to formance between start and end positions for 1PT-3.5-Turbbe while comparatively small, is still significant enough to indicate a stronger preference for information presented later in the context.
We defined 6 interpretations denoted by phrases of plausible English words in MAGNIFICO, amounting total of 279 examples (see Figure for an example). The results of evaluation over these examples are provided in Figure 11.
The difference in per-
5.5 Composing Multiple Novel Interpretations
Question: Are LLMs able to simultaneously learn multiple novel interpretations used compositionally in the same example?
We evaluate models total of 376 exam- ples that require simultaneously understanding two novel interpretations (see Figure 3 for an example). The results for all models across all three types of form interpretations using the 'Description' and
StarCoder outperforms prompt settings.

---

Relative Parformance Plausible
Acknowledgments
Rotatve Performancs (15)
We would like to thank Navin Goyal for initial dis- cussions related to the idea behind this work. We also thank the anonymous reviewers, and our col- leagues Mila and McGill University for helpful discussions and for providing valuable feedback. Arkil is also supported by the Canada Graduate Scholarship- Master's (CGS-M) funded by the Natural Sciences and Engineering Research Coun- cil of Canada (NSERC).
00
10
Sbrect
Descrptor
Prompt Setting
Figure 1I: Relative performance (1) of all models on MAGNIFICO across various prompt settings when the novel interpretation is a phrase composed of multiple English words.
Limitations
We created our evaluation suite MAGNIFICO over a single task, text-to-SQL semantic parsing. While semantic parsing is a fundamental task in language processing with general applicability, it would be useful to verify the findings across other tasks and domains. In the future, we aim incorporate tasks from diverse domains such as classification to better support our claims.
6 Final Remarks
We studied the ability of LLMs to interpret new words and phrases in-context using their descrip- tion or a few demonstrations. We also extended this study to a realistic scenario: understanding user- defined interpretations from long form dialogue.
Our results indicate that current LLMs can, to an extent, acquire novel interpretations from diverse forms of context. However, interpreting unfamiliar words or multiple novel words simultaneously still poses significant challenge for existing L.LMs. These tasks can serve as a measure to evaluate the compositional generalization abilities of future LLMs in practical settings.
The execution accuracies over base examples in MAGNIFICO are low for smaller models. This results in higher variance in the results of small models. While enforce threshold of minimum 5% accuracy the base examples for each inter- pretation be included the results, in the future, we shall also include experiments over a task that is more easily solvable by smaller models.
It is interesting to note that instruction fine- tuming leads significant improvements in learn- ing from descriptions across three different LLM families. Considering that instruction fine-tuning doesn't 't involve acquiring novel semantics, could be useful to understand why it has this impact.
The number of data points for some of our ex- perimental settings (such as multiple novel inter- pretations) is not large. However, note that study was exploratory in nature and our main focus was on analysing the in-context learning abilities of LLMs for acquiring novel interpretations rather than proposing a general benchmark for evaluat- ing L.Ms.. Our findings revealed that LLMS face more difficulty when there are multiple novel in- terpretations in the same example. This motivates us to look more closely this particular setting in the future and potentially create challenging benchmark.
In the past few years, several works (Lake and Baroni, 2018; Kim and Linzen, 2020) showed that sequence models were limited in their ability to generalize novel words semantic parsing tasks based on few examples in the training set. Many specialised methods and approaches (Liu et al., 2021: Chen et al., 2020) were designed to address this problem. It is therefore fascinating to see that contemporary general LLMs are able to generalize to novel words from not only processing few ex- amples in-context, but also from natural language descriptions and conversations. While large part of the compositional generalization challenge still remains unsolved, we feel it is important high- light this paradigm shift. We hope our work paves the way for further studies of practical setups that require LLMs to generalize compositionally.
Ethics Statement
We have extensively discussed the limitations of our work in the previous section. We use exist- ing dataset, Spider (Yu et al., 2018), which is pub- licly available and commonly used in NLP research. We generate additional data by modifying the ex- amples in Spider in rulc-based manner. Since we focus on a text-to-SQL. semantic parsing task, there

---

are minimal risks and biases associated with our data and we believe that it does not require ethical consideration. We also employed Large Language Models to automatically generate data, and each example of the generated data went through man- ual verification, ensuring that it does not pose any significant risk. We have discussed the experimen- tal details in Appendix A. The research presented in this paper focuses on analysing the in-context learning abilities of LLMs targeted towards inter- preting novel interpretations and we believe that our work does not raise any ethical concerns.
Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint vxXiv:207..0337.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. 2020. Compositional generaliza- tion via neural-symbolic stack machines. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1690-1701. Curran Associates. Inc.
Maxime Chevalier-Boisver, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2019. BabyAl: First steps towards grounded language learning with a hu- man in the loop. In International Conference on Learning Representations.
References
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. 2021. Meta-learning to compositionally gen- eralize. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 3322-3335, Online. Association for Computa- tional Linguisties.
Ekin Akyurek and Jacob Andreas. 2021. Lexicon learn- ing for few shot sequence modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4934 4934, Online. Association for Computational Linguisties.
Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nan- ning Zheng, Jian-Guang Lou, and Dongmei Zhang. 2023. How do in-context examples affect compo- sitional generalization? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 11027 11052, Toronto, Canada. Association for Computa- tional Linguisties.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- mologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguisties.
Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics. pages 7556-7566, Online. Association for Computational Linguistics.
Longxu Dou, Yan Gao, Xuqi Liu, Mingyang Pan, Dingzirui Wang. Wanxiang Che, Dechen Zhan, Min- Yen Kan, and Jian-Guang Lou. 2022. Towards knowledge-intensive text-to-SQI. semantic parsing with formulaic knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 5240-5253, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song. Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2023. Compositional semantic parsing with large language models. The Eleventh International Conference on Learning Representations.
PENG Bo. 2021. Blinkdl/rwkv-Im 0.01.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877191-19
Julian Martin Eisenschlos. Jeremy R. Cole, Fangyu Liu, and William W. Cohen. 2023. WinoDiet: Probing language models for in-context word acquisition. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 94-102, Dubrovnik, Croatia, Associa- tion for Computational Linguistics.
Paco Calvo and John Symons. 2014. The archisecture of cognition: Rethinking Fodor and Pylyshyn's sys- tematicity challenge. MIT Press.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming
Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph,
Jeny Fodor and Ernest Lepore. 2002. The composi- lionality papers. Oxford University Press.

---

Jerry A Fodor and Zenon W Pylyshyn. 1988. Connec- tionism and cognitive architecture: A critical analysis. Cognition, 28( I 1-2333771.
Andrew Lampinen and James McClelland. 2017. One-shot and few-shot learning of word embeddings. arXiv preprint arXiv: 1710. 10280.
Jonathan Gordon, David Lopez-Paz, Marco Baroni, and
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gri- bovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom, 2021. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Information Processing Systems.
Diane Bouchacourt. 2020. Permutation equivariant models for compositional generalization in language. In International Conference on Learning Representa- lions.
Demi Guo, Yoon Kim, and Alexander Rush. 2020. Sequence-level mixed sample data augmentation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 5547-5552, Online. Association for Computa- tional Linguistics.
Lee, Oleksandr Polozov, and Matthew Richardson. 2021. KaggleDBQA: Realistic evalu- ation of text-to- parsers. In Proceedings of the 59th Anmual Meeting of the Association for Compu- rational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume I: Long Pupers), pages 2261-2273, Online. As- sociation for Computational Linguisties.
Robert F Hadley. 1994. Systematicity in connectionist language learning. Mind & Language, 933):247-272.
Coleman Haley. 2020. This is a BERT. now there are several of them. they generalize novel words? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 333-341, Online. Association for Com- putational Linguisties.
Jinyang Li. Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang. Bowen Li, Bailin Wang. Bowen Qin, Rongyu Cao, Ruiying Geng, et al. 2023a. Can llm already serve as a database interface? a big bench for large- scale database grounded text-to-sqls. arXiv preprint 1rXiv:2305.03111
Aurélie Herbelot and Eva Maria Veechi. 2016. Many speakers, many worlds. Linguistic Issues in Lan- guage Technology. 13.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo Thomas Wang. Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin. Muhtasham Oblokulov, Zhiruo Wang. Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh. Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov. Manuel Romero. Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf. Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson. Carolyn Jane Anderson, Brendan Dolan-Gavitt Dan- ish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023b. Starcoder: may the source be with you!
Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. 2021. Grounded language learning fast and slow. In International Conference on Learning Representa- tions.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov. Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Mare van Zee, and Olivier Bousquet. 2020. Measur- ing compositional generalization: A comprehensive method realistic data. In International Conference on Learning Representations.
Najoung Kim and Tal Linzen. 2020. COGS: A compo- sitional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105, Online. As- sociation for Computational Linguistics.
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to toicce-to--cee to-uuuueceeeeeeeeeeeeeeeee-eeeee recurrent networks. In Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Ma- chine Learning Research, pages 2873-2833 PMLR.
Yuanpeng Li, Liang Zhao, Jianyu Wang. and Joel Hest- ness. 2019, Compositional generalization for primi- tive substitutions. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4293-4302, Hong Kong, China, Association for Computational Linguistics.
Brenden M Lake. 2019. Compositional generalization through meta pequeeceto--sequence learning. In Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Chenyao Liu, Shengnan An, Zeqi Lin, Qian Liu, Bei Chen, Jian-Guang Lou, Lijie Wen, Nanning Zheng,

---

and Dongmei Zhang. 2021. Learning algebraic re-
Empirical Methods in Natural Language Processing, pages 648-669 Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
combination for compositional generalization. In Findings of the Association for Computational Lin-
guistics: ACL-UJCLL 2021. pages 1129-1144, On-
line. Association for Computational Linguistics.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin. Percy Liang. and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable cnattuctioo-ollowinne model. Stan- ford Center for Research on Foundation Models. https://rrfm stamford. edu/2023/03/ 3/aappacaa huml.
Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen,
Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng. and Dongmei Zhang. 2020. Compositional gener- alization by learning analytical expressions. In Ad- vances in Neural Information Processing Systems, volume 33, pages 11416-1 1427. Curran Associates, Inc.
Tristan Thrush, Ethan Wilcox, and Roger Levy. 2020. Investigating novel verb learning in BERT: Selec- tional preference classes and alternation- tactic generalization. In Proceedings of the Third BlackboxNIPP Workshop on Analyzing and Interpret- ing Neural Networks for NLP, pages 265-275, On- line. Association for Computational Linguistics.
syn-
Gary Marcus, 2003. The algebraic mind: Integrating connectionism and cognitive science. MIT press.
OpenAl. 2023. Gpt-4 technical report.
Long Ouyang. Jeff Wu, Xu Jiang. Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu. Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami. Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten. Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez. Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison. Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019, Pytorch: An imperative style, frigh-eeffrmance deep learning li- brary. In Advances in Newral Information Processing Systems, volume 32, Curran Associates, Inc.
Arkil Patel, Satwik Bhattamishra, Phil Blunsom, and Navin Goyal. 2022. Revisiting the compositional generalization abilitics of neural sequence models. In Proceedings of the 60th Annual Meeting of the A Association for Computational Linguistics (Volume
2: Short Papers). pages 424 434, Dublin, Ireland. Association for Computational Linguistics.
Ngoc-Quan Pham, Jan Niehues, and Alex Waibel. 2018. Towards one-shot learning for rare-word translation with external experts. arXiv preprint arXiv:180.003182.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Process- ing Systems.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabilities of large language models.
Timo Schick and Hinrich Schütze. 2019. Learning se- mantic representations for novel words: Leveraging both form and context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6965-6973.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances in neural information processing systems, 30.
Ankur Sikarwar, Arkil Patel, and Navin Goyal. 2022. When can transformers ground and compose: In- sights from compositional generalization bench- marks. In Proceedings of the 2022 Conference on
Su Wang, Stephen Roller, and Katrin Erk. 2017. Distri- butional modeling on a diet: One-shot word learning from text only.

---

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct Aligning language models with self-generated instructions.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- iez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-arr natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Hitomi Yanaka, Koji Mineshima, and Kentaro Inui. 2021. SyGNS: systematic generalization testbed based on natural language semantics. In Findings of the Association for Computational Linguistics: ACL-
IJCNLP 2021, pages 103-119, Online. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, richard socher, and Caiming Xiong. 2021. Gra(pp)as la Grammar-augmentee pre-training for table semantic parsing. In International Conference Learning Representations.
Tao Yu, Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingn- ing Yao. Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods Natural Lan- guage Processing, pages 391 1-3921, Brussels, Bel- gium. Association for Computational Linguistics.
Chen Zhao, Yu Su, Adam Pauls, and Emmanouil An- tonios Platanios. 2022. Bridging the generalization gap in text-to-SQL parsing with schema expansion. In Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers). pages 5568-5578, Dublin, Ireland. Association for Computational Linguistics.
Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-SQL. with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 396-411, Online. Association for Computa- tional Linguistics.

---

A Implementation Details
Execution Accuracy on Base Examples
Direct
Experiments using
and GPT-4
Execution Accuracy
30
were performed using the OpenAl API" All other experiments were done single NVIDIA A100 GPU with 80 GB memory. Our code is imple- mented in PyTorch (Paszke al., 2019) and makes use of the Hugginglace Transformers library (Wolf
25
20:
15
20
et al., 2020).
RedPayams-77 RedPajama instruct Models
ULMM-130 138
B Additional Information on MAGNIFICO
Figure 12: Average execution accuracy (1) of models base examples in MAGNIFICO across various prompt settings.
We provide examples for each of the 18 single- word form and 6 phrase form interpretations in MAGNIFICO in Table 4 and Table 5 respectively.
Execution Accuracy on Base Examples
Direct
60
10
B.1 Populating Tables with Edge Cases
40
Execution
The metric for evaluating the generated SQL queries in text2SQL benchmarks is execution accu- racy, which compares the output of the execution of the generated query with the ground truth. Since we are introducing new interpretations in existing databases, possible that the output of the cor- responding SQL query is trivial, like printing all values in the Table. Apart from this, it is possible that an incorrect SQL query leads to the ground- truth output because there are no edge case values present in the Table. To handle such cases, tomatically populate the tables by inserting new values that act as edge cases (Zhong et al., 2020).
20-
20.
LLaM6-278
LLaMA-2-13
Models
Figure 13: Average execution accuracy (1) of 2 and GPT-4 models on base examples in MAGNIFICO across various prompt settings.
C.3 Impact of Pre-existing Semantic Interpretation
Figure 18 provides the results for all models across all experimental settings.
Additional Experimental Results
C.4 Results for LLaMA-2 and GPT-4
The performance of LLaMA-2 and GPT-4 models base examples in MAGNIFICO can be seen in Figure Figure 13. Their performance across all experimental settings can be seen in Figure 14.
C.1 Performance Base Examples
The performance of models on base examples in MAGNIFICo can be seen in Figure 12. We found
the base text-to-SQL performance of RWKV-14B to be extremely low and hence do not experiment with
D Additional Related Works
it in other settings.
Word Acquisition
Lazaridou et al. (2021) analyse the temporal generalization capabilities of LLMs and showed that the perplexity increases when modelling text containing new words. There is also some related work in the domain of grounded language learning. Chevalier-Boisver et al. (2019) focus learning synthetic language which is subset of English. However, they do not carry out any systematic evaluation focused on word learning. Hill et al. (2021) propose an approach for fast-mapping, i.e.,
C.2 Impact of Description and Few-Shot Examples
Figure Figure 16 and Figure 17 illustrate the impact of providing description and few-shot ex- amples of the novel interpretation when the novel interpretation represented by a plausible. or an adversarial form respectively for all models.
"hatps//plaaffmmooppaa..omm

---

the ability bind novel non-sense word to an object in their RL framework. However, their framework and approach are specifically designed to cater to word learning, while we wish to evaluate the word learning abilities of general NLP models across various NLP tasks. Tsimpoukelli et al.
Offect pre-existing semantic Interpretation
Rubaive Porter Poftermance (50)
(2021) focus on using frozen pretrained models for learning words that only act as names of objects in images. We wish study word learning broader level, by considering more complex types of words and interpretations.
Driige woiiiin
inteoretation
Figure 14: Relative performance (1) of LLaMA-2 and GPT-4 models across forms of different types for all prompt settings.
Compositional Generalization
Many works in the past (Fodor and Pylyshyn, 1988; Hadley, 1994; Fodor and Lepore, 2002; Marcus, 2003; Calvo and Symons, 2014) have argued that artificial neural networks incapable of exhibit- ing systematic compositionality. However, recent successes of neural models (Bahdanau al., 2015; Vaswani et al., 2017; Devlin al., 2019) across various NLP tasks have revived this debate with a focus on investigating the presence and extent of compositional biases in models.
very different for inputs like, 'Find the prevalant age of students' or, 'What the number of students that do not have the prevalant last name?").
There have been many compositional gener- alization benchmarks proposed in recent years (Keysers et al., 2020; Yanaka et al., 2021), almost all of them illustrating deficiencies of neural models at generalizing compositionally. Many approaches have also been proposed to solve compositional generalization benchmarks (Li al., 2019; Lake, 2019; Gordon et al., 2020; Chen al., 2020; Andreas, 2020; Liu et al., 2020; Guo et al.,
Lake and Baroni (2018) investigated the com- positional generalization abilities of contemporary neural sequence models such as RNNs and LSTMs based on their performance on a synthetic bench- mark called 'SCAN'. Their conclusions were con- sistent with past work in that they found neural sequence models generalize poorly when tested systematically held-out novel combinations of words and phrases. Follow-up work by Kim and Linzen (2020) reached similar conclusions using their semantic parsing benchmark, 'COGS'.
2020; Akyurek and Andreas. 2021; Conklin et al., 2021; Liu et al., 2021). However, most of these approaches are task-specific and cannot be gener- ally applied for language processing. Moreover, LLMs achieve very high level of performance on compositional generalization benchmarks based on just few examples in-context (Drozdov et al., 2023). In this work, we seek to analyse the compositional generalization capabilities of LLMs more realistically, by grounding our evaluation to possible use case scenarios, for e.g. generating SQL queries for user inputs that require understanding novel interpretations from long conversation context.
While novel word learning has not been explic- itly studied in previous compositional generaliza- tion literature, some of the experiments carried out by Lake and Baroni (2015 8) and Kim and Linzen (2020) do implicitly assess the abilities of models to one-shot acquire a novel word. However, the words used in these experiments are of a primitive nature and have a context- inneeext-nddeeeddent direct mapping in the output space (for e.g.. in SCAN, models simply need to learn to map the input word 'jump' to its corresponding output token 'JUMP'). In our work, we broaden the scope to also understand how well models acquire more functional words, i.e., words that act over other words in context-dependent manner generate the output (for e.g., consider the interpretation frequett represented by the form prevalant in Table 1. The output looks
E Example Prompts
Figure 19 shows example of a prompt used to generate long form dialogue using GPT-4. Fig- ures 20, 21, and 22 show examples for the DDecerip- tion', "Few-shot', and 'Dialogue' prompt types respectively.

---

Relative Performance Plausible
LL 78
MPT-78
RedFajama
LiaMA 138
LLaMA. 30B
100
Alpaca
MPT Instruct
RedPajama Instruct
StarCoder
GP1:3.5 lurbo
Relative Performance (%)
Direct
Description
Few-Shot
Prompt Setting
Figure 15: Relative performance (1) of all models MAGNIFICO various prompt settings when the TOKEN is plausible word.
Relative Performance Foreign
LLaMA-70 78
MPT-7B MPT-78-Instruct
RedPajama RedRajama-instrur
LLaMA- 139
LLaMA-30B
Relative Performance (%)
Alpaca
StarCoder
60
40
20
Direct
Dascription
Few-Shot
Prompt Setting
Figure Relative performance (1) all models on MAGNIFICO across various prompt settings when the TOKEN is foreign word.

---

CATEGORY
INTERPRETATION EXAMPLES
Inper: What are the name, latinade, and city of the station with the baseline larinde??
Minimum
Outpet: SELECT nane, lat, city FROM station ORDER BY LIMIT
Maximum
Inpot:
Which film covered rental nite? And what the rate?
Output: SELECT ticle, rental_rate FROM film GRDER BY rental_rate DESC LIMIT 1
Input:
representative price flights LA to homolutu?
Averaee
Output: SELECT AVG(price) FROM flight WHERE origin "Los Angeles' AND destination
Basic Operations
'honolulu'
Inpot: What is the nccumulated employee number for each office of professons?
Sum
Outpet: SELECT prof_office, SUN(eno__mu) FROM professor GROUP prof_office
Count
Inpot:
Show magninade unique transaction types.
Output: SELECT COUNT (DISTINCT transaction_ FRON transaccionsssssiinn
Inpet: Display the and first name of stadents with the prevalent major.
Most-froquent
Output: SELECT Sex Fraee FROM Student WHERE Major IN (SELECT Major FROM Student
GROUP BY Major ORDER BY COUNT DESC LIMET 1)
Inpot: Which major has runnerupp number of stadents?
Second maximum
Output: SELECT najor FROM (SELECT najor (OUNT()) AS FROM student GROUP najor ORDER COUNT(H) DESC LINIT AS t_tab ORDER BY t_teb.t_prop t.prop LIMIT
Input:
players who get satisfactory points?
avverge
Output: SELECT nane FROM player MHERE points (SELECT AVG(points) FROM pleyer)
Subquery
buuuue bused Operations
Inpot: How many customers are absent from having an account!
Value not present
Output: SELECT COUNT(+) FROM custoners MHERE custoner__e .id NOT IN (SELECT customer_id accounts)
Inpet: Which songs dominate those with rating below in terms of resolution?
Output: SELECT f_id FROM song MERE resolution (SELECT MAX(resolution) FROM song
(conditionall)
MIERE rating
Inpat: the names heavy comrses by their tirles credits.
4 credit courses
Output: SELECT title FROM course WIERE credits ORDER BY title, credits
Salary more than 30000
Input: Display information on those overpaid employees who joined after 1st April, 1995.
Oulput: SELECT FROM employees WIERE sulary 19986 AND hire dare '1995-84-01'
Input: What are the addresses of pure science subject departments?
Physics and Biology departments
Oulpul: SELECT dept_address FROM degar trent MERE dept__nane IN ("Physics', 'Btology") )
Value conditioned
Input: How many aggressivee players have more than 1800 hours of training?
Yellow card
Output: SELFCT (o)NT(*) FROM player MHERE his 1880 AND yeard "yes"
Input: How many trips did not end in tech town??
Mountain View and Palo cities
Output: SELECT COUNT(+) FROM
AS e1 JOIN station t2.id MERE t2.city NOT IN ('Hountain tain View", 'Palo Alto')
ON sttatinsstaainn_i
Concatenarion of last
Inpat: How many students are there with "gE" alias?
name
Outpat: SELECT (*) FROM student MHERE Inane I| fname "SgFX'
Subraction end
Inpet: What unique joh ids history when
than
Columa Operations
and start dates
Output: SELECT DISTINCT job_id FRON jistory WHERE end date start_date
Inpat: How many courses prenequisite roquirement-id
100000?
Product of Course
Output: SELECT CO)NT(*) FROM course MIERE course_id IN (SELECT course_id FROM prereq MHERE
and Pretoquisite IDs
prereq_id 106000)
Table 4: Examples of all single-word novel interpretations used in MAGNIFICO. Illustrated examples use a plausible English form.

---

INTERPRETATION
EXAMPLES
Number
Input: Find
price for track. Display outputs the name within system length constraints. Outpul: SELECT AVG(unitprice) FROM track MERE LENGTI (Nare)
Inpot: Find the campuses whose cumpus in first onder ouilier range. campust ee
greater difference avenge and the standard deviation
Output: SELECT campus FROM feu_fees
(SELECT
AVli(carpusTie)
STDEY (canpusfee) FROM cse_)
Value loss than average
Inpat:
Outpet: SELECT roomnnee FROM roons WHERE baseprice SELECT Abosease FROM roons)
cost lying community
spectrum.
Input: details employees who manage department, Show outpuss only for those employess hired during the momths of union labour strike.
Hire date July August 1987
Output:
SELECT DISTINCT FROM employees AS tl JOIN departnnntts AS ON L1 dl.depar tnen._id
hire_date "1987-87-01" hire_date "1987-09-01" t1. e1.erployoe... oyee_id = t2. nanager_id
depprtprrenn teeet_id
Inpal:
physicions never any appointmen.. Output: SELECT nome FRON physician EXCEPT SELECT t2. FRON appointment AS t1 JOIN physician tl.physician
buard
Physiciums that are not an intern
erployee1d WHERE position
('Staff Internist')
Input: How many biking ussociation compliant stations are Mountain View?
Number of docks greater than 19
Output: SELECT COUNT(>) FROM station MERE dock_count be AND city 'Mountain View
Table Examples all novel interpretations represented by a phruse English words used MAGNIFICO.
Relative Performance Adversarial
78
MPT-78 MPT-7B-lintruct
RedPajama RedPajama-Instruct
LLaMA 138
LLaMA-30B
Relative Performance (%)
Alpaca
StarCoder
GPT-3.5-Turbo
40
20
Direct
Description
Few-Shot
Prompt Setting
Figure Relative performance (1) of all models on MAGNIFICO across various prompt settings when the TOKEN is adversarial word.

---

Effect of pre-existing semantic interpretation
Direct
Description
Few-Shot
LLaMA-7B
MPT-78-Instruct
LLaMA-138
StarCoder
Alpaca
RedPejama-78
LLaMA-308
GPT-3.5 o-Tur.5-Turbo
RedPajama
Relative Performance (%)
Plausible
Foreign Pre-existing Semantic Interpretation
Adversarial
Vigure 18: Relative performance (1) of all models across forms of different types for all prompt settings.

---

System Prompt:
You are DialogueGPT a tool to generate realistic long-form multi-turn dialogues based on the si situation provided.
User Prompt:
You are given a database schema. Examples of the data in each of the tables is provided:
CREATE TABLE IF NOT EXISTS "departments" ( 'DEPARTMENT ID" decimal (4,0) NOT NULL DEFAULT '0', "DEPARTMENT _NAMEE v30) NOT NULL, "MANAGER_ID decimal (6,0) DEFAULT NULL,
"LOCATION_ID decimal (4,0) DEFAULT NULL, PRIMARY KEY ("DEPARTHENT_IDD)
/*
3 example rows:
SELECT FROM
departments" LIMIT 3;
DEPARTMENT_ID
DEPARTMENT_ME NAME MANAGER_ID
LOCATION
10
Administration
200
1708
Marketing
201
1800
Purchasing
114
1708
CREATE TABLE IF NOT EXISTS 'jobs"
"JOB_ID v10) NULL DEFAULT "1
TITLE" v3rchar(35) "MIN_SLARR decimal(6,0) 8) DEFAULT NULL, "MAX_ SALARY decimal(6,0) 8) DEFAULT NULL, PRIMARY KEY ('JOB_ID")
NULL,
example rows:
SELECT FROM "oobs" LIMIT 3;
JOB_ID JOB_TITL MIN MIN_SALAN SALARY
MAX_ SALARY
AD_PRES President
20000
40080
AD_VP Administration Vice President
15008
30000
AD_ASST Administration Assistant
3080
5008
*/
Generate a 20-turn dialogue between two users of this database. Somewhere near the start of the conversation, user1 says that based on the schema, some people are overpaid. In response to user2 asking what user1 means when they say someone is overpaid, user1 will casually mention that according to them, anyone that earns salary more than 30, 000 is overpaid. The rest of the conversation should make no mention of overpaid. The conversation should not include SQL queries.
Figure 19: Prompt used for generating long form dialogues using GPT-4,

---

CREATE TABLE IF NOT EXISTS "departments" ( DDEPARTMENT ID' decimal (4,0) NULL DEFAULT '0', DEPARTMENT NAME varchar (30) NOT NULL, MANAGER decimal(6, 0, DEFAULT NULL, "LOCATION_ ID" decimal (4,0) DEFAULT NULL, PRIMARY KEY ('DEPARTHENT_IDD)
example rows:
SELECT FROM
"departments" LIMIT 3;
DEPARTMENT_ID
DEPARTMENT_ NAME MANAGER_ID
LOCATION ID
10
Administration
200
1708
20
Marketing
201
1890
30
Purchasing 114
1788
*/
/*
CREATE TABLE IF NOT EXISTS "jobs"
"J"BIID" v1r) NOT NULL DEFAULT
JOB "OT_E v35) NOT NULL, "MIN_SALAA MILASSA d6cimal 8) DEFAULT NULL, "MAX_SALARY decimal(6,0) DEFAULT NULL, PRIMARY KEY ('JOB_ID") **
3 example rows:
SELECT FROM jobss' LIMIT 3;
JOB_ID JOB_TITL
SALARY
MALASAL
AD_PRES President
20000
48800
AD_VP Administration Vice President
15800
30000
AD_ASST Administration Assistant
3000
6800
Using valid SQLite, answer the following questions the tables provided above.
The word 'overpay' refers to those salary more than 30000.
what all the information about overpaid employees hired before April 2, 1995? SELECT
Figure 20: Example prompt for Create Table + Select where the prompt contains a description of the novel interpretation.

---

CREATE TABLE IF NOT EXISTS "regions" REGION ID decimal (5,0) NOT NULL, "REGION NAME" varchar DEFAULT NULL, PRIMARY KEY ('REGION_ID")
*
create table prereq
(course_id
varchar(8),
prereq_id
varchar (8).
primary key (course_id, prereq_id), foreign key (course_id) references course (course_id)
delete cascade,
foreign key (prereq_id) references (course_id)
example rows:
SELECT FROM prereq LIMIT 3;
course_is prereq_id
376
130
768
169
403
345
Using valid SQLite, answer the following questions for the tables provided above.
on what dates were overpaid employees without the letter m in their first names hired? select hire_date from employees where salary > 30008 and nirst_name not like 'Xmx'
give the distinct department ids of department impatsments in which manager is in charge of one or more overpaid employees?
select distinct department_id from employees where salary 30000 group by department_id manager_id having (d) >=
find the names of all overpaid instructors in comp. sci. depar tment select name from instructor where dept_name 'Comp. Sci. and salary > 38000
what is all the information about overpaid employees hired before April 1995? SELECT
Figure 21: Example prompt with few-shot examples for the novel interpretation represented by the plausible form 'overpaid' followed by the query input.

---

User1: Hey, I've been working with this database schema and wanted discuss of the tables and their relationships.
User2: Sure, I'm familiar with schema. What do you know?
User1: noticed there's table for employees th salary column. which employees would you consider to overpaid
User2: Well, that's subjective depends on various factors. But the sake this conversation, let's say anyone earning salary more than 38,00 000 overpaid. What do you want to discuss in relation to schema?
conversation containing 2000 words
*/
User2: The benefits_enrollmeee enrollment table has a foreign key EMPLOYEE_IID indicating which employee the benefit enrollment is associated with. It also has colurns for the benefit name, enrollment date, and status.
User1: Suppose you are given the following schema:
CREATE TABLE IF NOT EXISTS "regions" "REGION_IN ID" decimal (, NOT NULL, "REMI varchar (25) DEFAULT NULL, PRIMARY KEY ("REGION_ID")
/N
3 example rows:
SELECT FROM "regions" LIMIT 3;
REGION_ID REGION_NAME
EuropeAr
2
Americas\r
Asia'r
*/
example rows:
SELECT FROM 'locations" LIMIT 3;
LOCATION_ID
STREET_ADDRES POSTAL_CODS CCDE
CITY
PROVINCE
COUNTRY_ID
1080
1297 Via Cola Rie 989
1100
93091 Calle della Testa 18934
Venice
1200
2017 Shinjuku-ku
Tokyo
Tokyo Prefecture
Using valid SQLite, answer the following question with the corresponding SQL query: what is all the information about overpaid employees hired before April 2, 1995?
User2: SELECT
Figure 22: Example prompt which involves long form dialogue containing the description of the novel interpreta- tion. Note that the truncated section the dialogue has over 2000 words.


Reviews:
[{"review_id": "rhBgiaUnhu", "reviewer": "Reviewer_wHLA", "summary": "", "questions": "Question A: How do you define what a \"novel\" interpretation is? \n\nQuestion B: How did you choose the interpretations that you want to include in the the benchmark? \n\nQuestion C: How many annotators verified the ability of gpt-4 in generating conversations? \n\nQuestion D: what is the motivation behind using foreign forms?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your time spent in reviewing our paper and providing helpful comments. We are glad that you recognise the utility of our benchmark and find our analysis about the capabilities and limitations of LLMs interesting. Please see our response below to specific comments:\n\n*Restricted the analysis to one task only which is not clearly motivated*\n\nWe chose the text-to-SQL semantic parsing task because:  \n(1) It is a fundamental NLU task that has been widely studied.  \n(2) It illustrates real-world scenarios in which an automated system must learn novel interpretations.  \n(3) Many existing LLMs are able to achieve good performance on the task via in-context learning.\n\n*Results are scattered between the body and appendix and the reader must refer to the appendix quite often*\n\nWe believe that the current body of the paper cohesively presents all main results and discussions along with necessary background and details. We used the appendix to provide auxiliary information regarding the exact prompt formats and dataset creation. Results of some models such as RWKV and RedPajama as well as some additional related works were shifted to the appendix because of lack of space - we shall add these to the main body in the final version.\n\nAns A. A “novel” interpretation in the context of our work refers to a part of the language input that has to be interpreted in a specific well-defined manner (often in a way that the LM would not have seen during pretraining) when generating the output SQL query.\n\nAns B. We defined broad categories of interpretations based on patterns that were repeated in the SQL queries in existing text-to-SQL datasets (See Table 1). We then defined each interpretation as a specific instantiation of one of these categories, influenced by the quantity of data that could be generated for it.\n\nAns C. All GPT-4 generated conversations were manually reviewed by one author.\n\nAns D. Our experiments with foreign form illustrate a scenario where a system needs to learn a word that is not present in its vocabulary (for e.g., a new word coined after the system was trained) and therefore has no existing semantic disposition for that token.\n\nIn light of our response above and other reviews, could you kindly reconsider your evaluation?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Providing a benchmark to test the ability of LLMs in learning novel interpretations. \n- Interesting analysis about the capabilities and limitations of LLMs in multiple settings", "reasons_to_reject": "- Restricted the analysis to one task only which is not clearly motivated. \n- Results are scattered between the body and appendix and the reader must refer to the appendix quite often.", "questions_to_authors": "Question A: How do you define what a \"novel\" interpretation is? \n\nQuestion B: How did you choose the interpretations that you want to include in the the benchmark? \n\nQuestion C: How many annotators verified the ability of gpt-4 in generating conversations? \n\nQuestion D: what is the motivation behind using foreign forms?", "paper_topic_and_main_contributions": "The authors studied the ability of LLMs to acquire novel interpretations using in-context learning. They considered three different scenarios (In the context of text-to-sql semantic parsing). In the first scenario, they considered providing a description of the novel term. In the second scenario, they illustrated the term via few shot samples. In the third scenario, they considered understanding the interpretation from long-from dialogue. The contributions of the paper are: \n- magnifico benchmark: a benchmark in which each sample is a text-to-sql semantic parsing problem that require models to understand one or more novel interpretations \n- Analysis on 11 different LLMs in which they showed that models are highly capable in learning novel interpretations. \n-  Analysis on the limitations of LLMS where they showed that such models fail at learning multiple novel interpretations simultaneously however, they fail at learning multiple novel interpretations at once.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "nknVV9PlMr", "reviewer": "Reviewer_jQoo", "summary": "", "questions": "A: Line 336, how does the selection of few-shot examples affect the results? How large is the variance based on your experience?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your time spent in reviewing our paper and providing helpful comments. Please see our response below to specific comments:\n\n*it is still not very clear why SQL is chosen as the task*\n\nOur main reasons to work with the text-to-SQL semantic parsing task were:  \n(1) It is a fundamental NLU task that has been widely studied.  \n(2) It illustrates real-world scenarios in which an automated system must learn novel interpretations.  \n(3) Many existing LLMs are able to achieve good performance on the task via in-context learning.\n\n*how does the selection of few-shot examples affect the results? How large is the variance based on your experience?*\n\nOur initial analysis did not reveal much variance in the results when using different sets of few-shot exemplars. We did not run a comprehensive quantitative analysis due to time and resource constraints during submission, but plan to include that in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  This work collected a new benchmark dataset that can test for LLMs' in-context learning ability on new meanings of words. The task is useful as it can be an indicator of LLMs' in-context learning ability in general, and it has execution accuracy as a clear metric for performance. The method to collect the data is well articulated so it can scale. A bonus point is that the author checked all the samples collected.\n2. The analysis is very through and answered a lot of questions regarding in-context learning, which includes introducing new info with text description, few-shot examples, and dialogue. The paper also discusses the impact of description and LLMs' semantic info obtained during the pretraining, etc. \n3. The paper is well written, with plentiful examples in the appendix showing what is done in each stage. The illustrations are clear and visually pleasing. The experiment section is structured in a way to answer different research questions, which makes the results very clear.", "reasons_to_reject": "1. Although mentioned in section 3, it is still not very clear why SQL is chosen as the task, as the justification is quite general. Despite being easy to evaluate, SQL task restricts the range of new interpretations as it has to be a valid condition for a query. ", "questions_to_authors": "A: Line 336, how does the selection of few-shot examples affect the results? How large is the variance based on your experience?", "paper_topic_and_main_contributions": "This work collects a text-to-SQL dataset containing words/phrases that have a new meaning(interpretation), and uses it to conduct a very comprehensive study of how well LLMs'  generalize to the novel interpretation under the in-context learning setting. \n\nThe author come up with 24 interpretations, e.g. the word \"prevalent\" can have an interpretation as \"Most-frequent\".  Based on those interpretations, the author retrieve samples from Spider containing content that can apply the new interpretation with three different methods, and also manually verify the dataset obtained. \n\nThe author then tests various LLMs on the task in zero-shot, few-shot, and a setup including additional text description. A list of research questions is proposed and answered based on those results. \n\nMain Contribution\n\n1. Created a dataset that can benchmark LLM's ability on generalizing to new definitions of words. As the generated SQL results can be evaluated in terms of execution accuracy, it has a more clear definition of performance than free-text generation. Manual check also ensures its quality. \n\n2. Well-conducted analysis on how different in-context learning settings affect the model's ability to generalize to novel interpretation.  \nFurther explored other experiment settings, such as multiple new interpretations in one sentence, new interpretations through dialogue, and multiple new interpretations in one sample. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Very minor point for Figure 1, the different forms, especially foreign form and adversarial form, are mainly just for results in section 5.2 and maybe add extra confusion and not deserved to be put in Figure 1."}}, {"review_id": "RLESagmGoi", "reviewer": "Reviewer_9Hzn", "summary": "", "questions": "1. Why didn't you evaluate the GPT-4 model with the MAGNIFICO tool? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your time spent in reviewing our paper and providing helpful comments. Please see our response below to specific comments:\n\n*this work should have compared their results with the recent GPT-4 model*\n\nAt the time of submission, we were unable to run **evaluation** experiments with GPT-4 because of rate and resource limitations (experimenting on multiple prompt and form settings becomes very expensive). We prioritized the use of GPT-4 for the more important task of helping **create the MAGNIFICo benchmark**.\n\n*Their tool is built to assess the LLMs while only covering conversational large language models*\n\nGPT-3.5-turbo is the only conversational LLM covered in our analysis. All other models that we experimented with (StarCoder, LLaMA, RedPajama, MPT, and RWKV) are general LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They present a novel evaluation suit called MAGNIFICO which is applied to assess the in-context learning capabilities of LLMs. This tool could be valuable to other researchers in the field.\n\n2. They extensively investigate how LLMs interpret and understand new words and phrases in different context settings, providing useful insights about their limitations and capacities.\n\n3. The paper identifies key areas where LLMs struggle, such as understanding unfamiliar words or multiple novel words simultaneously. Identifying these areas could guide future improvements in the development of LLMs.\n\n4. This study evaluates the performance of different LLMs which leads to a comprehensive understanding of LLM capabilities.", "reasons_to_reject": "1. Most importantly, as mentioned in the \"limitation\" section, the number of test data used for some experiments is inadequate to conclude or make a strong analysis, which weakens the analysis for those subsections. Since the data generation is automated, extending the dataset is necessary. \n2. While clearly having access to  GPT-4, this work should have compared their results with the recent GPT-4 model. Their work would have been much more substantial with the latest models in their results. \n3. Their tool is built to assess the LLMs while only covering conversational large language models. \n4. The conclusion in line 542 regarding 'instruction fine-tuning' doesn't have clear supporting evidence and needs more clarification.", "questions_to_authors": "1. Why didn't you evaluate the GPT-4 model with the MAGNIFICO tool? ", "paper_topic_and_main_contributions": "This work investigates the ability of LLMs to learn and comprehend novel interpretations using in-context learning within the limitations of their knowledge cutoff and the high cost of repeated fine-tuning.\n\nThe authors introduce a new evaluation tool called MAGNIFICO, developed within a text-to-SQL semantic parsing framework. This tool is designed to incorporate diverse tokens and prompt settings to imitate the complexity of real-world scenarios.\n\nThe authors conducted experimental studies on MAGNIFICO to assess the LLMs' capacity for understanding new interpretations from natural language descriptions, long conversations, and compositional generalization. The findings reveal that LLMs still need to improve in understanding unfamiliar words or combining multiple novel interpretations simultaneously in the same example.\n\nMoreover, the authors analyze the semantic predispositions in LLMs and explore the impact of recency bias when information is presented in lengthy contexts. This suggests that recent details may have a more significant influence on the models' understanding and interpretation.\n\nThe authors conclude that there is a need for further development and improvement in these areas, especially in handling the complexity and novelty of interpretations, suggesting that their work can serve as a basis for future research on improving LLMs' in-context learning capabilities.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Finetuning and Fine-tuning should have singular writing. Lines 48, 388 542... "}, "extra_text": {}}, {"review_id": "BKc4Fgazsi", "reviewer": "Reviewer_AoAT", "summary": "", "questions": "Question A: It was not clear to me how the Descriptions have been generated. Is this actually explained in the paper and I missed it? If it's not, can you please explain?\n\nQuestion B: In the case where the expression (form) is a phrase rather than a single word, is the distinction between plausible, nonce, and adversarial also made?\n\nQuestion C: What was the cost of using models that are not freely available to make this work possible (e.g., using GPT-4 to generate the dialogues or analysing the output of a non-free models)? (I find this relevant for assessing the limitations and ethical consequences of the work)\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your time spent in reviewing our paper and providing helpful comments. Please see our response below to your questions:\n\nAnswer A: The descriptions for each interpretation were manually written by one of the authors and verified by another.\n\nAnswer B: No, this distinction is only made for single-word forms.\n\nAnswer C:  \nWe used GPT-4 for:  \n(1) constrained paraphrasing: we estimate usage of ~50,000 tokens, which would amount to a cost of not more than \\\\$3.  \n(2) generating dialogues: we estimate usage of ~600,000 tokens for creating the 125 dialogues, which would amount to a cost of not more than \\\\$30.  \nWe used GPT-3.5-Turbo for evaluation:  \nWe estimate ~2M output tokens, which would cost ~\\\\$4 and ~30M input tokens, which would cost ~\\\\$45.  \n\nIn total, we estimate the OpenAI API costs to be ~\\\\$80-\\\\$100.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths: \n\n- Novel way to re-frame the problem of word learning and compositional generalisation in LLMs \n- Creation of an evaluation dataset with clear and interesting design choices\n- Solid experimental design\n- Extensive analysis and comparison of a substantial amount of LLMs that differ with respect to model size and the use or not of instruction training\n- Clearly written\n", "reasons_to_reject": "The paper naturally has some weaknesses, but no major reasons for rejection. See the box on \"Presentation Improvements\" below for a few things that should be improved.\n", "questions_to_authors": "Question A: It was not clear to me how the Descriptions have been generated. Is this actually explained in the paper and I missed it? If it's not, can you please explain?\n\nQuestion B: In the case where the expression (form) is a phrase rather than a single word, is the distinction between plausible, nonce, and adversarial also made?\n\nQuestion C: What was the cost of using models that are not freely available to make this work possible (e.g., using GPT-4 to generate the dialogues or analysing the output of a non-free models)? (I find this relevant for assessing the limitations and ethical consequences of the work)\n", "paper_topic_and_main_contributions": "This paper investigates the ability of LLMs to adapt to new expressions/meanings through in-context learning. The task considered is mapping natural language text to an SQL semantic representation (i.e., to a query that could be executed on a data base). The paper makes two main contributions: the creation of a dataset to study this problem and an extensive analysis of LLMs using this dataset. \n\nThe authors create an evaluation dataset (MAGNIFICo) building on an existing text-to-SQL benchmark (Spider by Yu et al. 2018). They define a set of 24 meanings and map them to three types of textual form (word of phrase): an existing word for which the new meaning is plausible, a nonce word, and an existing word with contrary meaning (adversarial). \n\nThis dataset is then used to evaluate a range of LLMs in an in-context learning setup that considers three types of prompts: Direct (zero-shot introduction of the new interpretation within the query in natural language), Description (the prompt also includes a natural language definition of the new interpretation), and Few-shot (the prompt includes 5 examples of language-SQL pairs illustrating the new interpretation, without any explicit definition). In addition, for a limited number of instances, the prompt includes a description of the new interpretation within the context of dialogue between two simulated users. \n\nThe results show that most LLMs under scrutiny can recover novel interpretations well in the Description and Few-shot setups when the forms are plausible, and that there is a decrease in generalisation ability particularly when the forms are adversarial. When the description is included as part of a dialogue, the performance is lower (with plausible forms) and exhibits recency bias (the results improve when the description is introduced near the end of the dialogue/context window). Finally, there results show that models struggle when multiple interpretations are introduced within a single query and when the forms consists on multi-word phrases rather than single words.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The related work section is too concise. In particular, I miss some more background on text-to-SQL. (I realised later that there is an appendix with further related work, which is not mentioned in the main paper. Space constraints are not an excuse here: the paper does not use space very efficiently.)\n \n- The Spider dataset is not described and this would help to understand better the starting point for the proposed dataset. \n\n- At the beginning of the Results section, I would find it natural to comment on the results obtained in the Direct zero-shot setting. \n\n- The font of pretty much all figures is too small. I wasn't able to read many of the legends and text on a print out of the paper. \n"}}]

Author rebuttal:

Meta-review: This paper focuses on the ability of LLMs to learn nover interpretations of word and phrases via in-context learning. Specifically, they introduce a new text-to-SQL semantic parsing  dataset called MAGNIFICo which contains diverse tokens and prompt settings. Theirexperiments indicate that LLMs do resonably well on this task but there is still room for improvement.

Overall, reviewers agree that the task is interesting and the paper is well-written with analysis and experiments. While it does have some drawbacks ((1) only one task for studying a broader phenomenon, and (2) fewer data points for some settings), the contributions are above the threshold for acceptance.
Decision: Accept-Main