How Do In-Context Examples Affect Compositional Generalization?
Year: 2023

Abstract:
Compositional generalization—understanding unseen combinations of seen primitives—is an essential reasoning capability in human intelligence. The AI community mainly studies this capability by fine-tuning neural networks on lots of training samples, while it is still unclear whether and how in-context learning—the prevailing few-shot paradigm based on large language models—exhibits compositional generalization. In this paper, we present COFE, a test suite to investigate in-context compositional generalization. We find that the compositional generalization performance can be easily affected by the selection of in-context examples, thus raising the research question what the key factors are to make good in-context examples for compositional generalization. We study three potential factors: similarity, diversity and complexity. Our systematic experiments indicate that in-context examples should be structurally similar to the test case, diverse from each other, and individually simple. Furthermore, two strong limitations are observed: in-context compositional generalization on fictional words is much weaker than that on commonly used ones; it is still critical that the incontext examples should cover required linguistic structures, even though the backbone model has been pre-trained on large corpus. We hope our analysis would facilitate the understanding and utilization of in-context learning paradigm.

heading: 1 Introduction
text: Compositional generalization is an essential capability of human intelligence. It means to understanding and producing novel expressions by recombining known components in language (Chomsky, 1957; Montague, 1974; Fodor and Lepore, 2002). Taking examples in Figure 1, after learning the combination “baby in a room”, human intelligence can easily generalize to “Jackson in a room”. On exploring this human-like capability
∗Work done during an internship at Microsoft Research.
bination in COFE. The phrases modify the objects in examples but are recombined with subject in test input.
in deep learning models, several benchmarks such as SCAN (Lake and Baroni, 2018), CFQ (Keysers et al., 2019) and COGS (Kim and Linzen, 2020) have been proposed based on semantic parsing1 tasks. In these benchmarks, the training set cover all the primitives while lacking certain combinations, and the test set focuses on these missing combinations. By fine-tuning generic neural models on these benchmarks, much work reported that these models exhibit poor compositional generalization (Furrer et al., 2020; Shaw et al., 2021; Bogin et al., 2022).
Recently, in-context learning with large language models exhibits impressive performance on various tasks (Brown et al., 2020; Rae et al., 2021; Wei et al., 2022). By conditioning on few-shot incontext examples, the pre-trained language model, with extremely large model size and pre-trained
1Semantic parsing means translating natural language (NL) expressions into semantic representations (i.e., logical forms).
11027
corpus, can perform downstream tasks without any update on pre-trained parameters.
Behind the impressive performance of in-context learning, we are curious whether this prevailing paradigm can take a step towards compositional generalization. To investigate this, we first take an initial exploration: for each test case in COGS, we select in-context examples from its training set and ensure that all primitives in each test case are covered by the equipped in-context examples. Our initial exploration suggests that compositional generalization can be easily affected by in-context examples: with only covering primitives, davinci 175B lags behind fine-tuned GPT2-Large with 24.2% accuracy (similar to the observation in Qiu et al. (2022)); with also covering some local structures (inspired by Bogin et al. (2022)), davinci outperforms fine-tuned GPT2-Large with 3.9% accuracy. Based on these initial observations, we raise and investigate the question: How do in-context examples affect compositional generalization?
We construct the test suite COFE (based on COGS) to facilitate our systematic investigation. Taking the coverage of primitives as a basic principle in COFE, we further define and inject three factors in selecting in-context examples: similarity, diversity, and complexity. Similarity is considered as the matching of hidden structures behind concrete expressions. Diversity reflects whether the context presents repeated patterns or not. Complexity portrays the amount of information contained in each example. By controlling these factors in constructing COFE, we can systematically investigate how would in-context examples influence the performance on compositional generalization.
Our experiments demonstrate that all three factors matter for in-context compositional generalization. We leverage six large language models in GPT series: davinci, code-cushman-001, codecushman-002, text-davinci-002, text-chat-davinci002, and code-davinci-002. The observations are consistent across models: to better perform compositional generalization, all backbone models prefer in-context examples with higher structural similarity to the test case, higher diversity among different examples, and lower complexity in each individual example. Furthermore, beyond the influence from these factors, in-context compositional generalization still faces two challenges. One is that in-context learning has difficulty recombining fictional words (e.g., random tokens) rather than com-
monly used ones. The other one is that in-context examples are still required to cover the linguistic structures in NL expressions, even though the backbone model has been pre-trained on large corpus.
Our contributions are three-fold: 1) to answer the research question posed, we investigate three factors in selecting in-context examples and draw consistent conclusions across models; 2) we construct COFE to conduct our systematic investigation, and will release it to facilitate further exploration of in-context compositional generalization; 3) we also point out two remaining challenges that in-context learning still struggles to handle. We hope our analysis would provide insights on how to select proper in-context examples, and to shed light on the future research of in-context compositional generalization. COFE is publicly available at https://github.com/microsoft/Contextua lSP/tree/master/cofe.

heading: 2 In-Context Compositional Generalization
text: In-context compositional generalization refers to understand and produce novel combinations through recombining the building blocks presented by in-context examples. We first introduce some basic settings for testing this desired capability, then show our initial observations.

heading: 2.1 Principles for Measuring In-Context Compositional Generalization
text: To measure in-context compositional generalization under a test suite, each test case and its equipped in-context examples should satisfy two principles.
• Combination held-out principle: to test generalization on certain combinations, incontext examples should exclude these combinations while test cases contain them.
• Primitive coverage principle: the primitives contained in each test case should be fully covered by in-context examples. Primitives are the minimum indivisible units in expressions. In this work, we mainly consider primitives as lexical items (e.g., the noun “baby” and the verb “observed” in Figure 1).
We say that a model exhibits in-context compositional generalization if it performs well on a test suite that satisfies these two principles.
Category In-Context Examples Test Case Illustration of Combination
Primitive
Substitution
Primitive
Structural Alternation
Phrase
Recombination
Longer Chain
Deeper Nesting
input: shark output: SHARK input: A girl drew the boy . output: DRAW ( GIRL , BOY , NONE )
input: The shark drew a boy . output: DRAW ( SHARK , BOY , NONE )
𝐗𝑁𝐗𝐿 + 𝐗𝑆1 input: The goose baked . output: BAKE ( GOOSE , NONE , NONE ) input: A teacher noticed a chicken . output: NOTICE ( TEACHER , CHICKEN , NONE ) input: A teacher baked the chicken . output: BAKE ( TEACHER , CHICKEN , NONE ) + 𝐗𝑆1 𝐗𝑆2
input: Logan mailed Stella the cake in the pile . output: MAIL ( LOGAN , IN ( CAKE , PILE ) , STELLA ) input: The goose rolled a baby in a room . output: ROLL ( GOOSE , IN ( BABY , ROOM ) , NONE ) input: A visitor in the pile rolled a resident . output: ROLL ( IN ( VISITOR , PILE ) , RESIDENT , NONE )
𝐗𝑆1 +
𝐗𝑆2
input: The boy admired that Noah confessed that \
Emma was given a cookie .
output: ADMIRE ( BOY , NONE , NONE ) \
CCOMP CONFESS ( NOAH , NONE , NONE ) \ CCOMP GIVE ( NONE , COOKIE , EMMA )
input: The girl wished that a crocodile declared that \
the boy admired that Emma liked that \ Evelyn was passed a drink .
output: WISH ( GIRL , NONE , NONE ) \
CCOMP DECLARE ( CROCODILE , NONE , NONE ) \ CCOMP ADMIRE ( BOY , NONE , NONE ) \ CCOMP LIKE ( EMMA , NONE , NONE ) \ CCOMP PASS ( NONE , DRINK , EVELYN )
𝐘𝑆𝑛 𝑛 times recursion
input: Noah appreciated a girl in a house \
beside the chair .
output: APPRECIATE ( NOAH , \
IN ( GIRL , \
BESIDE ( HOUSE , CHAIR\
) ) , NONE )
input: A dog painted the girl beside the chair \
in a house beside a road on a dish .
output: PAINT ( DOG , \
BESIDE ( GIRL , \
IN ( CHAIR , \
BESIDE ( HOUSE , \
ON ( ROAD , DISH\
) ) ) ) , NONE )
𝐘𝑆𝑛 𝑛 times recursion
𝐗𝐿 𝐗𝑁
Figure 2: Five categories of aiming combinations. The key parts in combinations are marked with underlines and colors (blue in NL-side and purple in code-side). The last column follows the notations defined in Section 3.2.

heading: 2.2 COGS (Under In-Context Learning)
text: COGS is a compositional generalization benchmark designed for the fine-tuning paradigm: based on a semantic parsing task, the training set of COGS covers all primitives in this task, while several combinations of primitives in the test set are excluded from the training set. We term these excluded combinations as aiming combinations.
We measure in-context compositional generalization based on COGS, by converting it from the original fine-tuning paradigm to the in-context learning paradigm. For each COGS test case, we select in-context examples from the training set B, ensuring that the two principles are satisfied. Note that, for each test case, there are usually different collections of in-context examples satisfying the two principles. Our basic setting is to use a random one among them, and we show that this casual strategy could lead to an underestimation of in-context compositional generalization (Section 2.3).
To facilitate testing on more complex logical forms, we reconstruct some target-side clauses from the chain structure into the nested-function format (illustrated in Figure 2). This reconstruction follows An et al. (2023) and is similar to the conversion from Lambda calculus to FunQL in Geo domain(Zelle and Mooney, 1996; Kate et al., 2005; Zettlemoyer and Collins, 2012). Moreover, to improve human readability, we omitted two types of details: the special marker for definite descriptions and the Skolem constants. These details do not affect the testing of compositional generalization.
Apart from these omitted details, the logical forms in COFE unambiguously represent the main semantics in the domain of COGS, such as semantic roles, modifications, and orders among clauses and modifications. More details about COFE logical forms are contained in Appendix A.
Categories of aiming combinations. The aiming combinations in COGS can be divided into five categories, of which two are low-level combinations (i.e., focusing on specific primitives) and three are high-level combinations (i.e., focusing on high-level structures), illustrated in Figure 2.
• Primitive Substitution (PrimSubs): Compose a primitive (e.g., “shark”) with a grammatical role (e.g., “subject”).
• Primitive Structural Alternation (PrimAlte): Compose a primitive (e.g., “baked”) with a sentence structure (e.g., “subj. verb obj.”).
• Phrase Recombination (PhraReco): Compose a prepositional phrase (e.g., “A in B”) with a grammatical role (e.g., “subject”).
• Longer Chain (LongChain): Extend the tail of the logical form with CCOMP clauses ∈ Y1S . The max recursive times of CCOMP clauses in B is 2, while in test case it is 12.
• Deeper Nesting (DeepNest): Expand the arguments in functions with IN/ON/BESIDE clauses ∈ Y1S . The max recursive times in B and test cases are the same with LongChain.
Note that PrimSubs and PrimAlte are low-level combinations while others are high-level ones.

heading: 2.3 In-Context Learning vs Fine-Tuning
text: Compositional generalization under the fine-tuning paradigm has been widely studied (Furrer et al., 2020; Shaw et al., 2021; Bogin et al., 2022), while there is little observation under in-context learning. To first get a general sense about in-context compositional generalization, we conduct an initial exploration to compare with a fine-tuning baseline.
Models and setups. We test in-context compositional generalization with six large models in GPT series: davinci, code-cushman-001 (cuchman001), code-cushman-002 (cuchman002), text-davinci002 (text002), text-chat-davinci-002 (chat002), and code-davinci-002 (code002). The sampling temperature is 0 (i.e., greedy decoding), and the max decoding length is 500. The reported metric is exact-match accuracy. To set a fine-tuning baseline, we take GPT2-Large with 0.7B parameters. We fine-tune it on the whole B and test without in-context examples. We set learning rate as 1e-5 and batch size as 8 during fine-tuning, and set beam size as 5 for inference. Appendix B includes more details.
Casual selection leads to low performance of in-context compositional generalization. For selecting in-context examples, we first take a casual selection: while satisfying the primitive coverage principle, we randomly select 10 examples without other preference. We conduct initial exploration on PrimSubs category. Figure 3 shows that under the casual selection, all six models lag behind the fine-tuned GPT2-Large on PrimSubs. In particular, although the size of davinci is more than 200 times that of GPT2-Large, there is a 24.2% accuracy gap between davinci and the fine-tuned GPT2-Large. These observations are close to Qiu et al. (2022).
However, we suppose the potential of in-context learning is still not fully revealed. Specifically, the selection of in-context examples does not yet take full advantage of available examples in B. In next try, while still following the primitive coverage
principle, we consider injecting some additional preference in the selection of in-context examples.
Preference in selection could bring huge improvement on PrimSubs. Inspired by Bogin et al. (2022) that suggests the influence of unobserved local structures, we consider to prioritize examples that have similar hidden structures to the test case. Figure 3 shows that with this preference in selection, results on PrimSubs hugely change: davinci now outperforms the fine-tuned GPT2-Large; codedavinci-002 even performs near-perfectly. These changes strongly suggest that the selection of incontext examples can significantly affect in-context compositional generalization.
Based on these initial results, to further reveal the potential of in-context learning, we perform in-depth investigations on how the selection of incontext examples affects compositional generalization.

heading: 3 Factors Under In-Context Examples
text: To facilitate our systematic investigation, we construct COFE (COmpositional generalization with FEw-shot examples), which is derived from COGS. For selecting in-context examples in constructing COFE, we identify, inject, and control three potential factors: similarity, diversity, and complexity.

heading: 3.1 Conceptual Definitions
text: We first give conceptual definitions of our considered factors and discuss our intuitions behind them.
Similarity has been widely considered as the main factor in selecting in-context examples (Liu et al., 2022; Shin et al., 2021; Rubin et al., 2021; Poesia et al., 2021). The primitive coverage principle can be regarded as a basic lexical similarity on the surface of expressions. Beyond this surface similarity, we consider that the structural similarity hidden behind expressions could be a beneficial factor. From the view of syntactic structure, the recombination of primitives is equivalent to the reconstruction of the parse tree. Similar structures would ease the difficulty of recombination because the model does not need to completely reconstruct the entire structure of in-context examples. Moreover, some work has suggested that the challenge of compositional generalization under fine-tuning lies in unobserved structures (Keysers et al., 2019; Shaw et al., 2021; Bogin et al., 2022).
Diversity concerns the repetitiveness among incontext examples. It portrays the property among in-context examples. Specifically, the context is under low diversity if it contains many repeating patterns among in-context examples, otherwise it is under high diversity. Under in-context learning, the low diversity can easily lead to biased observations on the full task space, as there are only few examples for the model to learn. Thus, we suppose that the low diversity among examples could block in-context compositional generalization. Moreover, some work also demonstrated that the diversity in training data could affect compositional generalization under fine-tuning (Oren et al., 2021).
Complexity reflects the amount of information contained in each individual in-context example. The higher complexity means that the example could provide more information to the model, but these information could be redundant. In addition, the difficulty in directly learning from complex examples has been flagged at the intersection of cognitive science and machine learning (Elman, 1993; Bengio et al., 2009). Such difficulty may be more severe for in-context learning, since the parameters of the model cannot be updated to fit these complex examples. Thus, we suppose that too high complexity might hinder performance.

heading: 3.2 Incorporate Three Factors Into Test Suite
text: To inject these factors in selecting in-context examples, we design a matching score based on the parse trees behind concrete expressions. Formally, considering the primitive coverage, structural similarity, diversity and complexity, the matching score of two parse trees T and T′ is defined as follows,
Match(T,T′) =wp · |P(T) ∩ P(T′)|+ ws · |S(T) ∩ [ S(T′)−S(C) ] |−
wc · depth(T′), (1)
in which P(·) contains primitives, S(·) contains partial structures (defined later), C contains already selected examples, S(T′) − S(C) means to exclude already covered parts in S(C) from S(T′), and depth(·) reflects the complexity of the tree.
The meaning of three factors in Equation 1 is that: the structural similarity means covering S(T), the high diversity means to avoid repeatedly covering the same element in S(T), and the low complexity is to prioritize low-depth structures.
Jackson observed a baby
subject verb object
root 𝐓𝑺𝟏 𝐓𝑺>𝟏
④
①
② ③
Jackson observed a baby
subject verb object
root①+②+④
in which both the matching of source side (i.e., NL expressions) and target side (i.e., logical forms) are considered. Poesia et al. (2021) has demonstrated the importance of target-side similarity in semantic parsing and code generation tasks, and this work will further investigates the necessity of source-side matching. In the following, we will give a more detailed description of notations in Equation 1.
Detailed description: Figure 4 shows an illustration of notations. Considering an expression e with the parse tree T, TL represents leaf nodes (e.g., “Jackson”) and TN contains internal nodes (e.g., “subject”). T1S contains one-depth sub-structures in T. Each T1s ∈ T1S (e.g., ① in Figure 4) contains one parent node (e.g., “root”) and a set of child nodes (e.g., “subject”, “verb” and “object”). T>1S contains deeper sub-structures that are composed from several one-depth sub-structures in T1S (e.g., ①+②+④ in Figure 4). In Equation 1, the primitives P(T) = TL, and the partial structures S(T) = T1S∪T>1S . Note that aiming combinations ⊂ S(T). Appendix E includes more details.

heading: 4 Experiments and Analysis

heading: 4.1 Experimental Settings and Hyper-Parameters
text: We take a greedy-search algorithm to sequentially select 10 examples for each test case. Models and setups follow our initial explorations in Section 2.3. For the investigation of each factor, hyperparameters in Equation 1 are set as follows2.
2Appendix C contains our detailed implementations.
In all settings, we prioritize the matching of primitives (i.e., |P(T) ∩ P(T′)| in Equation 1) since the primitive coverage principle should be firstly satisfied. Concretely, we set wp = 100 and ensure wp ≫ ws and wc in all settings.
For investigating structural similarity3, we set ws = 1 and wc = 0, and exclude S(C) term.
For investigating the effect of higher diversity, we add the S(C) term and keep other settings.
For complexity, we set |wc|·max(depth(T′)) < ws, such that the of preference of complexity will not influence the priority of structural similarity. Concretely, as max(depth(T′)) = 12 in COFE, we set wc = 0.01 for the low-complexity experiments and wc = −0.01 for the high-complexity experiments, and exclude S(C) term.
Some basic statistics for COFE under full similarity setting are listed in Table 2, and Appendix C.5 contains statistics under other settings. These statics show that the primitive coverage principle is well satisfied, since the cover rates of TL are almost 100%. Note that the coverage on T1S ∪T>1S must be lower than 100% since the aiming combination must be excluded.

heading: 4.2 Similarity
text: Structural similarity brings significant gains. Table 1 shows the performance with structural similarity. Compared to the results without structural similarity (i.e., only with the coverage on primitives), there are considerable gains on all five categories and across all six models. These gains clearly demonstrate that beyond primitive coverage, the structural similarity under in-context examples
3This setting is named full similarity setting.
are essential for compositional generalization.
More precise structural similarity brings larger gains. As mentioned in Section 3.2, the structural similarity considers to match S(T) which contains two parts, T1S and T >1 S . Specifically, we regard that T1S describes the rough structure of T, and T>1S determines a more precise structure. Based on the results in Table 1, we are curious about whether a rough structural similarity is enough. To verify this, we remove T>1S from S(T), which means that now we do not restrict the selected incontext examples to match precise structures in test cases. Figure 5 shows that the performances on four categories significantly drop with only a rough structural similarity, indicating that matching the precise structure of test case is still required for in-context examples. The only exception lies in PhraReco. It suggests that similarity is not the only influential factor for in-context compositional generalization. In Section 4.3, we will show that the low diversity and high complexity potentially cause this exception.
With structural similarity, low-level combinations are almost solved while high-level combinations still have large room for improvement. Specifically, for code-davinci-002, which exhibits the best performance among all backbone models, it performs near-perfectly on low-level combinations (i.e., PrimSubs and PrimAlte) while still does not achieve >95% accuracy on high-level combinations (i.e., PhraReco, LongChain and DeepNest). Although in-context learning greatly exceeds the fine-tuning baseline on high-level combinations, we suppose there is still potential for improvement. Compared to low-level combinations, han-
dling high-level ones requires more creation than imitation, thus just considering similarity for incontext examples is not enough. In the following, we will further investigate these high-level combinations from the view of diversity and complexity.

heading: 4.3 Diversity and Complexity
text: High diversity brings considerable gains on PhraReco. Figure 6 shows how diversity among in-context examples affects generalization on highlevel combinations. It shows that increasing the diversity could bring considerable gains in PhraReco, while not affecting the other two categories. For the performance on PhraReco, the improvements from higher diversity are in line with our speculations in Section 3.1, that low diversity leads to biased observations, thus blocking high-level structural generalization. For LongChain and DeepNest, beyond biased structures, their difficulty also lies in length generalization, thus just increasing structural diversity brings less effect to them.
Low complexity brings considerable gains on PhraReco. Figure 7 shows how the complexity in each individual example affects generalization on high-level combinations. For PhraReco, there are ∼10% gains in accuracy when the high complexity setting is changed to low complexity setting. We suppose the reason behind this gain is that simple examples could reduce the learning difficulty for the model. Moreover, simple examples also contain less redundant information thus would not confuse the model4. For LongChain and DeepNest, there is still less change on performance. Note that the max depth in these two categories is 13 while the max depth in the whole example bank is only 3. Therefore, changing the complexity of in-context examples would bring negligible influence for test cases in LongChain and DeepNest.
4Note that low and high complexity settings keep the same coverage rate on S(T), as demonstrated in Appendix C.5.

heading: 4.4 Analysis: Robustness to Prompt Order
text: Some previous work on in-context learning showed that the order of exemplars in prompt could sometimes hugely influences the performance of LLMs (Zhao et al., 2021; Lu et al., 2022). Here, we examine whether our observations above are sensitive to the prompt order. Based on the full similarity setting (Section 4.2), we consider three different strategies for ordering exemplars: 1) random order; 2) atom closer: exemplars with higher coverage on atomic blocks are placed closer to the test input; 3) structure closer (default): examples with higher similarity on linguistic structures are placed closer to the test input. Implementations of different strategies for prompt order are detailed in Appendix C.3.
Results in Table 3 show that the performance only slightly changes under different prompt orders. These results indicate that the main results revealed by COFE is consistent and reliable. It also indicates that in-context learning could be less sensitive to the prompt order when the in-context examples are chosen properly.
4.5 Discussion: Difficulty in DeepNest Among all five categories, in-context learning performs worst on DeepNest. Compared to LongChain which also test recursive structures, the results on DeepNest still lag far behind. There is an interesting observation from the study of error cases (such as Figure 10): in-context learning frequently makes word-level mistakes, while the overall nested struc-
ture in the prediction is close to the ground truth. It suggests that the performance bottleneck in DeepNest is to correctly fill the details in the complex structure, rather than generating the sketch of the structure. Appendix F.1 provides further analysis.

heading: 5 Remaining Challenges
text: Our investigation has revealed a huge potential of in-context learning on performing compositional generalization5. Despite this potential, for achieving the ideal in-context compositional generalization, there remains the following two challenges.
In-context examples are still required to match linguistic structures in NL expressions. Since all backbone models have been pre-trained on large natural language corpus, we expect that these models could already handle the high variety in NL expressions without further hints from in-context examples. Motivated by this, we conduct experiments on another variant of COFE: the source-side term Match(X,Xc) is removed from Equation 2, and the coverage of S(X) is limited (detailed in Appendix C.6). Figure 8 shows that on all five categories, the performance consistently drops if in-
5Appendix G shows the results of assembling factors.
context examples do not match the NL-side structure. It suggests that even having been pre-trained on large corpus, in-context learning still struggles to effectively recognize the semantic equivalence among different linguistic structures behind NL expressions (detailed in Appendix F.3).
In-context learning has difficulty leveraging fictional words6. The ideal compositional generalization requires that the recombination of primitives should be independent of the surface form in primitives. In COFE, we set the target-side primitives as the uppercase of source-side ones (e.g., “cat”→“CAT”). Such case conversion is commonly used in semantic parsing tasks. To test whether in-context learning could use fictional words, we replace each target-side word with random characters (e.g., replace “CAT” with “MXR”, detailed in Appendix C.7). Figure 9 shows the huge drops after changing words. Moreover, we investigate the structural accuracy by only keeping the structural terminals (e.g., parentheses and commas) in predictions. Figure 9 shows that the structural accuracy is also affected by fictional words. It indicates that on performing in-context compositional generalization, the prediction of structural sketch is not decoupled with word-level patterns.

heading: 6 Related Work
text: Compositional generalization (CG) has attracted much attention in NLP field. Most existing benchmarks measured CG under fine-tuning with synthetic semantic parsing tasks, suggesting the limitations of general-purpose neural networks (Lake and Baroni, 2018; Keysers et al., 2019; Kim and Linzen, 2020). Many approaches were proposed to enhance the CG on general-purpose models (Andreas, 2020; Akyürek et al., 2020; Guo et al., 2021; Oren et al., 2021; Shaw et al., 2021; Zhu et al., 2021) or design task-specific methods (Liu et al., 2020; Herzig
6The term “fictional words” means that these words are made up by us, so that large language models hardly encounter them during pre-training. Here, we generate fictional words by drawing random characters from the alphabet.
and Berant, 2021; Chen et al., 2020; Liu et al., 2021). Some influential factors that affect CG have been revealed, such as the length bias (Csordás et al., 2021), target-side format (Furrer et al., 2020; Herzig et al., 2021) and local structures (Bogin et al., 2022). Most existing work explored CG under the fine-tuning paradigm, while our work advances the exploration under the in-context learning paradigm.
In-context learning (ICL) along with large language models (LLMs) has shown surprising performance in many NLP tasks (Brown et al., 2020; Hendrycks et al., 2020; Patel and Pavlick, 2021; Rae et al., 2021; Zhang et al., 2022a; Hoffmann et al., 2022; Srivastava et al., 2022; Chowdhery et al., 2022; Smith et al., 2022; Wei et al., 2022). Most related to our work, Qiu et al. (2022) and Drozdov et al. (2022) also explored ICL on CG challenges. Qiu et al. (2022) utilized the targetside similarity on structural fragments and reported that LLMs still exhibited much poorer CG than finetuned small models on COGS, which is close to our initial observations. Drozdov et al. (2022) designed task-specific inference pipelines for performing CG under a least-to-most manner. Our work provides more general understandings on how to improve CG performance by revealing several factors in selecting in-context examples. In addition, some more recent work has similar observations on the potential of LLMs on CG (Hosseini et al., 2022), gains from diversity (Levy et al., 2022), and challenges under fictional words (Kim et al., 2022)
Selection of in-context examples is an essential

heading: Ground Truth:

heading: Prediction from code-davinci-002:
text: part for the utilization of ICL. Most existing work considered the similarity as the major metric during selection. Liu et al. (2022) selected k-nearest neighbors with similar sentence embeddings; Shin et al. (2021) regarded the conditional probability from a pre-trained LLM as the similarity score; Rubin et al. (2021) and Zhang et al. (2022b) separately trained a retriever to score the similarity; Poesia et al. (2021) and Madaan et al. (2022) estimated the target-side similarity. This work demonstrates the necessity of structural similarity in achieving CG, and also reveals the importance of two other factors beyond similarity, i.e., diversity and complexity.

heading: 7 Conclusion and Future Work
text: This work investigates how in-context compositional generalization is affected by the selection of examples. The test suite COFE is constructed to study three factors. Experiments show the effects of structural similarity, higher diversity and lower complexity. Two challenges under in-context compositional generalization are further revealed.
To apply our revealed factors outside the COFE test suite, one main challenge for future work is to determine the hidden structures behind expressions without knowing the exact generative grammar. Here, we consider two potential approaches. One is to use a pre-trained parser to generate a parse tree for the input query and then measure tree similarity. The other approach is to pre-train an embedding model with a structure-aware training objective and then compute embedding similarity.

heading: Limitations
text: GPU resources. This work utilizes extremely large language models and thus has a high cost on GPU resources. Concretely, experiments are conducted on the 8 x NVIDIA A100 GPU station. The maximum inference time on each version of COFE (containing 4,785 test cases) is ∼ 8 hours. The maximum estimation of costed computing resources in this study is ∼ 500 x 8 GPU hours.
Synthetic data. As in most previous work on compositional generalization (Lake and Baroni, 2018; Keysers et al., 2019; Kim and Linzen, 2020), the COFE dataset is constructed using synthetic data rather than natural one. The source-side sentences in COFE are from COGS, which account for 70–80% of naturally-occurring English sentences (Kim and Linzen, 2020; Roland et al., 2007). Thus, this synthetic test suite could be close to the real-world application scenarios.
Single run. Due to the high cost on computing resources, we do not take multiple runs with different sets of examples, nor did we take multiple samples with temperature > 0. Observations under different prompt orders (in Appendix 4.4) imply that with desired factors in selecting in-context examples, there could be low variance in experiments.

heading: Ethics Statement
text: Due to the utilization of pre-trained language models, this work could be exposed to some potential risks of ethical issues on general deep learning models (such as social bias and privacy breaches). As explored in this work that the model behavior can be hugely influenced by the provided context, we call for further investigation into how ethical issues can be avoided by controlling the provided context.

heading: Acknowledgments
text: We thank all the anonymous reviewers for their valuable comments. Shengnan An and Nanning Zheng were supported in part by NSFC under grant No. 62088102.

heading: A Grammar
text: Part of the grammar used in constructing COFE is listed in Table 4. Note that the max recursive times of R-Production Rules is 2 in prompting examples and 12 in test cases. The target-side grammar follows the reconstruction in An et al. (2023). Overall, the original target grammar of COGS is reconstructed to be chain-structured. Concretely, first, the original output tokens in COGS are capitalized; then, the variables (e.g., “x_1”) in the original grammar are aligned and replaced with their corresponding terminals; finally, the output clauses are grouped as the function format, in which the function name belongs to “PRED-FUNC” and the arguments are ordered as “AGENT”, “THEME”, and “RECIPIENT”. Moreover, if “PRED-FUNC” does not contain one or some arguments, the positions of these arguments are filled with “NONE” terminal. For the two R-Production rules in Table 4, the first is in chain structure and the second is in nested structure. Moreover, the whole nested “PPFUNC” will be filled into the “PRED-FUNC” as an argument, rather than concatenated to the tail of the “CLAUSE”.

heading: B Details of Fine-Tuning
text: The fine-tuned GPT2-Large contains 762M parameters. For fine-tuning, we take 50,000 training steps with 8 batch size and 1e-5 learning rate (without warm-up strategy). We set weight decay as 1e-2 and label smoothing factor as 1e-1. For inference with GPT2-Large, we set beam size as 5 and set max length as 1,024.

heading: C Details of Implementation

heading: C.1 Algorithm
text: Algorithm 1 shows the greedy searching algorithm for constructing COFE.

heading: C.2 Key Designs
text: We give detailed descriptions of some key designs in Algorithm 1.
• P(T): Return the leaf nodes TL on the tree;
• S(T): Return the structural combinations on the tree, i.e., T1S ∪T>1S ;
Algorithm 1 Greedy-Search Algorithm for Constructing COFE

heading: Given:
text: (X,Y): Source and target parse trees in one test case; B: Example bank; (Xi,Yi) ∈ B: One candidate case in example bank; XA and YA: Aiming combination; wp, ws, wc: Weights for primitive coverage, structural similarity, and complexity penalty; P(·): primitives; S(·): structural combinations;
Return: C: Selected in-context examples;
1: C = {} 2: while |C| < n do 3: max_score = 0 4: candidate = None 5: for (Xi,Yi) ∈ B do 6: Assert XA /∈ S(Xi) 7: Assert YA /∈ S(Yi) 8: prim_score = 0 9: stru_score = 0
10: for element ∈ P(Xi) ∪ P(Yi) do 11: if element ∈ P(X) ∪ P(Y) then 12: prim_score += wp 13: end if 14: end for 15: for element ∈ S(Xi) ∪ S(Yi) do 16: if element ∈ S(X)∪ S(Y) and element /∈ S(C) then 17: stru_score += ws 18: end if 19: end for
comp_penalty = wp · depth(Xi) score = prim_score + stru_score - comp_penalty
20: if score > max_score then 21: max_score = score 22: candidate = (Xi,Yi) 23: end if 24: end for 25: C.add(candidate) 26: end while
• wp, ws, wc: The initial scores for matching primitives, structural combinations, and complexity penalty, respectively. A higher w means that the corresponding element is prioritized in greedy search.
• element /∈ S(C): Already covered elements will not be awarded again, thus encouraging high diversity.
• depth(T): return the depth of the tree. Note that depth(Xi) = depth(Yi) in COFE.

heading: C.3 Prompt Order
text: We take the structure-closer order, i.e., the examples in C with a higher stru_score are placed closer to the test case. In Section 4.4, we show the robustness to the other two orders: random order, i.e., all selected in-context examples in C are randomly
shuffled, and atom-closer order, i.e., the examples in C with a higher prim_score are placed closer to the test case.
C.4 Max Depth in T>1S Since the max repetition times for LongChain and DeepNest are 2 (as described in Section 2.2), we set the max depth in T>1S as 2 in S(T).

heading: C.5 Similarity Under Diversity and Complexity Settings
text: While changing diversity and complexity in variants of COFE in Section 4.3, the primitive coverage and structural similarity are still satisfied. Table 5 shows that onPhraReco, the statistics of coverage in different diversity and complexity settings are kept identical to the full similarity setting in COFE.

heading: C.6 Excluding NL-Side Matching
text: For excluding source-side matching in Section 5, besides removing the first term in Equation 2, we also limit the matching of X1S . Concretely, we require that the sentence rule in test case should not be covered by in-context examples. The sentence rule is an N-Production rule that contains the nonterminal “sentence” as the left hand. To achieve this, we filter out test cases that can not meet this constraint. Finally, 1,037 out of 4,785 test cases are kept in this variant of COFE.

heading: C.7 Fictional Words
text: For each target-side word that contain l characters, we sequentially and randomly sample l characters from alphabet as a fictional word to replace the original word. In addition, for the experiments on fictional words, we take the atom-closer prompt order, since the model with this order performs better the default structure-closer order.

heading: D Excluding Target-Side Matching
text: In Section 5, we show that the performance drops with excluding the source-side matching. Here, we examine the effect of target-side matching. For constructing data, we directly remove the second term in Equation 2. As shown in Table 6, the performances with or without target-side matching are nearly identical. Such an observation is similar to the comparison between oracle and non-oracle settings in Qiu et al. (2022) that also utilized COGS benchmark, but different from Poesia et al. (2021) which suggested the importance of target-side similarity in code generation tasks. We suppose there are mainly two reasons that could cause this difference. On the one hand, different from general code generation tasks, the test suite for compositional generalization requires the exclusion of certain aiming combinations. Therefore, the performance bottleneck in compositional generalization benchmarks mainly lies in the lacked aiming combinations. On the other hand, in most compositional generalization benchmarks, the source-side matching could largely take over the target-side matching, since the terminals and rules in source grammar in these benchmarks are mapped many-to-one to the target grammar. Therefore, when seeking for the source-side matching, the target-side matching is also improved.
E Illustration of Defined Notations
Figure 11 illustrates the notations defined in Section 3.2 based on a concrete expression “Jackson in a room observed a baby”.
Note that for all sub-structures in T1S ∪T>1S , we require them to be complete sub-structures.

heading: Definition: Complete sub-structure (CSS). A
text: CSS is a subgraph in a tree T, satisfying that if an internal node in T and one of its child nodes are covered in this CSS, all other child nodes must be also covered in this CSS.

heading: F Case Study
text: We provide case study to further understanding the performance of compositional generalization observed in the main text. For ease of reading, we include the following contents in the caption of figures.
F.1 Two Types of Errors in DeepNest
Figure 12 shows two error cases in DeepNest with code-davinci-002 model and full similarity setting. The overall structure of predictions are close to the ground truth, but the model makes mistakes on some local parts. Concretely, some local semantics are incorrect (in red), and some words are redundant (in gray).
Moreover, we also calculate the word-level coverage in predictions. Besides the instance-level accuracy, we further investigate a word-level error rate on DeepNest. We find that in DeepNest, 96.8% of the words in the ground truth are contained by the predictions from code-davinci-002 (while only 48.8% for GPT2-Large). It indicates that the low instance-level accuracy is mainly caused by the wrong positions of words and redundant words.

heading: F.2 Structural Errors with Fictional Words
text: Figure 13 shows the comparison of performance between fictional words (left) and commonly used words (right). For the provided contexts on the left and right, the only difference is that the target-side words on the left are randomly selected characters while on the right they are uppercase of the source-side words. It shows that by changing only the target-side words, the model not only makes word-level errors (i.e., missing two words “ES” and “NVCWI” in prediction), it also generates the wrong parentheses structure (i.e., generate a 2-depth structure while in ground truth it is 3-depth).

heading: F.3 Fail to Recognize Semantic Equivalence
text: Figure 14 shows the comparison of performances between excluding NL-side matching (left) and containing NL-side matching (right). For the test input “Matthew shipped the professor a chair .”, it contains the sentence structure “subject verb object_1 object_2” behind the NL expression. Context on the left does not explicitly contain this sentence structure, but it contains a semantically equivalent structure (i.e., “subject verb object_2 to object_1”). However, the model generates the correct prediction on the right while fails on the left. Concretely, according to the wrong prediction on the left, the model perhaps considers that the semantics of “subject verb object_1 object_2” is equivalent with “subject verb object_1 to object_2”.

heading: F.4 Low Diversity Block Generalization
text: Figure 15 shows the comparison of performances on PhraReco under high diversity (left) and low diversity (right). For the test input “A girl in the house slept”, “subject slept” is one element contained in T>1S . This element is repeatedly covered in the context on the right (low diversity) while only covered once on the left (high diversity). However,
under high repetitiveness, the model fails on the test case, but succeed when there is low repetitiveness.

heading: F.5 High Complexity Block Generalization
text: Figure 16 shows the comparison of performance on PhraReco under low complexity (left) and high diversity (right). With low complexity, the test case is covered by simple and short in-context examples, and the model succeeds on the test case. With high complexity, the test case is covered by more complex and longer examples, and the model fails on the test case.

heading: G Full Results
text: Due to the page limitation for main text, here we list our full results in Section 4. The results in Assembling are the best performance under each category among all combinations of factors.

heading: Model: code-davinci-002
text: Category: DeepNest Setting: full similarity setting
context: context: input: A host was sold the donut on the bench beside a table . output: SELL ( NONE , ON ( DONUT , BESIDE ( BENCH , TABLE ) ) , HOST )
input: Liam gave the monkey a chalk in the container . output: GIVE ( LIAM , IN ( CHALK , CONTAINER ) , MONKEY )
input: A dog gave Emma the drink in the house . output: GIVE ( DOG , IN ( DRINK , HOUSE ) , EMMA )
input: The dog gave the spoon beside the table to Olivia . output: GIVE ( DOG , BESIDE ( SPOON , TABLE ) , OLIVIA )
input: A child was lended a cake on a stool in a garden by the dog . output: LEND ( DOG , ON ( CAKE , IN ( STOOL , GARDEN ) ) , CHILD )
input: Emma mailed a boy a cake on a paper in a room . output: MAIL ( EMMA , ON ( CAKE , IN ( PAPER , ROOM ) ) , BOY )
input: Emma gave the cake on the table on the bed to Olivia . output: GIVE ( EMMA , ON ( CAKE , ON ( TABLE , BED ) ) , OLIVIA )
input: A girl gave Olivia a cake on a table . output: GIVE ( GIRL , ON ( CAKE , TABLE ) , OLIVIA )
input: A dog gave a girl the cake beside the bed on a chair . output: GIVE ( DOG , BESIDE ( CAKE , ON ( BED , CHAIR ) ) , GIRL )
input: Charlotte offered the boy a cake on the tree on a plate . output: OFFER ( CHARLOTTE , ON ( CAKE , ON ( TREE , PLATE ) ) , BOY )
input: A dog gave Olivia a cake on the table on a stool in the container on the bench on a plate .
label: GIVE ( DOG , ON ( CAKE , ON ( TABLE , IN ( STOOL , ON ( CONTAINER , ON ( BENCH , PLATE ) ) ) ) ) , OLIVIA )
pred: GIVE ( DOG , ON ( CAKE , ON ( TABLE , ON ( STOOL , IN ( CONTAINER , ON ( BENCH , PLATE ) ) ) ) ) , OLIVIA )

heading: Model: code-davinci-002
text: Category: DeepNest Setting: full similarity setting
context: input: The girl posted a teacher the cake in a condo . output: POST ( GIRL , IN ( CAKE , CONDO ) , TEACHER )
input: Olivia lended a politician a game beside a cup in a room . output: LEND ( OLIVIA , BESIDE ( GAME , IN ( CUP , ROOM ) ) , POLITICIAN )
input: Emma sold the girl a cake in the trailer beside a warrior . output: SELL ( EMMA , IN ( CAKE , BESIDE ( TRAILER , WARRIOR ) ) , GIRL )
input: Oliver was given a melon in the car beside a bed . output: GIVE ( NONE , IN ( MELON , BESIDE ( CAR , BED ) ) , OLIVER )
input: A politician packed the cake in a house . output: PACK ( POLITICIAN , IN ( CAKE , HOUSE ) , NONE )
input: The girl ate the cup on a glacier on the bed . output: EAT ( GIRL , ON ( CUP , ON ( GLACIER , BED ) ) , NONE )
input: A cat was served a cake in the pod beside a table by Liam . output: SERVE ( LIAM , IN ( CAKE , BESIDE ( POD , TABLE ) ) , CAT )
input: Oliver ate the jacket in a bag on a table . output: EAT ( OLIVER , IN ( JACKET , ON ( BAG , TABLE ) ) , NONE )
input: Lucas was given the cake in a house in a garden by a goose . output: GIVE ( GOOSE , IN ( CAKE , IN ( HOUSE , GARDEN ) ) , LUCAS )
input: Emma ate the cake in the drawer on a tree . output: EAT ( EMMA , IN ( CAKE , ON ( DRAWER , TREE ) ) , NONE )
input: A politician ate the cake in a cup in the garden beside a bed in a house in a trailer in the car on the tree in the pod in a bag on a glacier in a condo .
label: EAT ( POLITICIAN , IN ( CAKE , IN ( CUP , BESIDE ( GARDEN , IN ( BED , IN ( HOUSE , IN ( TRAILER , ON ( CAR , IN ( TREE , IN ( POD , ON ( BAG , IN ( GLACIER , CONDO ) ) ) ) ) ) ) ) ) ) ) , NONE )
pred: EAT ( POLITICIAN , IN ( CAKE , IN ( CUP , IN ( GARDEN , BESIDE ( BED , IN ( HOUSE , IN ( TRAILER , IN ( CAR , ON ( TREE , IN ( POD , IN ( BAG , ON ( GLACIER , IN ( CONDO , NONE ) ) ) ) ) ) ) ) ) ) ) , NONE )
Figure 12: Two error cases in DeepNest with code-davinci-002 model and full similarity setting. The overall structures of predictions in error cases are close to the ground truth, but the model makes mistakes on some local parts. Concretely, some local semantics are incorrect (in red), and some words are redundant (in gray).

heading: Model: code-davinci-002
text: Category: PhraReco Setting: fictional words (full similarity setting)
context: input: William shortened the cookie on a tray . output: UPVIENL ( ZKCJBMB , ZY ( TXHMRM , UECQ ) , NONE )
input: The girl proved that Olivia ate the baby on a stage . output: QXFTO ( LAWW , NONE , NONE ) CCOMP ULX ( ESTEVR , ZY ( VYVA , PEDQY ) , NONE )
input: Elizabeth screamed . output: KXSKRS ( OWKPPDAJM , NONE , NONE )
input: A hero screamed . output: KXSKRS ( ZUNS , NONE , NONE )
input: The crocodile screamed . output: KXSKRS ( QPAKTMJAF , NONE , NONE )
input: Olivia respected that Asher returned a ball on the table in a house to a baby . output: GAFPLFB ( ESTEVR , NONE , NONE ) CCOMP IEFKEF ( AFQWF , ZY ( HFJZ , ES ( RGIJS , NVCWI ) ) , VYVA )
input: A kitty gave a scientist a cake in a can in the house . output: FMBI ( CYQUL , ES ( QXGW , ES ( VSP , NVCWI ) ) , GFJNRFBWV )
input: The cat was given the mandarin in the room in the house . output: FMBI ( NONE , ES ( PLNLARRT , ES ( ETIO , NVCWI ) ) , MXR )
input: The baby floated a cake beside the cabinet in the house . output: PSEGL ( VYVA , CIDPYO ( QXGW , ES ( DBZKXEC , NVCWI ) ) , NONE )
input: A boy confessed that Olivia returned the girl the donut on the tripod in the house . output: RKKPVRU ( CRD , NONE , NONE ) CCOMP IEFKEF ( ESTEVR , ZY ( QXKYR , ES ( OEHBXJ , NVCWI ) ) , LAWW )
input: The baby on a tray in the house screamed .
label: KXSKRS ( ZY ( VYVA , ES ( UECQ , NVCWI ) ) , NONE , NONE )
pred: KXSKRS ( ZY ( VYVA , UECQ ) , NONE , NONE )

heading: Model: code-davinci-002
text: Category: PhraReco Setting: commonly used words (full similarity setting)
context: input: William shortened the cookie on a tray . output: SHORTEN ( WILLIAM , ON ( COOKIE , TRAY ) , NONE )
input: The girl proved that Olivia ate the baby on a stage . output: PROVE ( GIRL , NONE , NONE ) CCOMP EAT ( OLIVIA , ON ( BABY , STAGE ) , NONE )
input: Elizabeth screamed . output: SCREAM ( ELIZABETH , NONE , NONE )
input: A hero screamed . output: SCREAM ( HERO , NONE , NONE )
input: The crocodile screamed . output: SCREAM ( CROCODILE , NONE , NONE )
input: Olivia respected that Asher returned a ball on the table in a house to a baby . output: RESPECT ( OLIVIA , NONE , NONE ) CCOMP RETURN ( ASHER , ON ( BALL , IN ( TABLE , HOUSE ) ) , BABY )
input: A kitty gave a scientist a cake in a can in the house . output: GIVE ( KITTY , IN ( CAKE , IN ( CAN , HOUSE ) ) , SCIENTIST )
input: The cat was given the mandarin in the room in the house . output: GIVE ( NONE , IN ( MANDARIN , IN ( ROOM , HOUSE ) ) , CAT )
input: The baby floated a cake beside the cabinet in the house . output: FLOAT ( BABY , BESIDE ( CAKE , IN ( CABINET , HOUSE ) ) , NONE )
input: A boy confessed that Olivia returned the girl the donut on the tripod in the house . output: CONFESS ( BOY , NONE , NONE ) CCOMP RETURN ( OLIVIA , ON ( DONUT , IN ( TRIPOD , HOUSE ) ) , GIRL )
input: The baby on a tray in the house screamed .
label: SCREAM ( ON ( BABY , IN ( TRAY , HOUSE ) ) , NONE , NONE )
pred: SCREAM ( ON ( BABY , IN ( TRAY , HOUSE ) ) , NONE , NONE )
Figure 13: Comparison of performance between fictional words (left) and commonly used words (right). For the provided contexts on the left and right, the only difference is that the target-side words on the left are randomly selected characters while on the right they are uppercase of the source-side words. It shows that by changing only the target-side words, the model not only makes word-level errors (i.e., missing two words “ES” and “NVCWI” in prediction), it also generates the wrong parentheses structure (i.e., generate a 2-depth structure while in ground truth it is 3-depth).

heading: Model: code-davinci-002
text: Category: PrimAlte Setting: exclude NL-side matching
context: input: A governor was passed a chalk beside the computer beside a stage by Matthew . output: PASS ( MATTHEW , BESIDE ( CHALK , BESIDE ( COMPUTER , STAGE ) ) , GOVERNOR )
input: A cat offered the rose to a professor . output: OFFER ( CAT , ROSE , PROFESSOR )
input: A chair was rented to Emma by a cat . output: RENT ( CAT , CHAIR , EMMA )
input: A cookie was given to Hazel by Matthew . output: GIVE ( MATTHEW , COOKIE , HAZEL )
input: Benjamin lended the brain to a professor . output: LEND ( BENJAMIN , BRAIN , PROFESSOR )
input: Grace was lended the chair by a boy . output: LEND ( BOY , CHAIR , GRACE )
input: A chair was sold to James by Ava . output: SELL ( AVA , CHAIR , JAMES )
input: The girl shipped a cookie in a container to Scarlett . output: SHIP ( GIRL , IN ( COOKIE , CONTAINER ) , SCARLETT )
input: A banana was fed to the professor by Emma . output: FEED ( EMMA , BANANA , PROFESSOR )
input: Matthew lended a cake beside the chair to the doctor . output: LEND ( MATTHEW , BESIDE ( CAKE , CHAIR ) , DOCTOR )
input: Matthew shipped the professor a chair .
label: SHIP ( MATTHEW , CHAIR , PROFESSOR )
pred: SHIP ( MATTHEW , PROFESSOR , CHAIR )

heading: Model: code-davinci-002
text: Category: PrimAlte Setting: full similarity setting
context: input: A chair was sold to James by Ava . output: SELL ( AVA , CHAIR , JAMES )
input: Grace was lended the chair by a boy . output: LEND ( BOY , CHAIR , GRACE )
input: A girl awarded a professor the box . output: AWARD ( GIRL , BOX , PROFESSOR )
input: The girl shipped a cookie in a container to Scarlett . output: SHIP ( GIRL , IN ( COOKIE , CONTAINER ) , SCARLETT )
input: Matthew lended a cake beside the chair to the doctor . output: LEND ( MATTHEW , BESIDE ( CAKE , CHAIR ) , DOCTOR )
input: Matthew lended Emma a cake . output: LEND ( MATTHEW , CAKE , EMMA )
input: Harper offered the professor the pickle . output: OFFER ( HARPER , PICKLE , PROFESSOR )
input: Matthew mailed Emma a cake . output: MAIL ( MATTHEW , CAKE , EMMA )
input: Matthew handed Emma a strawberry in the house . output: HAND ( MATTHEW , IN ( STRAWBERRY , HOUSE ) , EMMA )
input: A dog fed the professor the cake . output: FEED ( DOG , CAKE , PROFESSOR )
input: Matthew shipped the professor a chair .
label: SHIP ( MATTHEW , CHAIR , PROFESSOR )
pred: SHIP ( MATTHEW , CHAIR , PROFESSOR )
Figure 14: Comparison of performances between excluding NL-side matching (left) and containing NL-side matching (right). For the test input “Matthew shipped the professor a chair .”, it contains the sentence structure “subject verb object_1 object_2” behind the NL expression. Context on the left does not explicitly contain this sentence structure, but it contains a semantically equivalent structure (i.e., “subject verb object_2 to object_1”). However, the model generates the correct prediction on the right while fails on the left. Concretely, according to the wrong prediction on the left, the model perhaps considers that the semantics of “subject verb object_1 object_2” is equivalent with “subject verb object_1 to object_2”.

heading: Model: code-davinci-002
text: Category: PhraReco Setting: high diversity
context: input: A cake was forwarded to Levi by Charlotte . output: FORWARD ( CHARLOTTE , CAKE , LEVI )
input: A cake rolled . output: ROLL ( NONE , CAKE , NONE )
input: Emma rolled a teacher . output: ROLL ( EMMA , TEACHER , NONE )
input: A rose was helped by a dog . output: HELP ( DOG , ROSE , NONE )
input: The sailor dusted a boy . output: DUST ( SAILOR , BOY , NONE )
input: Evelyn rolled the girl . output: ROLL ( EVELYN , GIRL , NONE )
input: The girl needed to cook . output: NEED ( GIRL , NONE , NONE ) XCOMP COOK ( GIRL , NONE , NONE )
input: The captain ate . output: EAT ( CAPTAIN , NONE , NONE )
input: Emma broke a girl in the house . output: BREAK ( EMMA , IN ( GIRL , HOUSE ) , NONE )
input: The monster slept . output: SLEEP ( MONSTER , NONE , NONE )
input: A girl in the house slept .
label: SLEEP ( IN ( GIRL , HOUSE ) , NONE , NONE )
pred: SLEEP ( IN ( GIRL , HOUSE ) , NONE , NONE )

heading: Model: code-davinci-002
text: Category: PhraReco Setting: low diversity (full similarity setting)
context: input: Emma broke a girl in the house . output: BREAK ( EMMA , IN ( GIRL , HOUSE ) , NONE )
input: Liam respected that Noah slept . output: RESPECT ( LIAM , NONE , NONE ) CCOMP SLEEP ( NOAH , NONE , NONE )
input: Emma liked that Jack slept . output: LIKE ( EMMA , NONE , NONE ) CCOMP SLEEP ( JACK , NONE , NONE )
input: Luke slept . output: SLEEP ( LUKE , NONE , NONE )
input: Elizabeth slept . output: SLEEP ( ELIZABETH , NONE , NONE )
input: Amelia slept . output: SLEEP ( AMELIA , NONE , NONE )
input: The fish slept . output: SLEEP ( FISH , NONE , NONE )
input: The monster slept . output: SLEEP ( MONSTER , NONE , NONE )
input: A girl tolerated that Amelia slept . output: TOLERATE ( GIRL , NONE , NONE ) CCOMP SLEEP ( AMELIA , NONE , NONE )
input: The girl wished that Ava slept . output: WISH ( GIRL , NONE , NONE ) CCOMP SLEEP ( AVA , NONE , NONE )
input: A girl in the house slept .
label: SLEEP ( IN ( GIRL , HOUSE ) , NONE , NONE )
pred: SLEEP ( GIRL , IN ( NONE , HOUSE ) , NONE )
Figure 15: Comparison of performances on PhraReco under high diversity (left) and low diversity (right). For the test input “A girl in the house slept”, “subject slept” is one element contained in T>1S . This element is repeatedly covered in the context on the right (low diversity) while only covered once on the left (high diversity). However, under high repetitiveness, the model fails on the test case, but succeed when there is low repetitiveness.

heading: Model: code-davinci-002
text: Category: PhraReco Setting: low complexity
context: input: Liam appreciated a mouse on the gravel . output: APPRECIATE ( LIAM , ON ( MOUSE , GRAVEL ) , NONE )
input: A cat ate a basket beside the table . output: EAT ( CAT , BESIDE ( BASKET , TABLE ) , NONE )
input: Luna ate . output: EAT ( LUNA , NONE , NONE )
input: A dog ate . output: EAT ( DOG , NONE , NONE )
input: The prince ate . output: EAT ( PRINCE , NONE , NONE )
input: The coach ate . output: EAT ( COACH , NONE , NONE )
input: A monster ate . output: EAT ( MONSTER , NONE , NONE )
input: The priest ate . output: EAT ( PRIEST , NONE , NONE )
input: The captain ate . output: EAT ( CAPTAIN , NONE , NONE )
input: A mouse ate . output: EAT ( MOUSE , NONE , NONE )
input: A mouse beside the table ate .
label: SLEEP ( IN ( GIRL , HOUSE ) , NONE , NONE )
pred: SLEEP ( IN ( GIRL , HOUSE ) , NONE , NONE )

heading: Model: code-davinci-002
text: Category: PhraReco Setting: high complexity
context: input: Emma liked a mouse on a table beside the machine . output: LIKE ( EMMA , ON ( MOUSE , BESIDE ( TABLE , MACHINE ) ) , NONE )
input: The boy ate the drink in a house beside the table . output: EAT ( BOY , IN ( DRINK , BESIDE ( HOUSE , TABLE ) ) , NONE )
input: A girl liked that Emma ate . output: LIKE ( GIRL , NONE , NONE ) CCOMP EAT ( EMMA , NONE , NONE )
input: Emma said that a dog ate . output: SAY ( EMMA , NONE , NONE ) CCOMP EAT ( DOG , NONE , NONE )
input: The father liked that the boy ate . output: LIKE ( FATHER , NONE , NONE ) CCOMP EAT ( BOY , NONE , NONE )
input: A princess proved that Emma ate . output: PROVE ( PRINCESS , NONE , NONE ) CCOMP EAT ( EMMA , NONE , NONE )
input: Emma liked that the teacher ate . output: LIKE ( EMMA , NONE , NONE ) CCOMP EAT ( TEACHER , NONE , NONE )
input: A horse said that Emma respected that William ate . output: SAY ( HORSE , NONE , NONE ) CCOMP RESPECT ( EMMA , NONE , NONE ) CCOMP EAT ( WILLIAM , NONE , NONE )
input: The spokesman hoped that a girl liked that a turtle ate . output: HOPE ( SPOKESMAN , NONE , NONE ) CCOMP LIKE ( GIRL , NONE , NONE ) CCOMP EAT ( TURTLE , NONE , NONE )
input: A mouse hoped that a girl hoped that Olivia ate . output: HOPE ( MOUSE , NONE , NONE ) CCOMP HOPE ( GIRL , NONE , NONE ) CCOMP EAT ( OLIVIA , NONE , NONE )
input: A mouse beside the table ate .
label: EAT ( BESIDE ( MOUSE , TABLE ) , NONE , NONE )
pred: EAT ( MOUSE , BESIDE ( NONE , TABLE ) , NONE )
Figure 16: Comparison of performance on PhraReco under low complexity (left) and high diversity (right). With low complexity, the test case is covered by simple and short in-context examples, and the model succeeds on the test case. With high complexity, the test case is covered by more complex and longer examples, and the model fails on the test case.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1

heading: 7 A4. Have you used AI writing assistants when working on this paper?
text: Left blank.
B 3 Did you use or create scientific artifacts? Section 3 & Section 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 1
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 2.2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix D.5
C 3 Did you run computational experiments? Section 4 & Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 2.3 & Limitations
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 2.3 & Section 4.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix 1
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: To apply our revealed factors outside the COFE test suite, one main challenge for future work is to determine the hidden structures behind expressions without knowing the exact generative grammar. Here, we consider two potential approaches. One is to use a pre-trained parser to generate a parse tree for the input query and then measure tree similarity. The other approach is to pre-train an embedding model with a structure-aware training objective and then compute embedding similarity