SLOG: A Structural Generalization Benchmark for Semantic Parsing
Authors: Bingzhi Li
Lucia Donatelli
Alexander Koller
Tal Linzen
Yuekun Yao
Najoung Kim
Venue: EMNLP 2023 Main (2023)
Keywords: compositional generalization
structural generalization
long-distance dependencies
recursion
semantic parsing

Abstract: The goal of compositional generalization benchmarks is to evaluate how well models generalize to new complex linguistic expressions. Existing benchmarks often focus on lexical generalization, the interpretation of novel lexical items in syntactic structures familiar from training; structural generalization tasks, where a model needs to interpret syntactic structures that are themselves unfamiliar from training, are often underrepresented, resulting in overly optimistic perceptions of how well models can generalize. We introduce SLOG, a semantic parsing dataset that extends COGS (Kim and Linzen, 2020) with 17 structural generalization cases. In our experiments, the generalization accuracy of Transformer models, including pretrained ones, only reaches 40.6%, while a structure-aware parser only achieves 70.8%. These results are far from the near-perfect accuracy existing models achieve on COGS, demonstrating the role of SLOG in foregrounding the large discrepancy between models' lexical and structural generalization capacities.

SLOG: A Structural Generalization Benchmark for Semantic Parsing
Bingzhi Li
Lucia Donatelli
Alexander Koller
Tal Linzen
Yuekun Yaod
Najoung Kim*.
Université Paris Cité
*Vrije Universiteit Amsterdam
*Saarland University
"New York University
Boston University
bingzhi... 1i 11l0yyhoo... com, na najoung@bu.er edu
Abstract
The goal of compositional generalization benchmarks is to evaluate how well models gen- eralize new complex linguistic expressions. Existing benchmarks often focus on lexical gen- eralization, the interpretation novel lexical items in syntactic structures familiar from train- ing. Structural generalization tasks, where model needs to interpret syntactic structures that themselves unfamiliar from training, are often underrepresented, resulting in overly optimistic perceptions of how well models can generalize. We introduce SLOG, semantic parsing dataset that extends COGS (Kim and Linzen, 2020) with 17 structural generaliza- tion cases. In our experiments, the generaliza- tion accuracy of Transformer models, includ- ing pretrained ones, only reaches 40.6%, while a structure-awart parser only achieves 70.8%. These results are far from the near-perfect accu- racy existing models achieve on COGS, demon- strating the role of SLOG in foregrounding the large discrepancy between models' lexical and structural generalization capacities.
3
Emma
The cat
The oog
SUW
the dog
TM
(a) Lexical generalization: object - subject (COGS)
Enmra
The
8C
RCC
The dng That Mac
(b) Structural generalization: RC object->RC subject (SLOG)
Figure I: Examples of lexical generalization in COGS (a), and structural generalization in SLOG (b). The SLOG task requires mapping the generalization exam- ples to their logical forms; the corresponding logical forms are shown Table 1.
tests the ability to combine known structures into novel structure (Figure 1b). Importantly, most of the generalization types in COGS are lexical generalization (18 out of 21 generalization types, 86% of the dataset). As lexical generalization is arguably easier than structural generalization (e.g.. solvable by simple slot-filling filliiiling), this imbalance may lead to overall performance numbers that are overly optimistic with regard to model's capacity gen- eralize compositionally (Yao and Koller, 2022).
1 Introduction
Compositional gencralization benchmarks that test the ability to understand novel utterances based on composition of known parts (Montague, 1974; Partee, 1984; Fodor and Pylyshyn, 1988) have emerged as useful tool for model evaluation in semantic parsing. COGS (Kim and Linzen, 2020) in particular has become widely-used benchmark, designed expose a generalization gap be- tween training and testing data that many recent semantic parsers still struggle with.
To facilitate a more comprehensive evalua- tion of structural generalization. we introduce SLOG, Structural L.Ong-distance dependencies Generalization benchmark. SLOG extends COGS to include 17 cases of structural generalization in to- tal (14 new cases and 3 existing cases from COGS) ($2). The novel generalizations we introduce target two key structural features of human language ($3); recursion and filler-gap dependencies.
COGS distinguishes two distinct types of gen- eralization challenges: lexical generalization tests a model's ability to interpret novel combinations of known lexical items and known linguistic struc- tures (Figure la), whereas structural generalization
We use SLOG to evaluate a sequence-to- sequence (seq2seq) Transformer model trained
"This work was conducted during Bingzhi Li's visit to NYU. The middle authors are listed in alphabetical order.

---

Training
Generalization
COGS
Emma the dog.
The dog
dog (25) see.agent (z) Emma)
+dog (a1); agent (#2,
theme (2),
The
+cat (#1) run.agent (1)
SLOG
Emma saw the dog that Max held.
The dog that Max saw ran.
dog (23) see.agent (31.Rmma) A
dog (#1) dog nmod (01,
see. theme (21, A dog. nmod (23.26)
see aggnt (at,,Max) A see.theme (#4,
ho d.agent (ax,,ax) hold. theme (16, 2t) The cat
A run.agent (15,22)
scat (11);; agent (122
Table Examples of lexical generalization in COGS structural generalization SLOG with their corresponding COGS logical form (LF) representation. The task requires mapping (-+) the English sentences to their LFs.
from scratch (Vaswani al., 2017), two pretrained Transformers (T5-base: Raffel et al. 2020 and LLaMA; Touvron et al. 2023), and a structure- aware! model (AM-Parser; Weißenhorn et al. 2022). In comparison to their overall performance on COGS, all models exhibit considerably lower performance
only in object position during training must be rein- terpreted subject position during generalization. SLOG² is generated using manually specified rules (S3), adopting the same meaning represen- tation as COGS. The COGS logical form (LF), derived from Reddy al. (2017), uses indexed con- stants represent entities events. For example. in the first example of Table 1, mg denotes an entity that both a dog and the theme while 21 denotes the seeing event. The constant names are determined by the linear position of the phrasal head in the input sentence.
SLOG ($5). An error analysis reveals that the structure-aware AM-Parser gener- alizes well on the existing structural generalization cases in COGS but struggles with the gap construc- tions introduced in SLOG due to inherent structural limitations, which we discuss in $5.3. Transform- ers tend to crroneously repeat frequent meaning rep- resentation subsequences observed during training. Even with pretraining, they struggle with unseen long-distance dependencies, which we attribute to their bias towards shorter predicate-argumen de- pendencies. Overall, the discrepancy in perfor- mance between SLÖG and COGS demonstrates the utility of SLOG in exposing the overall limi- tations of current semantic parsing models shown to achieve high performance on existing general- ization benchmarks, well as highlighting the different weaknesses of these models.
seeing event,
SLOG contains 17 structural generalization cases grouped into four categories. These general- ization cases are primarily motivated by frequency asymmetries in natural language, where simpler structures are more common than complex ones; in other words, SLOG assesses whether NLP models can extrapolate from frequent patterns to rare ones. We deseribe the four categories below; see Ta- ble 2 for the full list of generalization cases.
2.1 Novel Recursion Depth
Recursion allows smaller phrases be combined to create larger phrases. This combination process can be repeated an unbounded number of times. COGS tests a model's s ability to apply recursion in two cases: sentential complements (tail CP recur- sion)3 and nominal prepositional phrase modifiers (tail PP recursion). For both cases, the training set contains recursive depths of 0-2, where 0 indicates the absence of any PP or CP. and the gencralization contains the strictly greater depths 3-12.
2 The SLOG Benchmark
SLOG follows the semantic parsing format used in COGS, where the task is to translate English ex- pressions into logic-based meaning representations. As in COGS, there is systematic gap between the training set and the generalization set: there constructions in the generalization set that are not included the training set, but pieces construc- tions included in the training set can be recombined to arrive at their correct meanings. For example, as illustrated in Table 1, noun phrases that appear
By contrast, the SLOG training set includes re- cursion of depths 0-2 and 4, and the generalization
"The generation code and SLOG dataset are available at https: 1/ ////gtt/ /ggttuub.b... com/ /ttd//iingg winghhi SLOG. *Nested clauses with bracch emheddings like [Max knows that [Mary knows (that Emuna cooks]crlerJcr
IIn this paper, 'structure-aware" refers specifically to mod- els that incorporate explicit representations of linguistic struc- ture.

---

Generalization cases
Training
Generalization
$2.1 Novel Recursion Depth
Deeper depth generalization Prepositional phrases (PP) max depth 4 - depth 5-12
Ava the ball in the bottle the table.
Ava the cat in the box the mat on the bed on the floor in the room.
V Tail CP recursion max depth 4 - depth 5-12
Ava believed that Emma said that a fish froze.
Ava said that Emma liked that Max believed that Noah found that Liam saw that the cat slept. Ava held the dress that store that a girl that a boy that a cat that man drew loved knew sold.
Center embedding max depth depth 5-12
Eva saw the cal that the horse that the dog liked chased.
Shallower depth gemeralization Prepositional phrases max depth depth
Emma saw the ball bottle the table the floor in the office.
Ava saw the cat on the mat on floor in the office.
Tail CP recursion max depth + depth 3
Ava believed that Emma said that Max found that cat saw that
Ava said that Emma liked that Max believed that the cat slept.
fish froze.
Center embedding max depth + depth
Eva the that the horse Emma bought the dress that the store that the woman that Mike knew liked sold.
that the dog that the man that
the girl loved found liked chased,
$2.2 Novel Combination of Modified Phrases and Grammatical Roles
PP in direct object NPs
< in subject NPs
Noah ate the cake the plate. Noah ate the cake on the plate.
The cake the table hurned.
+ PP in indirect object NPs
Max gave fish to a cat on a table.
RC direct ohject NPs
RC in subject NPs
Noah the cat that froze.
The cat that froze smiled.
+ RC in indirect object NPs
Noah saw the cat that froze.
Max gave a fish to a cat that ran.
$2.3 Novel Gap Positions
Subject. direct object- eettecteettactee RC + Indirect object-extracted RC
Noah the cat that gave fish to Liam. @ Noah saw the cal that Liam liked
Noah the cat that Emma gave a cake to
Subject, direct object wh-questions + Indirect object wh-questions
Who saw the cat? What did Emma see_?
Who did Noah give the cake to
$2.4 Novel wh-questions
Subject, object w/-Q of simple transitives
Who saw the cat?
Active subject wh-questions
Who wanted sleep
Emma wanted sleep.
Who did Emma see _?
Passive subject wh-questions
Who was helped by Emma?
@ The boy was found by Emma.
Direct object wh-questions with ditransitive verbs
What did Emma see ? @ Emma gave fish to the cat. What did the cat 4 the cat on the mat
What did Emma give the cat?
Wh-questions with modified NPs
What did the cat table see
Wh-questions long movement
What did the cat see ? Emma
What did Emma say that the found
said that the cat fish.
Table 2: A full list of SLOG generalization cases. Each sentence in the table corresponds to (sentence, logical form) pair, illustrated in Table denotes the composition of observed structures, which allows interpret the target novel structure. Some cases cover multiple constructions: c.g., all ditransitive verbs include both double-object and prepositional constructions. The three cases marked with '/" are structural generalization cases already present in the COGS dataset.

---

set contains both the intermediate depth 3 and the greater depths 5-12. Including both shallower and deeper embeddings in the generalization set allows us to determine if any difficulty in generalizing to an unseen embedding depth is consequence of the model's 's more general difficulty in processing longer sequences than observed in training (Lake and Baroni, 2018; Herzig et al., 2021; Anil et al., 2022) rather than more specific issue with apply- ing recursion generate novel constructions.
(2) Noah saw |a cat on the tableldobj.
(3) [The cat on the mat ran.
(4) Emma gave La cat the mat a fish.
We expect sub-cases of indirect object modifica- tion to pose challenges of varying difficulty, de- pending on the length of the predicate-argumen dependency. In particular, generalization to indi- rect object modification in active oblique datives (4) introduces a dependency between the verb gave and the direct object fish across the non-argument NP the mat.4 4 In contrast, sub-cases like (5a) and (5b), where the non-argument NP occurs at the end of the sentence, do not include dependency across an intervening NP; we therefore expect them to be relatively easier.
In addition to this new depth split, SLOG in- troduces a new recursion construction. COGS in- volves only tail recursion, which features recursive PPs and CPs with right-branch embeddings. SLOG extends this with center embedding, where phrase is embedded in the middle of another of the same type, leaving elements on both sides of the embed- ded component and producing rell-parenthesized long-distance dependencies, as denoted by the sub- script numbers:
(5) a. Emma gave fish to [a cat on the mat]iobj-
b. A fish was given to [a cat on the matlkobj-
SLOG's training set additionally includes stan- dalone PP-modified NPs (e.g., the NP the cat on the table on its own') to prevent modifiers from being associated with only particular range of token indices, as pointed out by Wu et al. (2023).6 Such standalone NPs, which are com- mon in child-directed speech (Wells and Bridges, 1981; Cameron-Faulknee a part of COGS, serve as a signal that the distri- bution of PP-modified NPs is not restricted to the object position.
(1) Eva saw the mouse [that the cat that the dog2 chased2 held; J.
At the same recursion depths, the average LF length increases from PP recursion to tail CP recursion to center embedding.
In natural language, recursion depth is rarely greater than five, and center embedding is gener- ally limited to two levels (Karlsson, 2007, 2010). By contrast, SLOG includes recursion up to depth 12. While this may surpass human processing abil- ities for reasons presumed to be linked memory constraints (Gibson and Thomas, 1999; Karlsson, 2007), deeper embedding depth remains grammati- cal, echoing Chomsky's s competence versus perfor- mance distinction. Importantly, we also note that our goal with SLOG is to evaluate the linguistic competence of NLP models, whose goal is not to simulate human performance limitations.
al., 2003) but were not
2.2.2 Relative Clause Modifiers
Similar to PP modifiers, NPs with relative clause (RC) modifiers, in (6), can occupy any position that unmodified NP can fill. We expect RC modifiers pose greater challenge compared PP modifiers, as they involve gap constructions, in which a phrase needs to be interpreted in a position other than its canonical position in a declarative clause. We refer to this as extraction (Sag, 2010), and we mark gap positions with an underscore. In (6), the dog should be interpreted as if occupies the gap position as the direct object of held; in the logical form, this is represented by the fact that 23 is filling both see t. theme and hold. theme.
2.2 Novel Combination of Modified Phrases and Grammatical Roles
SLOG also tests the capacity to interpret complex noun phrases (NPs) in new positions. In addition PP modifiers included in COGS, we introduce relative clause modifiers.
2.2.1 Prepositional Phrase Modifiers
"This observation also holds true for the generalization to subject modification shown in (3).
COGS, NPs modified by PPs are seen only as direct objects (2), and need to be interpreted as sub- jects during generalization (3). SLOG adds general- ization cases targeting indirect object modification
"the cat on the table <cat (#1) *table (24) cat . amod. on (21,
"PPs in COGS were restricted to the object position, so models never observed the association of modifiers with lincarly-carlice indices, which makes it difticult to isolate this effect from structural generalization.
(4).

---

(6)
Emma saw the dog that Max held
2.4 Novel Wh-queestins
<dog (ma) see. agent (01, Emma)
Next, we evaluate generalization to extraction cases that involve familiar gap positions-subiee and di- rect abject-paired with verb types that have never been observed in wh-questions during training. For this case, the training contains wh-questions with simple transitive verbs (13) and declarative sentences with various verb types: transitive, in- transitive and ditransitive. The generalization set includes five novel types of wh-questions that have not been observed during training, though their declarative counterparts have.
see. theme (#1, .g)) A
bog.nmon (#3, 26) A hold. (06.
Max) theme (Ig, 4z3)
Similar to the case of the PP modifiers ($2.2.1), the training set contains direct object NPs modified by RCs as well as standalone RC-modified NPs, as in (7). The generalization set contains RC modi- fiers for subject NPs, as in (8a), and indirect object NPs, in (8b):
(7) TRAINING
The novel wh-questions have varying distance between the wh-filler and the gap. Subject wh- questions, which maintain the same word order as their declarative counterparts, exhibit no gap (14a, 14b). Questions about direct objects of ditransitive verbs (14c), as well as questions with NPs modified by either PP or an RC (14d), have moderately long filler-gap distances. The filler-gap distance is longest for object extraction out of embedded clauses (14e).
a. Liam saw [the cat that Emma held
Idobj
the cat that Liam fed
(8) GENERALIZATION
a. [The cat that Emma found Juby smiled.
Liam gave [a cat that Emma held
Jloby a fish.
2.3
Novel Gap Positions
The SLOG training set contains both subject and di- rect object extraction in RCs (9); these are the most frequent extraction positions in both written and spoken English corpora (Roland et al., 2007). The generalization set includes extraction of indirect objects (10), less frequent construction.
(13) TRAINING
(The training set also includes the declara- tive counterparts of (14).)
a. Who saw cat
b. What did Emma see ?
(14)
GENERALIZATION
(9) TRAINING
a. Who froze ?
a. Liam saw the boy that ate a cake. Liam saw the boy that Emma loved
b. What was frozen
C. What did the boy give to Liam?
(10) GENERALIZATION
d. What did Max give cat that slept
a. Liam saw the boy that Emma gave a cake to
e. What did boy say that Max believed that the cat saw
SLOG also tests for the interpretation of novel gap positions wh-questions. As with RCs, the training set includes questions with either subject or direct object extraction (11), and the general- ization set contains questions with indirect object extraction (12).
3 Dataset Generation
Grammar SLOG is generated from a probabilis- tic Synchronous Context-Free Grammar (SCFG) implemented in Alto (Gontrum al., 2017). This grammar simultaneously generates the English ex- pressions and their corresponding meaning repre- sentations (see Appendix B for more details).
(11) TRAINING
a. Who did Emma love _?
Who ate cake?
(12) GENERALIZATION
Training and generalization sets We follow similar sampling procedure COGS. A total of
a. Who did Emma give a cake _..
a wh-question (a),,, wh-filler (who) in the initial position of the clause is interpreted as if occupied the gap (again indicated with an under- score) in the direct object position love.
Wi-qquestions with PP- or RC-modified NPs include vari- ous constructions where modifiers appear in subjects, direct objects, or indirect objects, exhibiting an average filler-gap distance similar to ditransitive verh wli-questions.

---

10,607 sentences are sampled from the probabilis- tic SCFG and then split into training, in-domain validation and in-domain test sets with an 8:1:1 ratio. The splits are then merged with the corre- sponding COGS splits. We then add 100 standalone PP-modified NPs and 100 standalone RC-modified NPs to the training set, as discussed in Section 2.2. We also include what we refer as primitive ex- posure examples for each ditransitive verb and verb accepting CP arguments, totaling 40 primitives. These are standalone verb lexical meanings, such as, hope ~ λa .a.b... how.pe agent (e, b) A sope.ccomm (e, a). This results in final training set of 32,755 examples and 4,046 exam- ples in both validation and in-distribution test sets.
et al. (2020) for our experiments, it yields the best overall results compared to alternative versions of AM-parser, such as the one in Groschwitz et al. (2018). We run each experiment with five different random seeds. See Appendix A for more details.
Evaluation metric Most studies report exact match accuracy on COGS, This metric has two limitations that may lead to an underestimation of model's generalization capacity. First, because the COGS LF is conjunctive, reorderings of the conjunets are semantically equivalent; yet, under exact match accuracy, only single order is consid- ered correct. Second, the COGS LF uses Skolem constants with naming scheme tied to the lin- ear indices of phrasal heads in the input. While a commitment to systematic naming scheme is necessary for consistent evaluation, different nam- ing schemes up to the renaming of the constants in the gold LF yield equivalent LFs (e.g., (15a) vs. (15b)). Such LFs would be considered incorrect under exact match.
For the generalization set, we use separate gram- mars for each generalization case. We sample 1000 examples from each of the 17 cases, yielding a total of 17,000 examples. For the training set and the generalization set, the maximum lengths of the in- put English sentences are 28 and 61 tokens, respec- tively. The maximum lengths of the corresponding output logic forms are 229 and 599 tokens. See Appendix B for more details.
To incorporate semantic equivalence up to con- junct reordering and constant renaming, at eval- uation time, we alphabetically sort the conjuncts of the gold LFs, and subsequently index variables based on their appearance order in the sorted LFs. The same modifications are applied to the model outputs. This process results in the reformatted out- put as shown in (16); applying these modifications to (15a) and (15b) yields the same outcome. Then, computing exact match on these postprocessed LIfs captures the targeted semantic cquivalence.
4 Experimental Setup
Models We evaluate the performance of seq2seq. autoregressive, and structure-aware models on SLOG. The seq2seq models we evaluate are a Transformer we train on SLOG from scratch (vanilla Transformer henceforth; Vaswani et al. 2017), and finetuned pretrained Transformer (T5; Raffel et al. 2020) that has demonstrated strong performance on multiple compositional generaliza- tion tasks (Herzig al., 2021). The autoregressive Transformer model we evaluate is LLaMa (Touvron et al., 2023). Finally, the structure-awarc model we evaluate is the AM-Parser (Groschwitz et al., 2018), which achieves near-perfect accuracy on COGS (Weißenhorr et al., 2022). Previous work has shown that structure-awert models perform well on compositional generalization tasks, specifi- cally those involving structural generalization (Yao and Koller, 2022). Following Weißenhorn al. (2022), we first have the AM-Parser predict an in- termediate dependency tree, and then convert it to a graph-based representation of the SLOG logical form. We use the * AM-parser from Lindemaan
(15) Gold LF and model-predicted LF for What did the baby ear?:
a. Gold: eat theme (x1, 7) A eat aat.a (X1, X1) A baby (Xx)
b. Out: eat. (X3, Xa) A
eat theme (xa, ?) A baby (86)
(16) Reordered and reindexed version:
baby (y2) eat eat.agent (y1, Y2)
eat theme (y1, ?)
This reformatted exact-match metric is used for all results reported in the main text; see Appendix C.1 and Table 5 for more details.
Results
Overall, seq2seq Transformers, both trained from scratch and pretrained, display low accuracy on
*Primitive examples of these two verb types let us incorpo- rate their infinitive forms, used in wi-qqesttons, into SLOG's vocabulary.
FFor detailed discussion, please refer to Appendix D.

---

traradorme
Transformer
50
175
AM Farser
70 LiaMta
70.0
50
40
901
2.1 Recursion depths
Modilier positions
2.3 Gap positions
244 Wi-muessoons
SLOG-all
COGS-all
COGS-structural
Figure 2: Accuracy on SLOG, with error bars indicating variations across runs. also show the best pub- lished results COGS (indicated with 1), reported in Yao and Koller (2022).
Figure 3: Aggregate accuracy on SLOG by general- ization category, with error bars denoting the standard deviation across generalization cases within each cate- gory over five model runs.
SLOG (Figure 2), in line with earlier studies on structural generalization in seq2seq models (Yao and Koller, 2022). This is also the case for the more recent autoregressive Transformer LLaMa, whose performance is similar that of T5. As Figure shows, high accuracy on the full COGS dataset, where 86% the generalization cases are lexical, can obscure low performance on structural gener- alization, highlighting the need for the expanded structural generalization tests included in SLOG.
levels of embedding). A considerably lower perfor- mance observed for seq2seq models with shal- lower tail CP recursion (<61%) %)); in particular, the Transformer trained from scratch consistently fails to generalize to shallower center embedding, with zero accuracy overall. Transformer models show systematically lower performance on deeper re- cursions (5-12 levels of embedding), whereas the structure- model is robust to depth variation.
SLOG additionally reveals weaknesses in the AM-Parser that COGS did not. While the AM- Parser achieves 90% accuracy on the structural generalization subset of COGS (Figure 2), it faces systematic difficulties with several generalization types introduced in SLOG (Figure 3).
Vanilla Transformer
AM
T5
LLaMa
purser
At or below mux training oulpil length PP recursion Thil CP recursion
29.3
37.0
46.0
100.0
3.0
17.7
40.2
100.0
Center embedding
0.0
0.0
0.0
100.0
Performance varied substantially across gener- alization categories (Figure 3); particular, all models achieved near-pen sear-perfect accuracy on Active subject wh-questions and Shallower PP recursion. These cases were the least structurally complex in their respective categories ($2.3 and $2.1). ). We high- light specific error types in the rest of this section; see Appendix C for full results and additional error analysis.
Beyond max training output length
PP recursion
0.0
100.0
Tail CP recursion
0.0
0.0
0.0
100.0
Center embedding
0.0
100.0
Table 3:
Mean accuracy (%) unseen deeper recursion cases, broken down by whether the expected output falls within exceeds the range of training output lengths (maximum training output = 229 tokens).
We investigate the relation between length and depth generalization further by dividing the deeper depth generalization cases into examples that are shorter longer than the maximum output length observed in training (229 output tokens). Results are shown in Table All tested Transformer mod- cls are unable to generalize to examples longer than the maximum output length observed training;
5.1 Unobserved Depth and Length Both Affect Depth Generalization
The maximum depth observed in training was four levels of embedding for all three recursive struc- tures tested. All models achieve greater than 90% accuracy on unseen shallower PP recursion (three

---

this result is consistent with the difficulty of length extrapolation observed in the literature (Hupkes et al., 2020; Anil et al., 2022). Length extrap- olation does not capture the full story, however: the model's performance is limited even when the length of the generalization examples falls within the range of observed output lengths. This indi- cates that unobserved depth indeed plays role in these models' poor generalization deeper struc- tures, in addition to known difficulties in length generalization.
ing (see Appendix C.4), the structure-aware AM- Parser demonstrates different generalization diffi- culties.
The AM-Parser systematically fails on every in- stance of wh-questions involving long movement (e.g. What did Ava say that the cal saw ??). This issue arises from its internal prediction of depen- deney trees, which represent how meaning repre- sentations are compositionally constructed. For these wh-questions, the required dependency trees are nonprojective since the edge from the embed- ded verb to the wh-pronoun crosses the matrix verb. However, the AM-Parser used in our study only supports projective dependency trees, leading to incorrect prediction of sentence structure." 10 This issue with projectivity for structural limitations of similar structure-aware parsers (Liu al., 2022; Qiu et al., 2022a).
5.2 Unobserved Long-distance Dependencies Make Generalization Difficult
Generalizing to subject modification (both PP and RC) is one of the most challenging cases, Seq2seq models achieve near-zero accuracy, even with the additional cue from the standalone modified NPs that modification appear outside of object posi- tions. This challenge echoes previous findings COGS (Akyurek and Andreas, 2021; Zheng and La- pata, 2022: Yao and Koller, 2022). The remainder of this section focuses on the analysis of PP modi- fication cases, but similar patterns are observed for RC modifiers, which we discuss in Appendix C.3. Common eтror patterns across Transformer mod- els reveal bias towards shorter predicate- arggumen dependencies. For instance, in sentences like A cat on the mat froze, models often misinterpret the closer NP the mat as the subject.
serve
diagnostic
Furthermore, on the indirect and direct object wh-questions, the AM-Parser performs very un- predictably, with accuracies ranging from 0 to 80 depending on the random seed. This is because at the bottom of its compositional process, the AM- Parser predicts the lexical meaning for each token in the sentence (supertag). In these generalization types, the gold meaning representations in the test set require supertags that are infrequent in training. Thus, while the AM-Parser can compensate the distribution shift of the meaning representations as whole, SLOG exposes its weakness to distribu- tion shifts in the lexical supertags. A more detailed discussion is provided in Appendix D.
further breakdown of the modifier generaliza- tion performance by construction shows that ex- amples involving longer predicate-argument depen- dency (i.c., there is an intervening von-argument NP between the predicate and the argument) tend to be more difficult for all models (Table 4). However, the Transformer-bane based models show stronger bias towards linearly adjacent pedicate--rgumeet struc- tures. Further analysis (Appendix C.2) shows that seq2seq models additionally fall prey to inference patterns akin to a modification rule "attach PPs to NPs in immediate post-verb position", which is compatible with the training data but leads to incorrect generalization.
Related Work
Previous research has shown that recurrent neu- ral network (RNN) architectures often struggle with learning complex long-range relations from simpler formal languages (Avcu et al., 2017; Ma- halunkar and Kelleher, 2019). Our results SLOG reveal that unseen long-distance predicate- argument dependencies pose considerable difficulty for Transformer-based models as well ($5.2). For filler-gap dependencies, prior work has centered on syntactic tasks involving wh-questions or rel- ative clauses (Wilcox et al., 2018; Marvin and Linzen, 2018; Li et al., 2023; i.a.). These studies primarily use language modeling the task and do not require mapping to semantic representations. SLOG incorporates both long-distance predicate-
5.3
Gap Generalizations Are Challenging for All Tested Models
For gap generalization cases, all models display low accuracy and considerable variability across different runs as shown in Figure 3. While Trans- former models are biased towards more frequent subsequences of output LFs observed during train-
Alternative versions of the AM-Parser that can handle non-projective trees exist and are discussed in Appendix D.

---

Long pred-arg dependency?
Vanilla
AM parser
Generalization cases
T5
LLaMa
Transformer
Solwerre objects
x
95.5
97.5
98.2
93.6
A fish was given to cat on the mat liobij
Sub-case: Indirect object in PP datives
22.9
50.5
75.5
100.0
Emma gave fish to
the liobj-
Suberre:: Indinect object double objoct daliees
4.5
9.7
36.3
77.9
Emma gave [ a cat on the mat liojj a fish.
Subject
0.0
0.8
28.9
57.6
[A cat mat]aubj ate fish.
Table 4: Performance of PP modification generalization broken down by construction. Bold orange words denote long predicate-argument dependencies, while bold black words indicate short ones.
argument and filler-gap dependencies within se- mantic parsing setup.
7 Conclusions
We introduce SLOG, semantic parsing dataset that extends the COGS benchmark with a focus on structural generalization, which is often underrep- resented in current benchmarks for compositional generalization. Using SLOG, we assess the struc- tural generalization capacity of Transformer mod- els (both pretrained and trained from scratch), well as AM- M-Parrer, structure- all models achieve good overall accuracy on COGS (2 78%), their performance on SLOG is substan- tially lower, especially for Transformer models (< 41 1).. Furthermore, even the structure-aware AM- Parser, which achieved strong performance on all structural generalization cases of COGS, performs poorly on several of the newly introduced general- ization types in SLOG. Our eгror analysis shows that all Transformer models tested struggle with in- terpreting unseen long- dingttistnce dependencies and deeper recursive constructions than observed in training. On the other hand, the AM-Parser, despite its stronger overall performance (71%), displays categorical failures on gap generalization due to its inherent parser design limitations. Overall, SLOG exposes the limitations of range of models that have previously been claimed to achieve good com- positional generalization, and can serve useful analytic tool for guiding future improvements.
Generalizing recursive patterns to deeper struc- tures has been investigated in both artificial neural networks and humans using artificial languages (Christiansen and Chater, 1999; Lakretz et al., 2021; McCoy et al., 2021). Our findings under- score Transformer- based models' limitations with deeper recursive structures, corroborating the ob- servations of Hupkes et al. (2020); Lakretz et al. (2021). In contrast, human studies have shown that they can learn and extrapolate center-embedding patterns greater depth in artificial languages (Fitch and Hauser, 2004; McCoy al., 2021).
model. While
Generalization cases in SLOG draw inspiration from the frequency gaps in natural language, where common patterns serve as a foundation for gen- eralizing to rarer structures. This has connec- tions to language acquisition in children, who have limited exposure to complex, less frequent struc- tures, yet need to generalize to novel complex ut- terances by extrapolating from familiar linguistic elements (Perfors et al., 2011; Tomasello and Ol- guin, 1993; Atkinson et al., 2018). Human profi- ciency in such generalizations attributed to in- ductive biases rooted in systematic compositional rules. However, the -ranseormer-based models we tested, despite excelling in lexical generalization scenarios, face challenges when presented with un- familiar linguistic structures requiring such rule induction, hinting at potentially different inade- quate underlying mechanisms. More broadly, how the compositional generalization cases proposed in this work can be connected to human language acquisition an interesting area of future study.
Limitations
SLOG is synthetic corpus and covers only frac- tion of the diverse structures in English. Further- more, previous research has demonstrated that the design of meaning representations (MR) have nontrivial effect on model performance in seman- tic parsing tasks (Guo et al., 2019; Herzig et al., 2021; Qiu et al., 2022b). For example, as noted by Wu et al. (2023), the variable indexing scheme

---

may introduce additional semantically irrelevant challenges when assessing structural generaliza- tion. SLOG's reformatted exact-match evaluation metric partially addresses this concern by taking into consideration several variations of MRs that are semantically equivalent, including MRs that are equívalent up to constant renaming. However, more comprehensive study of the effect of artifacts from the formalism is left to future work.
Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4934-4946, Online. Association for Computational Linguisties.
Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowezz, Vedant Misra, Vinay Venkatesh Ra- masesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. Exploring length gen- eralization large language models. In Advances in Neural Information Processing Systems.
There also exist challenges specific to the evalu- ation of pretrained models. That is, distributional shift between training and generalization sets in- tended by SLOG, such as withholding the construc- tions PPs modifying subject NPs from training, is difficult to strictly enforce when pretraining is in- volved (Kim et al., 2022). This potential violation of distributional control makes the interpretation of the obtained results difficult; we cannot disen- tangle whether generalization success in pretrained models derives from genuine compositional capa- bilities or simply exposure during pretraining to the target constructions meant to be withheld from the evaluated models. Still, corpus analyses such as Karlsson (2007) suggest that deep center embed- ding beyond three levels is very rare in naturally occurring data, so it is possible that very deep em- bedded structures are withheld as intended even from models exposed large amounts of pretrain- ing data. We hope the additional structural gen- eralization cases that SLOG offers can also help with future work investigating the interaction be- tween structures available in pretraining data and structural generalization.
Emily Atkinson, Matthew W. Wagers, Jeffrey Lidz, Colin Phillips, and Akira Omaki. 2018. Developing incrementality in filler-gap dependency processing. Cognition, 179:132-149
Enes Aveu, Chihiro Shibata, and Jeffrey Heinz. 2017. Subregular complexity and deep learning. CLASP Papers in Computational Linguistics, page 20.
Thea Cameron-Faulkan Elena Lieven, and Michael Tomasello. 2003. A construction based analysis of child directed speech. Cognitive science, 27(6) 873.
Noam Chomsky, 1957. Syntactic Structures. Janua linguarum (Mouton, Paris).: Series Minor. Mouton.
Morten H. Christiansen and Nick Chater. 1999. Toward connectionist model of recursion in human linguis- tic performance. Cognitive Science, 23(2):157-200
Róbert Csordás, Kazuki Irie, and Juergen Schmidhuber. 2021. The devil in the detail: Simple tricks im- prove systematic generalization of transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 619- 634, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Seales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022. Composi- tional semantic parsing with large language models. arXiv:2209.1 15003.
Aeknowledgments
We thank Zhengxuan Wu, Christopher Manning, Christopher Potts and all members of the NYU Computation and Psycholinguistics Lab for help- ful discussion. This work was supported in part through the NYU IT HPC resources, services, and staff expertise, and was funded by Labex EFL ANR-1O-LABX-ONS of Université Paris Cité, the DFG through project KO 2916/2-2, and the National Science Founda- tion (NSF) under Grants No. 2C4-204122, BCS- 21 14505 and IIS-2239862.
W Tecumseh Fitch and Mare D Hauser. 2004. Com- putational constraints on syntactic processing in a nonhuman primate. Science, 303(5656):37773388))
Jerry A Fodor and Zenon W Pylyshyn. 1988. Connec- tionism and cognitive architecture: A critical analysis. Cognition, 28(1-2)33-71
the laboratory LLF
Edward Gibson and James Thomas. 1999. Memory limitations and structural forgetting: The perception of complex ungrammatical sentences as grammatical, Language and Cognitive Processes, 14(3):225-248
Johannes Gontrum, Jonas Groschwitz, Alexander Koller, and Christoph Teichmann. 2017. Alto: Rapid proto- typing for parsing and translation. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 29-32, Valencia, Spain. Association for Computational Linguistics.
References
Ekin Akyurek and Jacob Andreas. 2021. Lexicon learn- ing for few shot sequence modeling. Proceedings of the 59th Annual Meeting of the Association for

---

Jonas Groschwitz, Meaghan Fowlie, and Alexander Koller. 2021. Learning compositional structures for semantic graph parsing. In Proceedings of the 5th Workshop on Structured Prediction for NLP (SPNLP 2021), pages 22-36, Online. Association for Compu- tational Linguistics.
Brenden Lake and Mareo Baroni. 2018. Generalization without systematicity: On the compositional skills of requence-to-sequene recurrent networks. In In- ternational conference on machine learning, pages 2873-2882. PMLR.
Yair Lakretz, Théo Deshordes, Jean-Rémi King. Benoît Crabbé, Maxime Oquab, and Stanislas Dehaene. 2021. Can mns learn recursive nested subject-verb agreements? arXiv preprint wrXiv:2101.02255
Jonas Groschwitz, Matthias Lindemann. Meaghan Fowlie, Mark Johnson, and Alexander Koller. 2018. AMR dependency parsing with a typed semantic al- gebra. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1831-1841, Melbourne, Australia. Association for Computational Linguistics.
Bingzhi Li, Guillaume Wisniewski. and Benoft Crabbé. 2023. Assessing the Capacity of Transformer to Ab- stract Syntactic Representations: A Contrastive Anal- ysis Based on Long-distance Agreement. Transac- tions of the Association for Computational Linguis- tics, 11:18-33.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian- Guang Lou, Ting Liu, and Dongmei Zhang. 2019. To- wards complex text-to-SOQ SQL in cross-domal database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages Italy. Association for Computational Linguistics.
Matthias Lindemann, Jonas Groschwitz, and Alexan- der Koller. 2020. Fast semantic parsing with well- typedness guarantees. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 3929-3951, On- line. Association for Computational Linguistics.
Florence,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034.
Chenyao Liu, Shengnan An, Zeqi Lin, Qian Liu, Bei Chen, Gian-Guang Lou, Lijic Wen, Nanning Zheng, and Dongmei Zhang. 2021. Learning algebraic combination for compositional generalization. Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 202 pages 1129-1144, On- line. Association for Computational Linguisties.
Jonathan Herzig, Peter Shaw, Ming- Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021. Un- locking compositional generalization in pre-trained models using intermediate representations. arXiv preprint
Puyuan Liu, Chenyang Huang, and Lili Mou. 2022. Learning non -an-orrrreresivee models from search for unsupervised sentence summarization. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 791 6-7929, Dublin, Ireland. Association for Computational Linguistics.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint (.Giv:210000066
Abhijit Mahalunkar and John Kelleher. 2019. Multi- element long distance dependencies: Using SPK lan- guages to explore the characteristies of long-distance dependencies. In Proceedings of the Workshop Deep Learning and Formal Languages: Building Bridges, pages 34-43, Florence. Association for Computational Linguistics.
Dieuwke Hupkes. Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795.
Fred Karlsson. 2007. Constraints multiple center- embedding of clauses. Journal of Linguistics, 3332).365-392.
Rebecca Marvin and Tal Linzen. 2018. Targeted syn- tactic evaluation of language models. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
Fred Karlsson. 2010. Syntactic recursion and iteration. Recursion and human language, pages 43-67.
Najoung Kim and Tal Linzen. 2020. COGS: A compo- sitional generalization challenge based on semantic interpretation. In Proceedings the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105, Online. As- sociation for Computational Linguistics.
Richard Thomas McCoy, Jennifer Culbertson, Paul Smolensky, and Géraldine Legendre. 2021. Infinite use of finite means? evaluating the generalization of center embedding learned from an artificial gram- mar. In Proceedings of the Anmual Meeting of the Cognitive Science Society, volume 43.
Najoung Kim, Tal I Linzen, and Paul Smolensky. 2022. Uncontrolled lexical exposure leads to overestima- tion of compositional generalization pretrained models. 22nn:22222..76
Richard Montague. 1974, English as formal language. In Richmond H. Thomason, editor, Formal Philos- ophy: Selected Papers of Richard Montague, pages

---

188-222. Yale University Press, New Haven, Lon- don.
you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Barbara H. Partee. 1984. Compositionality. In F. Land- man and F. Veltman, editors, Varieties of Formal Semantics, pages 281-311.
Pia Weißenhorn, Lucia Donatelli, and Alexander Koller. 2022. Compositional generalization with broad- coverage semantic parser. In Proceedings of the 11th Joint Conference on Lexical and Computational Se- mantics, pages 44-54, Seattle, Washington. Associa- tion for Computational Linguistics.
Dordrecht: Foris.
Amy Perfors, Joshua B. Tenenbaum, and Terry Regier. 2011. The learnability of abstract syntactic principles. Cognition, 1 11663)300--338
Gordon Wells and Allayne Bridges. 1981. Learning through interaction: volume 1: the study of language development, volume 1. Cambridge University Press.
Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. 2022a. Improving compositional generalization with latent structure and data augmentation. In Proceed- ings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4341-4362, Seattle, United States, Association for Computational Linguistics.
Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler-gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyz- ing and Interpreting Neural Networks for NLP, pages 211-221, Brussels, Belgium. Association for Com- putational Linguistics
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, and Kristina Toutanova. 2022b. Evaluating the impact of model scale for compositional generalization in semantic parsing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9157-9179, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yuk Wah Wong and Raymond Mooney, 2007. Learn- ing synchronous grammars for semantic parsing with lambda calculus. In Proceedings of the 45th Annual Meeting of the Association of Computational Lin- guistics, pages 960-967, Prague, Czech Republic. Association for Computational Linguisties.
Zhengxuan Wu, Christopher D Manning, and Christo- pher Potts. 2023, Recogs: How incidental details of logical form overshadow an evaluation of semantic interpretation. arXiv preprint 2rXiv:2303.13716.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text urasffrmer. Journal of Machine Learning Research, 21(140):1-67.
Yuekun Yao and Alexander Koller. 2022. Structural gen- eralization is hard for sequence-to-sequene models. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 5048-5062, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Siva Reddy, Osear Täckström, Slav Petrov, Mark Steed- man, and Mirella Lapata. 2017. Universal semantic parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 89-101, Copenhagen, Denmark. Association for Computational Linguistics.
Hao Zheng and Mirella Lapata. 2022. Disentangled sequence sequence learning for compositional gen- eralization. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4256-4268, Dublin, Ireland. Association for Computational Linguistics.
Douglas Roland, Frederic Dick, and Jeffrey L Elman. 2007. Frequency of basic english grammatical struc- tures: A corpus analysis. Journal of memory and language, 57(3):348 -399.
A Training details
Ivan A Sag. 2010. English filler-gap constructions. Lan- guage, pages 486-545.
The architecture of the Trans- former trained from scratch is the same as in orig- inal COGS, which consists of 2 encoder and 2 decoder layers, 4 attention heads per layer, and a feedforward dimension of 512. We use the best combination of hyperparameters from Csordás et al. (2021) on COGS: a learning rate of 0.0001 with no label smoothing, warmup, early stopping. Ab- solute positional embeddings with down scaling scheme (He et al., 2015; Csordás al., 2021) is used due to stability issues observed with relative positional embeddings in recursive depth general- ization cases, a similar phenomenon also noted in
Hyperparameters
Michael Tomasello and Raquel Olguin. 1993. Twenty- morec-month-old children have a grammatical cate- gory of noun. Cognitive Development, 8(4):451-48 464.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, L ukasz Kaiser, and Illia Polosukhin, 2017. Attention is all

---

Csordas and colleagues' experiments. Models are trained for 50k steps with batch size of 128. For the T5 experiments, we finetune T5-base'! using a learning rate of 1.5e-5 and no label smooth- ing, warmup or early stopping. We finetune the model for 50k steps using a batch size of 2048.
(18) A cat wanted sleep.
a. Variable-free LF:
want (agent=cat,
xcomp=sleep (agent=cat))
b. COGS LF:
For the LLaMA experiments, we finetune 1lama- 7b - with LoRA (Hu et al., 2021).13 We set the learning rate 3e-4, LoRA rank to 8, alpha to 32 and dropout 0.1. We finetune the model for 5K steps with a batch size of 64, with 100 warmup steps and no label smoothing or carly stopping. We apply LoRA to W, and W weight matrices in the model.
cat (Φ1) A want.agent (I2, 21) A
want xaom. (42, A
sleep. (T4, 21
In the original COGS LF, entities or events spec- ified by the predicates are represented by indexed constants (17b). In its variable-free counterpart (17a), sleep denotes the sleeping event, cat ex- presses the existence of a cat entity and fills the agent role of the sleeping event. In this way, each predicate in the LF has a set of arguments directly connected their thematic roles without using variables.
All our experiments were run five times, us- ing different random seeds. The final checkpoints from each run were used for evaluation both in-domain test and out-of-domain generalization sets.
B Data generation
Since the variable-free LF often results in a more compact LF, it has been adopted as the primary meaning representation in several prior work (Qiu et al., 2022b; Drozdov et al., 2022). We move away from this practice and keep the original COGS LF the main meaning representation-as briefly mentioned above, the variable-free LF cannot rep- resent binding relations accurately unless some ex- ternal heuristic or constraint introduced for dis- ambiguation. For example, the variable- free LF in (18a) is ambiguous between the meaning of A cat wanted to sleep and A cal wanted a (different) cat to sleep, whereas the COGS LF in (18b) unambigu- ously represents the meaning of A wanted sleep.
B.1 Meaning representations
We use Alto (Gontrum et al., 2017) to implement probabilistic Synchronous Context-Free Grammar (SCFG), which simultaneously generates pairs of English expressions and their corresponding mean- ing representations. Since SCFG cannot handle logical variables (Wong and Mooney, 2007), we use a variable-free representation proposed by Qiu et al. (2022a) (17a) as an intermediate representa- tion during generation. The variable-free logical form (LF) can be deterministically postprocessed into the original COGS LF (17b) with additional information and specific constraints: (i) We rely on the word order in the input sentence label the Skolem constants (i.e. variables); (ii) While the variable-free LF is unable to represent binding re- lations correctly as pointed out by Wu et al. (2023), an additional constraint that disallows duplicate nouns enables the intended binding relations to be identified unambiguously.
While we release the SLOG dataset in both LFs and report the results using the variable-free LF in Appendix E to enable comparison with existing work, we strongly recommend using the original COGS LF for evaluation on SLOG in future work.
B.2 Grammar and sampling details
(17) A cat slept.
SLOG expands upon the COGS vocabulary, which consists of 503 nouns and 113 verbs, to addition- ally include wh-words (who, what) and that used as relative pronoun. In SLOG, for the sake of sim- plicity, we only consider restrictive relative clauses introduced by that regardless of the animacy of the head NPs. For indirect object-extracted instances. we use the preposition stranding structure, such as the boy that Emma give a cake to, rather than the boy to whom Emma gave a cake.
Variable-free LF:
sleep (agent=cat)
b. COGS LF: cat (#1) A 8 aleep.agent (1)
http://////a.ggiaaaeeeeeeeeeeeee alpaca lore
http::///gghhbbh/totttaeeaaeeeee lora

---

The dataset includes the 30,000 examples from the initial COGS training set, and new examples that fall into one of the following categories:
all meaning representations. Consequently, SLOG ensures that PP modifiers are consistently inter- preted as nested NP-attachments-AAe saw [the ball fin the bottle [on the mat]]l, although a VP- attachment might sometimes seem more intuitive depending on the context. This approach ensures that there exists an unambiguous target meaning representation for each expression in the dataset (and this is clearly signaled by the training data), rather than preserving the ambiguity which may complicate the evaluation protocol.
Relative clauses within object NPs, equal in number to instances with PP modifications
Subject and object wh-qqeestons matching the quantity of their corresponding declarative sentences
An equal number of four-level-nestnng recur- sion constructions as the depth-2 instances in initial COGS
C Full results and additional analyses
All models perform very well on the in-domain test set (accuracy over 99%). All experiments this work were conducted on the out-of-domain generalization set, and we report the full results of the experiments discussed in Section 5 in Table 5.
A primitive example for cach ditransitive verbs and verbs accepting complement clause (CP) arguments
Finally, the SLOG sampling process excludes sen- tences with duplicate nouns (e.g. Emma saw Emma.), mentioned in Section B.1.
C.1 Effect of the reformatted exact-match metric
Semantic plausibility Following COGS, our grammar implements simplified selectional restric- tions, focusing mainly animacy constraints. For instance, the subjects of unergative verbs are lim- ited animate entities, as in the cat smiled. As result, our generated sentences may include seman- tically odd but syntactically well-formed sentences, such as non-edible object being the theme of eat or spatial incongruities like a house in a bottle. While these semantic limitations are unlikely to affect models trained from scratch, they may in- fluence the performance of models that have been pretrained naturalistic language data. It's impor- tant note that our primary aim is assess the extent to which models rely on compositional struc- tural generalization to derive meaning. In line with the classic example "colorless green ideas sleep furiously" (Chomsky, 1957), which demonstrates that syntactic structure can be independent of se- mantic coherence, argue that model capable of compositional generalization should be able to map such sentences an appropriate logical form long as they are structurally well-formed.
All models exhibit higher overall accuracy with the reformatted exact-match evaluation compared to the initial metric, notably pretrained models with an increase of over 10 percentage points (Table 5). This suggests that the initial exact-match metric may have underestimated model performance.
C.2 PP Modifiers unseen positions
As discussed in Section 5.2, generalization to PP modification involving unseen long predicate- argument dependencies is challenging for all evalu- ated models. Among such constructions, PP modi- fication the indirect object position (20a) is less challenging than subject position (19). A possi- ble explanation is that the former has a closer sur- face resemblance to direct object modification- modifiers attach to an immediate post-verb NP. In- deed, we observe that higher proportion of in- direct object modifications are partially correct; models correctly predicted the PP-modified NP, but erred the argument structure.
Table 4 also shows that Transformers perform worse on Indirect object in PP datives (20c) com- pared to Passive indirect objects (20b), although neither subcase introduces long predicate-argument dependencies.
Structural disambiguation choice In SLOG, mappings to logical forms are designed to be un- ambiguous, particularly for sentences that are in- herently ambiguous due to prepositional phrase attachment ambiguity, such as Ava saw the ball in the bottle on the mat. This design choice, following COGS, is to use branchnn disambiguation for
(19) PP within subject NPs:
[A cat on a mat]subj ate a fish.
(20) Sub-cases of PP within indirect object NPs:

---

Vanilla
Generalization cases
T5
L.LaMa
AM-Parser
Transformer
Deeper PP recursion
13.1
13.
15.7
16.6
19.8
20.6
100.0=a.0
Deeper tail CP recursion Deeper center embedding
0.2
0.9
0.8
5.3g0.4
3.9.0.4
+2.7
100.0
0.0
0.0+0.
0.0gn.
0.0+00
0.0ga0
0.0g0.
99.5
Shallower PP recursion
98.7.0.
98.7+0.8
90.24222
93.1
97.3+00
98.9+0.6
100.0=0.0
Shallower tail CP recursion
32.6g3.6
55.2
44.8g22
60.922.1
85.4
+8.1
100.0=0.0
Shallower center emhedding
0.0gno
0.0go.
0.0go.
64. +19.1
0.0ga.
50.7
100.0
subject NPs
0.0g0.0
0.0g0.0
0.0g0.
0.8g0.5
12.3+g44
28.943.5
57.6+8.1
PP in indirect object NPs
42.5g2.2
42.5+22
50.1
53.8214
55.0
71.2
90.418.1
RC in subject NPs
0.0
0.0goa
0.0gn.
0.2g0.2
3.4
29.5
55.8
RC indirect object NPs
34.4
34.8
35.1
36.642.1
48.6+1.9
55.042.1
74.4+6.4
Indirect object-extracted RC
4.7.5.6
4.7.5.7
0.0
0.0
0.1
2.5
0.0g0.0
Indirect object wh-quuesions
35.9
42.4
0.0gn.
0.4
27.9
73.5
41.44-224
Active subject wf-qquestions
96.722.
97.1
90.5+4.
18.7
92.816.4
93.3100
99.810.
Passive subject que-ttuss
27.4117
31.945.4
20.3
100.0.0.0
4.821.
15.3
100.0.0.1
Direct object wh--questions Wh-questiuns with modified NPs
2.843.4
16.0
47.2
98.5
0.5
8. 8.6
29. 29.4
17.640.
17.8+1.3
20.5+1.0
36.8+0.4
15.840.6
20.8
55.64125
Wh-questions long movement
4.0g7.8
4.929.5
23.3243
24.9
0.8.1.4
3.0
0.0g0.0
Overall
2..2
27.1
23.4
40.6
27.6
40.1
70.8
Table 5: Mean accuracy (%) using exact-match is shown in gray, accuracy using reformatted exact-match described in Section 4 is shown black. AM-Parser's graph-based output yields identical scores for both metrics hence only a single column is reported.
a. Indirect object in double object datives: Emma gave cat on the mat liobj fish.
C.3
RC Modifiers in unseen positions
Generalizing RC modifiers to unseen positions presents a similar challenge as PP modification cases, due to unobserved long-distance dependen- cies. As shown in Table 6, all models exhibit significant performance discrepancy between constructions involving unseen long predicate- argument dependencies and those that do not.
b. Passive indirect objects: A fish was given cat on the mat liobj. Indirect object in PP datives: Emma gave a fish to cat on the mat liobj-
The predominant error pattern in the former sub- case was incorrect attachment of PP modifiers to the direct object NP. For example (21b), modifier the mat denoted by 29 was attached to fish in- stead of the cat. This suggests that Transformers additionally fall prey to inference patterns akin to a modification rule "attach PPs to NPs immediate post-verb position", which is compatible with the training data but does not lead to correct general- ization.
For novel positions that introduce long predicate- argument dependencies, RC modification in the indirect object appears more difficult than in the subject position, contrary to the case with PP mod- ifiers. The primary error pattern (22) demonstrates that models struggle to detect the RC boundary when the relative clause ends with verb. They systematically misinterpret the indirect object fi sh of the main verb gave the direct object of the adjacent embedded verb slept.
(21) Gold LF and model-predicter LF for Emma gave a fish to the cat on the mat:
(22) Gold LF and model-predicted LF for Emma gave a cat that slept a fifish:
a. Gold: caat (ze); Amat (xa);
a. Gold:
glve.agent (31, Rmma)
give.agent (x), Emma)
A dive.reccpiien
(X1,
A give. theme (t)
A give.theme (w1,47)A cat (#a) A cat. nmod (xa, A
give. recipient (11
(ish(ra) A
(X6,
sleep. (3) A (irh
b. Out: <cat (xa); wmat (29);
b. Out:
give.agent (21, Emma)
give. (æ1, Emma)
A give.theme (x1,4s) A
A give.theme (I4,43) A give. recipient (21,
cat (x3) A cat. (x3. s lleep.agen (26,43) A s sleep. theme (25, Afish
(ish A fish. omod. (x)

---

Long pred-arg dependency?
Vanilla
AM
Generalization cases
T5
LLaMa
Transformer
parser
Sub-cssc: Pasive inditect objects
X
12.
74.2
97.1
19.5
A fish was given to cat that slept liobj-
Sub-csse: Infircct object in PP dutiess
X
27. 27.0=9.8
38.946.3
72.7=7.8
99.3
Emna gave fish to a that slept liobj-
Sub-cssc: Indirec object in double object dutiees
7.9=8.5
0.2+0.2
0.3+0.3
28.9
Emma gave that slept liobj fish.
Subject
0.0+0
0.2+0.2
29.4=3.4
51.7
[A cat that subp ate fish.
Table 6: Performance of RC modification generalization broken down by construction.
C.4 Gap constructions
types of tokens can appear post-comma in the out- put LF space: 2,, ? denoting wh-words, or a proper noun (PropN), such as Enma. The subsequence theme (y) is 20 times more frequent than heem (0i, ?) and theme (æ; PropN). This discrepancy does not affect all models equally; in fact, T5 can generalize correctly for some con- structions despite this skewed label distribution, achieving near-perfect accuracy for direct object wh-questions. However, when it comes less fre- quent constructions indirect object wh-questions, T5 overgeneralizes. In 94.6% of these cases, it erroneously produces the observed direct object wh-questions pattern theme (x,, ?), instead of the correct but unseen recipient (xi, ?). This observation aligns with the findings of Wu et al. (2023); Yao and Koller (2022), who noted that the decoder of Transformer models tends to exhibit a heavy bias towards generating observed n-grams.
While performing poorly on indirect object- extracted relative clauses (23), all tested models systematically mirror the direct object-extracted RC pattern in training, as demonstrated by the in- correct output (23b). They furthermore show dis- tinct difficulties when handling wh-questions cases, will be discussed the remainder of this section.
(23)
Input: Ella cooked the servant that Emma gave a tool to
a. Gold:
+servant (23) cook. agent (21, Ella) A cook. theme (21,
A servant nmod { v3,2s)
A give. agent (æu, Emma)
A give. theee theme (io, A recipient (æa, va) A tool (2s)
b. Models output:
*servant (ra) cook agent (2),
Ella) A cook, theme (31)
A servant nmod us,2a)
A give.agent (æa, Emma)
A give. theme (ma) A
C.4.2 Wh-questions with long-distance movement
give. recipient 3s) tool (Zg)
All models achieve very low accuracy when gen- eralizing to longer filler-gap dependency across CPs, but an error analysis shows that Transformer and structure-aware models face distinct challenges. As shown in (25b), the Transformer trained from scratch commonly misinterprets the complemen- tizer that (corresponding ccomp in LF) relative pronoun (nmod). Additionally, tends to interpret the wh-word as the direct object the CP verb, egg. say. In the most common errors for T5 and LLaMa (25c), the whole gap conjunct (paint theme (27, ?) ) is missing, re- vealing their difficulties in establishing long-range filler-gap dependencies between the initial wh- word and the embedded gap position. On the other hand, AM-Parser cannot decode non-projective de- pendencies, thus has 0% accuracy (see more de- tailed discussion of the issue $D).
C.4.1 Direct and indirect wh-questions
The Transformer trained from scratch and LLaMa frequently misinterpret the theme role in direct ob- ject wh-questions. For example, they often fail to map wordsrds to "?" as illustrated in (24b):
(24) Input: What did Emma sell to Liam
Gold:sel1, theme (xs. 21
sel sell.agent (X3, Emmal se reliiincip (x3, Liam)
b. Output of Vanilla Transformer and LLaMa:
dellitheme (x3, xs) A
se agln.agent (x3, Enma)
relipinct (x3, Liam)
AM-Parser's output output:
aglnt (xs,
A
sell.theme (x3, Enma)
sell. sell. recipient (xs, Liam)
This error pattern can be traced back to frequency of the subsequences in the training data. Three

---

(25) Input: What did Liam say that the bear painted
cook
Gold:
*bear (Xa); say.agent (Na, Liam) say.ccomp (Na, X7)
theme Vigent
A paint. agent (x7, Xo! paint. theme (X7, ?)
S2
b. Output of vanilla Transformer: +bear (Xe)
say .ggnnt (Xa, Liam) A (xa, say.nmod (x3,
Figure 4; Example of a supertag in an AM dependency tree.
A mrint.agent X6)
paint.thems (x7, ?)
c. Output T5 and LLaMa: +bear (xa); say.agent (xa, Liam) ccy.p (x), A paint.agent agent (x7,
b. Output of Vanilla Transformer and LLaMa: eat theme (82, x4) A eat aagntt (x2, Emna)
C.4.3 Wh-questions with modified NPs
In wh-questions with PP and RC modifiers, even though the SLOG training set only contains wh- questions with unmodified NPs, all models gener- alize well (accuracy > 80%) to direct object NPs with modifiers (e.g.. Who ale cake on the table?). These are cases where the modification pattern is observed in training as part of declarative sen- tences. In contrast, performance declines when models encounter wh-questions with modifiers in the indirect object position (i.e., modification struc- ture not observed as part of declarative sentences). Similarly, for wh-questions with subject position modifiers, the performance is very low: both T5 and vanilla Transformer achieve near-zero accu- racy, and LLaMa achieves around 5%.
As discussed in Section C.4, this enror pattern may result from the highly imbalanced label distribu- tion in training output space. Both LLaMa and Transformer trained from scratch are inclined to repeat the substantially more common subsequence heme (Iz, Zzz) over teme (æi ?).
D AM-Parser-speccfiic issues
While the AM-Parser achieves strong performance most generalization types, it faces systematic difficulties in handling novel gap structures. In particular, its accuracy wh-questions involv- ing long-distance movement and indirect object- extracted relative clauses is always 0. Additionally, its accuracy significantly fluctuates across runs for both direct and indirect object wh-questions. Here, we give a detailed explanation of error patterns for these challenging types.
This observation mirrors the patterns discussed in $5.2, attributed to difficulties introduced by seen subject-verb dependencies across PPs or RCs. In contrast, the structure-and aware model exhibits sig- nificantly better performance in wh-question with subject modification.
Background The AM-Parser maps input sen- tences graphs by parsing each input sentence to an AM dependency tree, which is then deter- ministically evaluated to a graph (Groschwitz et al., 2018). In the AM dependency tree, each token is la- beled with supertag- small graph illustrated in Figure ththat captures the lexical meaning of the token. The tree's edges represent the compositional structure of the sentence, which specifies how the meaning of the sentence recursively computed from the supertags. The supertag in Figure 4 repre- sents the meaning of cooked the sentence Ella cooked the servant that Emma gave tool to. The blue markers "S1" and "S2" indicate that two argu- ments are still needed to fill the agent and theme roles of cook.
C.4.4 Passive subject wh-questions
For subject wh-questions, which exhibit no gap, T5 and AM-Parser perform near-perfectly on both active and passive subject questions. Transformer trained from scratch and LLaMa also perform well on active subject questions, but achieve much lower performance on passive subject questions. This performance discrepancy is the most evident in sub-cases where passive subjects function theme (e.g., (26)) the Transformer trained from scratch has near-zero accuracy for these sub-cases, sys- tematically failing to map wh-words to '?' as in (26b):
(26) Input:
What was caten by Emma
Wh-questions with long movement We show an example of a predicted AM dependency tree for a wh-question with long movement in Figure 5
a. Gold: eat.theme (x2r ?) A
eat.agent (X2, Emma)

---

and the corresponding gold AM dependency tree in Figure 6. As discussed in Section 5.3, the parser used in this paper limited to predicting projective AM dependency trees, but the gold AM depen- dency tree in Figure 6 is non-projective (the edge snapped -> Who crosses the edge root -> appreciate). Thus it is impossible for the AM- Parser to predict the correct compositional struc- ture.
AM-Parser is limited to using supertags that it ob- served during training (possibly with different node labels to accommodate novel lexical material). For the direct wh-question case, the correct supertag was actually present in the training data, but was much less frequent than the erroneous one in Fig- ure 7b. We observe similar discrepancy in the frequency distribution between predicted and gold supertags for indirect object-extracted RCs and in- direct wh-questions.
Instead the A* parser, one could instead use the fixed-trea tree decoder of Groschwitz et al. (2018), which is capable of predicting non-projective AM dependency trees. This parser achieves nonzero accuracy (36%) on wh-questions with long move- ment, confirming our hypothesis that the projec- tivity is the issue. However, the A parser outper- forms the fixed-tree decoder on most other gener- alization types, which is why we only report its re- sults in the main body the paper. The transition- based AM-Parser of Lindemann et al. (2020) can also predict non-projective trees, but uses a differ- ent probability model that is incompatible with the training algorithm of Groschwitz et al. (2021) that use here.
We conjecture that the AM-Parser was overly sensitive to the supertag distribution in the training data, pointing to a further architectural limitation. Thus, while the AM-Parser can compensate the distribution shift of the meaning representations whole, SLOG exposes its weakness to distribution shifts in the lexical supertags.
E Results with variable-free LFs
Table reports the accuracy on SLOG using variable-free logical forms. The AM-Parser is un- able to handle the variable-free format and there- fore is omitted. The hyperparameters for the three tested models are the same the experiments de- scribed in Appendix A.
Note that the A* AM-Parser shares its limitation to projective structures with many other structure- aware models. For instance, the LeAR model of Liu et al. (2021) uses phrase-structur trees as com- positional structures, and the CSL-T5 parser of Qiu et al. (2022a) uses phrase-structure trees during the data augmentation process. Because phrase struc- ture trees are equivalent to projective dependency trees, they are likely encounter similar difficul- ties on SLOG.
The variable-free LF, as discussed in Appendix B and Wu et al. (2023), exhibits certain limitations and umbiguities which render direct comparisons with variable-based LF results inappropriate. Re- gardless, all three models achieve higher accuracy scores on the variable-free LFs compared to the COGS LFs, with pretrained models experiencing particularly significant boost. This aligns with the observations of Qiu al. 2022b.
Despite the change in LF, the overall trends and challenges remain consistent. The Transformer trained from scratch struggles with the same gen- eralization cases, failing to extrapolate to deeper recursion depths and struggling with cases in- volving unseen long-distance dependencies. Pre- trained models, while exhibiting better overall per- formance, continue struggle with more struc- turally complex generalization cases in their re- spective categories. These include deeper center embedding, indirect object-extracted RC and wh- questions with long movement.
Direct & indirect wh-questions and indirect object-extracted RC The AM-Parser consis- tently shows zero accuracy for indirect object- extracted RCs and exhibits big performance fluc- tuation across different runs for direct and indirect wh-questions. This is because in these general- ization types, the gold meaning representations in the test set require supertags that are infrequent in training.
We show an example of AM dependency trees for a direct object wh-question in Figure 7, with gold supertags in Figure 7a and predicated su- pertags in Figure 7b. The issue here is that the model predicts the wrong supertag for sell, treating What as its agent instead of theme, and Emma as its theme rather than agent, which results in the erroneous output LF as shown in (24c). The

---

(IDDa)
BLOD
(PP
APPs2
W Who
did
the
cut
appmxiate
that
Oliver
snapped
(S00)
(S10: S20)
(S0(), S2()
appeacinte
Who
promp
Olver
S0
S1
Figure Example of predicted AM dependency tree for wh-questions with long movement
APP
M00
AMN
AT/F
Who
the
call
appreciate
thab
Oliver
snapped
(S0()
(S1(), $2()
(S1(), S2()
the
Who
Gleer
Damelpamm
SO
S1
S2
S2
SI
Figure 6: Example of gold AM dependency tree for wh-questions with long movement
APP
APPac
ArPa
What
did
Emma
Liam
(S0()
T
(S1())
(SUO, $10, $2()
(S20)
T
What
Emua
theme
pecipient
Liam
SO
SI
S2
What
Emma
theme
recipient
Liam
SO
S1
S2
Figure 7: AM dependency tree for direct object wh- wh-qquetion. (a) displays the gold supertags and (b) shows the incorrect predicted supertags.

---

Vanilla
Generalization cases
T5
LLaMa
Transformer
Deeper PP recursion
7.8
63.012.9
90.913.3
Deeper tail CP recursion Deeper center embedding
1.0
46.2
44.1
0.0
7.8
9.4
Shallower PP recursion
98.2+1.6
99.640.9
100.0+0.0
Shallower tail CP recursion
89.343.3
99.3
100.0+0.0
Shallower center embedding
0.1
99.840.3
99.8+0.4
PP in subject NPs
0.2
73.2
93.4
PP in indirect object NPs RC in subject NPs
29.3
97.412.1
98.1+1.9
0.1
60.8
73.9
RC in indirect object NPs
4.0+1.9
71.940.8
73.6+3.9
Indirect object-extracted RC
0.0+0.0
62.447.5
3.34..8
Indirect object wh-questions
34.1
93.444.8
83,8+11.3
Active subject wh-questions
99.0+0.5
99.8+0.3
96.242.6
Passive subject wh-questions
57.3 12.88
99.940.1
96.0
Direct object wh-questions
41.8+3.8
48.4+0.4
44.1
Wh-questions with modified NPs
18.3
68.0
69.4
Wh-questions long movement
7.4+3.7
45.644.
35.7
Total
28.7+4.1
72.7+1.1
71.3
Table 7: Mean accuracy (%) SLOG using the variable-free logical form of Qiu al. (2022a).


Reviews:
[{"review_id": "n90ghSDyvX", "reviewer": "Reviewer_up6o", "summary": "", "questions": "a.) How do the bounds in the syntactic patterns compare to those in natural language?  For instance, are 4+ depth center-embedding clauses reasonably understandable by humans?  Similarly, it seems the max output token lengths is very long -- maybe beyond what a human could keep track of?  Even for a benchmark with a more theoretical leaning, this is generally good information to present, and is especially pertinent when using pre-trained models as baselines.  This is touched upon a little in the related work section (~504) but is not discussed specifically in the context of the depth and length bounds of this work.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> How do the bounds in the syntactic patterns compare to those in natural language? For instance, are 4+ depth center-embedding clauses reasonably understandable by humans? Similarly, it seems the max output token lengths is very long -- maybe beyond what a human could keep track of?\n\nWe thank the reviewer for drawing attention to this important aspect.\n\nIn natural language, the depth of embedding is rarely greater than five (Karlsson, 2010). SLOG tests deeper recursive patterns up to depth 12. While this may surpass human processing abilities due to memory constraints, it remains grammatical, echoing Chomsky’s competence versus performance distinction. We also note that our goal with SLOG is to evaluate the linguistic competence  of NLP models, whose goal is not to simulate human performance limitations. That is, it is desirable that these structures are correctly interpreted by NLP models, and therefore the difficulty of human processing should not limit the target capacity of the models we build.\n\nWe will incorporate the above discussion into the revised draft, as well as providing explicit details on depth and length bounds.\n\n> For those who are not familiar with COGS, it could be made clearer what the inputs/outputs are for the prediction task. [...] The actual predictive task is not well described. Some assumption of familiarity with COGS, its contents and the task definition, seems implicit.\n\nWe will make the specific predictive task of COGS and how it relates to SLOG clearer in the revision. \n\nWe thank the reviewer again for their insightful comments, and we will furthermore incorporate your suggestions for stylistic and presentation improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A well-thoughtout benchmark which extends existing benchmarks in important ways for studying the theoretical linguistic capabilities of recent models.\n\n- Generally well-written paper, lots of good references, clear understanding of the state of the field and related literature.\n\n- A respectable set of baseline models.  Clear demonstration of the inability of existing models to easily solve the more difficult benchmark examples.", "reasons_to_reject": "- In total, this is only a small extension of the existing benchmark.\n\n- Many of the task examples appear to be of theoretical interest only.\n\n- The actual predictive task is not well described.  Some assumption of familiarity with COGS, its contents and the task definition, seems implicit.", "questions_to_authors": "a.) How do the bounds in the syntactic patterns compare to those in natural language?  For instance, are 4+ depth center-embedding clauses reasonably understandable by humans?  Similarly, it seems the max output token lengths is very long -- maybe beyond what a human could keep track of?  Even for a benchmark with a more theoretical leaning, this is generally good information to present, and is especially pertinent when using pre-trained models as baselines.  This is touched upon a little in the related work section (~504) but is not discussed specifically in the context of the depth and length bounds of this work.\n", "paper_topic_and_main_contributions": "This paper proses SLOG, a benchmark dataset made for evaluating a model's compositional generalization performance on a set of syntactic patterns in the task of semantic parsing.  The extended syntactic patterns focus on recursion and filler-gap dependencies.  Compared to previous work, recursion depths are deeper and include center-embedding patterns, NPs can have relative clause modifiers, and PP-modified NPs can appear outside of the object position.  The data is produced by synchronous tree grammars which produce both the example and its corresponding logical form.  A wide set of relevant models is evaluated, and existing methods are shown to fail to reach high performance on the task, making the benchmark a useful resource for testing the linguistic understanding of ever more performant models, and where it is difficult to find cases unseen in ever increasing training set sizes.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Some mild confusion in presentation in that SLOG is presented as an extension of COGS, but this conflicts with the initial motivation of COGS being for lexical generalization and SLOG being for structural generalization.  The language is also midly confusing in other areas -- L069, regarding the AM-Parser performing well on the structural generalization cases in COGS.  It becomes clearer later in the paper that COGS does test structural generalization to a limited depth, but the exposition can come across as contradictory at some points.  \n\n- For those who are not familiar with COGS, it could be made clearer what the inputs/outputs are for the prediction task.\n\n- It would have been good to include some structured stack-based transduction models when dealing with modeling recursion.\n\nL314, confusing phrasing\n\nL545, typo, compositional"}}, {"review_id": "uz3dO1kSOX", "reviewer": "Reviewer_b6uL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> It is probably time to expand these datasets outside of English. Only measuring the performance of one performance-aware parser is a little disappointing.\n\nWe agree with the importance of expanding our corpus beyond English and incorporating diverse symbolic parsers. We are considering these aspects as a part of future work. Some of the internal discussions we had regarding these points are:\n* Including Chinese and German given their distinct linguistic properties compared to English. For example, Chinese uses left-branching for modifiers, center-embedding in German comes with gender marking.\n* Adding results from additional structure-aware parsers, for instance, LeAR (Liu et al. 2021) and the decompositional prompting approach of Drozdov et al. (2022). \n\nWe thank the reviewer again for their insightful comments, and we would love to include results using other structure-aware parsers to the camera-ready as discussed above, if the paper is accepted.\n\n\nLiu et al. (2021): https://aclanthology.org/2021.findings-acl.97/\n\nDrozdov et al. (2022): https://arxiv.org/abs/2209.15003", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "New datasets are always appreciated. I think this one is thoughtfully constructed, non-trivial, and will be of use to other researchers. ", "reasons_to_reject": "It is probably time to expand these datasets outside of English. Only measuring the performance of one performance-aware parser is a little disappointing. In general, the experiments in this paper are not particularly exciting, though given that it is a dataset paper, I think this is acceptable. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides a dataset (SLOG) aimed at probing the compositional generalization of semantic parsing models at the \"structural\" level, compared to the mostly \"lexical\" level that COGS (previous work) is meant to probe for. The authors show that standard transformer baselines struggle deeply on this set. A structure-aware parser that achieves near perfect accuracy on COGS fails to do so on COGS, indicating that this set is better able to test parsers at the upper end of the spectrum (on the axis of compositional generalization).  ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wICUXB09mw", "reviewer": "Reviewer_t68a", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n>  I might have liked more justification of whether it was important that the new unseen patterns were learnable. Particularly in terms of the recursive forms, some of which are highly unnatural (who embeds 12 center clauses in practice?) and seem to be asking the semantic parser to OVER-generalize\n\nThe generalization cases in SLOG are largely motivated by the observed frequency gaps in natural language, aiming to generalize from common patterns to their less common counterparts. For instance, typical recursion depths to deep ones, common extraction positions to less common ones. Given the natural data distribution, less frequent cases are more likely to be underrepresented in the training data---like you also point out, large depths of center embeddings are unlikely to occur naturally. However, given such an input (insofar as it is well-formed), the ideal behavior of an NLP model is to still provide an appropriate meaning representation rather than to fail. Therefore, it’s desirable that models can extrapolate from frequent patterns to interpret the rare scenarios. Thus, we believe it's worth assessing models on this capability (regardless of human ability to generalize under the same conditions or human performance limitations).\n\nWe refer the reviewer to our response to R1 regarding more detailed rationale behind testing deep recursive patterns that may be unnatural and/or humans may face limitations in processing.\n\n\n> I wish that there had more justification of why the new unseen patterns were deemed to be learnable from the patterns included in the training data. For instance, I guess there are some Occam's razor reasons to see sentences like \"What did the cat see?\" and \"Emma said that the cat saw a fish\" and then to infer the meaning of \"What did Emma say that the cat found?\", but why should such a generalization be preferred over others, in the absence of any examples?\n\nThe target generalizations all share the property that they can be compositionally built up from parts that constitute the training examples. Still, like you correctly point out, multiple generalizations may be plausible given the building blocks available in the training data. Nevertheless, arriving at the meaning that is compatible with how humans (i.e., users of the system) would interpret the target example is the ideal, desired behavior of an NLP system when the training data contains distributional gaps (this also relates to our motiviation regarding frequency gaps discussed above). This is equivalent to testing whether our models have the right inductive bias to arrive at the desired interpretation of unobserved complex expressions. We again note that this is a desideratum we set for NLP models (we want our models to arrive at the target meaning given limited data), which may deviate from questions regarding whether human learners can or do generalize this way given the same set of observations.\n\n\n> A sentence like \"Ava saw the ball in the bottle on the table.\" is inherently ambiguous -- are we rewarding systems for jumping to the conclusion of a tail-recursive interpretation when that isn't how human language would operate?\n\nRegarding structural ambiguity in regular English, any corpus with complex sentences will face this issue. Our choices were either to uniformly disambiguate all sentences, which we did, or to force parsers to learn disambiguation, which is potentially diverting from our paper's main focus.\n\n\nWe thank the reviewer again for the insightful comments and will integrate these justifications into the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The extension of COGS is nontrivial, including 17 new phenomena into the SLOG benchmark. Moreover, they improve the evaluation criteria (which was \"exact match\" in COGS) to make it more robust by scoring any parse as correct if it is semantically equivalent to the gold parse. They also show that SLOG offers a different perspective than COGS. While COGS scores the fine-tuned LLM approaches similarly to the AM-Parser, SLOG highlights more of the benefits provided by a structure-aware approach like AM-Parser. I suppose one could cynically suggest that the SLOG benchmark was designed to favor structure-aware approaches and highlight their advantages, but regardless it offers an alternative perspective and a finer-grained way to assess the relative strengths and weaknesses of different approaches to semantic parsing.", "reasons_to_reject": "I wish that there had more justification of *why* the new unseen patterns were deemed to be learnable from the patterns included in the training data. For instance, I guess there are some Occam's razor reasons to see sentences like \"What did the cat see?\" and \"Emma said that the cat saw a fish\" and then to infer the meaning of \"What did Emma say that the cat found?\", but why should such a generalization be preferred over others, in the absence of any examples?\n\nAlso, I might have liked more justification of *whether it was important* that the new unseen patterns were learnable. Particularly in terms of the recursive forms, some of which are highly unnatural (who embeds 12 center clauses in practice?) and seem to be asking the semantic parser to OVER-generalize. A sentence like \"Ava saw the ball in the bottle on the table.\" is inherently ambiguous -- are we rewarding systems for jumping to the conclusion of a tail-recursive interpretation when that isn't how human language would operate?\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper expands the COGS benchmark (Kim and Linzen, 2020) to evaluate semantic parsers on a broader set of compositional generalization phenomena. For instance, if training data contains semantic parses for sentences of the form \"Noah saw the cat that froze\", i.e. NP VERB NP_WITH_RELATIVE_CLAUSE, then the benchmark tests whether it can generalize to sentences of the form \"The cat that froze smiled.\", even if it hasn't seen a sentence of the form NP_WITH_RELATIVE_CLAUSE VERB and its corresponding semantic parse. In addition to extending the COGS benchmark, they use their new benchmark (called SLOG) to conduct an empirical investigation of the effectiveness of several semantic parsing approaches: a vanilla transformer, some fine-tuned LLMs, and something called the AM-Parser, based on a model that explicitly incorporates syntactic structure.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 2.3, example (8):\nLiam saw the boy that Emma GAVE a cake to."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a benchmark dataset for structural generalization in semantic parsing, as an extension to an earlier dataset focusing on lexical generalization. The benchmark is thoughtfully constructed, non-trivial, and useful for studying the theoretical linguistic capabilities of recent models. The reviewers found the paper also generally well-written.

The reviewers noted a few issues with the paper, most of which concern the presentation and can be addressed in the camera-ready version:
- The similarities and differences between COGS and SLOG should be better highlighted and made accessible for readers not familiar with COGS.
- Some of the included patterns seem at odds with human generalization capabilities (e.g. 4+ depth center-embedding clauses, inherently ambiguous sentences that are uniformly disambiguated). A more thorough discussion and justification of the choices made by the authors would be welcome.
- A reviewer also wished an extension of the dataset to languages other than English. While I second this wish, it cannot be viewed as a flaw of the current paper.
Decision: Accept-Main