BLM-s/lE: A structured dataset of English spray-load verb alternations for testing generalization in LLMs
Authors: Giuseppe Samo
Vivi Nastase
Chunyang Jiang
Paola Merlo
Venue: EMNLP 2023 Findings (2023)
Keywords: verb alternation
generating rules
sentence embeddings

Abstract: Current NLP models appear to be achieving performance comparable to human capabilities on well-established benchmarks. New benchmarks are now necessary to test  deeper layers of understanding  of natural languages by these models.

Blackbird's Language Matrices are a  recently developed framework that draws inspiration from tests of human analytic intelligence. The BLM task has revealed that successful performances in previously studied linguistic problems do not yet stem from a deep understanding of the generative factors that define these problems. 

In this study, we define a new BLM task for predicate-argument structure, and develop a structured dataset for its investigation, concentrating on the spray-load verb alternations in English, as a case study.  The context sentences  include one alternant from the spray-load alternation and the  target sentence is the other alternant, to be chosen among a minimally contrastive and adversarial  set of answers. We describe the generation process of the dataset and the reasoning behind the generating rules. The dataset aims to facilitate investigations into how verb information is encoded in sentence embeddings and how models generalize to the complex  properties of argument structures. 

Benchmarking experiments conducted on the dataset and qualitative error analysis on the answer set reveal the inherent challenges associated with the problem even for current high-performing representations.

BLM-s/EE: A structured dataset of English spray-load verb alternations for testing generalization in LLMs
Giuseppe Samo, Vivi Nastase, Chunyang Jiang, Paola Merlo Department of Linguistics
University of Geneva
(Giuseppe. Samo, Chunyang- Jiang, Paola. Merlo)@unige.ch vivi. a nastase@gma 1. com
Abstract
Current NLP models appear to achieving performance comparable to human capabilities on well-established benchmarks. New bench- marks are now necessary to test deeper layers of understanding of natural languages by these models.
CONTEXT
ANS WER SET
Tean load lugs with the food
Visioes can lond suppl
bos
Blackbird's is Language Matrices are a recently developed framework that draws inspiration from tests of human analytic intelligence. The BLM task has revealed that successful perfor- mances in previously studied linguistic prob- lems do not yet stem from a deep understanding the generative factors that define these prob- lems.
Clothes were landed by me Boses wem lonked the can by visiiess Booes were loaded bags The car was loaded by usera box loaded with clothes The loaded
was londed clothes in bags
Visioon con load the food the cent
Users can load the mois for esperienttaiinn
Supplies can load the customer the suiicose
tooks hoad
Figure I: Upper panel: An example of Raven's 's Progres- sive Matrices. The task is to identify missing element in visual pattern from a list of candidate options (on right) based on given matrix (on left). The matrix is created using specific rules for the placement of visual elements (e.g. the red dot and the blue square). Lower panel: An example of the Blackbird Language Matrices. The task is to identify a missing element in a linguistic pattern from list of candidate options (on the right) based the provided matrix (on the left). The matrix of sentences is constructed using rules of syntactic and semantic nature.
this study, we define new BLM task for predicate-argument structure, and develop structured dataset for its investigation, concen- trating on the spray-load verb alternations in English, as a case study. The context sentences include one alternant from the spray-load al- ternation and the target sentence is the other alternant, to be chosen among a minimally con- trastive and adversarial set of answers. We describe the generation process of the dataset and the reasoning behind the generating rules. The dataset aims facilitate investigations into how verb information is encoded in sentence embeddings and how models generalize to the complex properties of argument structures.
New benchmarks are now necessary to investi- gate deeper layers of comprehension of natural languages (Ruder, 2021).
A recently developed framework is the Black- bird's Language Matrices (BLM; Merlo et al. 2022; Merlo 2023a,b). This approach draws inspiration from tests of analytic intelligence, specifically the task of detecting visual patterns (Raven's Progres- sive Matrices (RPM), Raven 1938). The BLM task has revealed that correct predictions in previously studied linguistic problems do not yet stem from a deep understanding of the generative factors that define these problems. For example, previous work has demonstrated that RNN can predict the correct verb form with high accuracy based on the gram- matical number of the subject (Linzen et al. 2016; Gulordava et al. 2018; see Linzen and Baroni 2021
Benchmarking experiments conducted on the dataset and qualitative enror analysis on the answer set reveal the inherent challenges asso- ciated with the problem even for current high- performing representations.
1 Introduction
In recent years, fuelled by the rise of deep learn- ing methods, NLP models have undergone re- markable transformation. becoming exceptionally powerful and achieving performance comparable to human capabilities on well-established bench- marks (Wang et al., 2019; Rajpurkar et al., 2018).

---

for an overview). A more in depth exploration us- ing BLM task showed that despite the high results of previous studies, detecting the factors relevant to the subject-verb agreement rule is a more chal- lenging problem (An et al., 2023).
We define new BLM matrix template, ap- plied to spray/ladd verbs, which can also be of general interest as blueprint for many al- ternations.
We create a large data set on this phenomenon, which supports probing of neural network models.
In this paper, we adopt BLMs to investigate current models reach deeper understanding of a core property of language, a verb's ss argument struc- ture. The argument structure of a verb, the mapping of semantic roles to grammatical functions frames, is core building block in claiming understand- ing of any sentence. Argument structure learning finds a particularly difficult case in those verbs that allow alternations, the multiple mappings of semantic roles to grammatical function and subcat- egorisation frame for the same verb. We exemplify this problem by studying linguistic phenomenon in English, known as the spray/load alternation (Rappaport and Levin, 1988; Levin, 1993). Analo- gously to the subject-verb agreement problem, revisit the recent results that typologies of neural architectures handle binary classification of this phenomenon with ease (Kann et al., 2019; et al., 2022).
We provide a qualitative error analysis of the learned knowledge of core syntactic and se- mantic properties for different neural network models. This analysis shows that the learned properties in more complex configurations are limited to the syntactic aspects of the argu- ment structure.
2 BLM-s/IE generation process
in RPM identify patterns based on the available information to identify the missing piece that completes the matrix (Kisielewska et al., 2016). Similarly, BLMs correspond to a problems where only one answer satisfies the constraints defined by the given contexts. In a BLM problem, a lin- guistic phenomenon is presented as incomplete sequence of sentences (contexr), deliberately de- signed to follow given linguistic rules and given linguistic properties.
The syntactic behaviour involves an alternation in which a selected class of verbs combines three arguments to describe an event where an AGENT causes motion of a THEME to LACION)) Both THEME and LOC can function as the syntactic di- rect objects of the structure, resulting in two pos- sible alternating configurations. In one alternate,
CONTEXT
Alternte |
2
Passivized THEME with overt AGENT
3
Passivized THEME with overt AGENT and LOC
THEME immediately follows the verb, and Loc is introduced by preposition (The student sprays the paint onto the wall). In the other alternate, Loc follows the verb, and THEME is introduced by a preposition (The student sprays the wall with the painr).
Passivized THEME with overt LoC
5
Passivized LOC with overt AGENT
Passivized LOC with overt AGENT and THEME Passivized LOC with overt THEME
7
?
Alternate 2
Figure 2: Contexts template for BLM-s/IE
We develop BLM data that expose the model to syntactic properties of the arguments THEME and Loc, implicitly showing that the two alternates share common properties. To learn the alternation then, the model must be able generalize from these shared properties.
The task is to identify the missing element that continues the context pattern based on the observed relationships. The missing element is the only cor- rect answer that represents the solution to the con- text pattern, among many elements in a carefully curated answer set consisting of minimally differ- ing sentences (Merlo et al. 2023a,b).
We present the generation process of spray/load BLMs in Section and the generation of the asso- ciated dataset for spray/lodd alternation in Section 3. Experiments with different types of architec- tures are presented in Section 4. The code and the data are available here: https: //g thub. com/ CLCL- ELCL-Genevall Ceneeva//LLI-SMFDDiiettaanggng SNFDisentangling
2.1 The context indicators of underlying rules
While the spray/load alternation is the expression of complex syntactic and semantic interactions in verb argument structure, visible surface indi- cator of these properties is the use the passive. Passivizability in English commonly assumed to
The contribution of this paper are three-fold,

---

CONTEXT
CONTEXT
NP-Ageet
Verb
NP-Loc
PP- Theme
NP-Agent
Verb
NP-Theme
PP-Loc
NP-Theme
VerbPass
PP-Agent
NP-Theme
VerbPass
PP-Agent
NP-Theme
VerbPass
PP-Loc
PP-Agent
3
NP-Theme
VerbPass
PP-Loc
PP-Agent
NP-Theme
VerbPass
PP-Loc
Nh-Theme
VerbPass
PP-Loc
NP-Loc
VerbPass
PP-Agent
5
NP-Loc
VerbPass
PP-Agent
NP-Loc
VerbPass
PP-Theme
PP-Agent
6
NP-Loc
VerbPass
PP-Theme
PP-Agent
NP-Loc
VerbPass
PP-Theme
7
NP-Loc VerbPass
PP-Theme
78
Ag-Agent
Verb
NP-Theme PP-Loc
78
NP-Agent
Verb
NP-Loc
PP-Theme
ANSWERS
ANSWERS
NP-Agent Verb NP-Theme PP-Loc
CORRECT
T NP-Agent Verb NP-Loc PP-Theme
CORRECT
NP-Agent VerbPass NP-Theme PP-Loc AGENTACT
2
NP-Agent * *VerbPass NP-Loc PP-Theme
AGENTACT
NP-Agent Verb NP-Theme *NP-Loc
ALT
NP-Agent Verb NP-Loc "NP-Theme
ALTT
NP-Agent Verb *PP-Theme PP-Loc
ALT
NP-Agent Verb *PP-Loc PP-Theme
NP-Agent Verb "[NP-Theme PP-Loc]
NOEMR
NP-Agent Verb *[NP-Loc PP-Theme]
NOEMR
AP-Agent Verb NP- Nh-Theme *PP-Loc
LEXPREP
NP-Agent Verb NP-Loc *PP-Theme
LEXPREP
*NP- Theme Verb NP-Agent PP-Loc
SSM
*NP- Loc Verb NP-Agent PP-Theme
SSM
*NP-Loc NP-Agent PP-Theme
SSM
*NP-Theme Verb NP-Agent PP-Loc
SSM
*NP- Theme Verb NP-Loc PP-Agent
AASSM
*NP-Loc Verb NP-Theme PP-Agent
AASSM
Figure 3: BLM context and answers for the spray/load alternation from AGENT-LOC-THEME to AGENT- THEME-LOC. CR Corrupted rule(s), locus of the rule corruption, angled brackets syntactic embedding.
Figure 4: BLM context and answers for the spray/load alternation from AGENT- THEME-LOC to AGENT-LOC: THEME. CR Corrupted rule(s), * locus of the rule comuptiio, angled brackets = syntactic embedding.
reflect the argument structure of the active sentence, specifically its transitivity (Hopper and Thompson, 1980). In the spray/load verb class, the THEME and Loc can be syntactically transformed into the sub- ject of a passive structure (e.g., paint was sprayed onto the wall vs. the wall was sprayed with paint, they both share direct object properties (D Elia, 2016, 156-159).
2.3 The contrastive answer set
The target sentence is to be chosen from set of candidates that exhibit minimal differences. The semantic-syntacte mapping of the alternation be decomposed into set of smaller patterns that describe the sentences in the alternation and that can be violated to construct incorrect answers. We list them below: subpatterns 1-3 govern the cor- rect learning the syntactic form of the sentence; subpattern 4 governs the proper lcxical selection and pattern of type governs the proper mapping syntax-semante mapping (which can be expressed in many ways).
The matrix then is structured to provide one of the two alternates the alternation, and provide shared properties (passivization of THEME and LoC) and to interrogate the model to output the other alternate as the missing, target sentence. This structure is illustrated in Figure 2, and represents the BLM for the spray-load alternation in English, referred as BLM-s/IE.
1. Agent position in an active sentence (AGENTACT): If the voice of the sentence is active, the agent is an NP in subject position.
2.2 The matrix templates
2. Preserve to the alternation pattern (ALT): The verb is followed by a NP and a PP.
Each sentence can be described in terms of one
oftributioon-o--hree--llll rule, governing the se- mantic roles (Agent, Theme, Locative), and two
3. No Embedding (NoEmB): The PP following the NP is not embedded in it.
distributionn-o---wovvaad rules governing syntac-
The lexical choice of the preposition (LexPREP): When PPs, arguments are introduced by given prepositions (c.g. onto the wall vs. under the wall).
tic types (nominal phrase NP vs. prepositional phrases PP) and the mood of the verb, whether ac- created two templates, one for each of the two alternates, as shown in Figures& and Figure 3.
tive (Verb) or passive (VerbPass).
5. Syntax-Scmantic Mapping (SSM): the order of the con- stituents and their role is fixed (e.g. a Theme cannot be in an Agent position in active mood).
We call ALT-ATL data, the data produced from the matrix in Figure 3, and ATL-ALT data those produced from the matrix in Figure 4. Examples are discussed in section 3.
'Multiple rules can be corrupted at the same time. For example, AGENTACT and SSM can be both violated at the same time (c.g. The wall sprayed paint with the girl). We label such violation as AASSM.

---

2005) identify a of 30 verbs that undergo the sprayload alternation."
Naturally occurring instances of class verbs
The data generation process begins by select- ing naturally occurring example for each verb the class. To accomplish this, we use the SPIKE platform' (Shlain et al., 2020), which offers a col- lection of English corpora from various text genres (encyclopedic entries, internet reviews, among oth- ers). We favoured extracting natural contexts, and we prioritized examples obtained from the Ama- zon Reviews subcorpus. We performed the search of different inflected forms the verbs and anal- ysed manually the naturally occurring examples in the output. For example, the search for different inflected forms of the verb spritz allowed us to re- trieve sentence like spritz the fish with low-fat oil (Amazon Customer; A19VW07IUHNVIR,
the
Masked augmentation
(the)) (thh)) (KOASS) vantz the (isk with lowe-for ou reotte (ime) (loas))) weth lnn-jorol
rpeetr the pict with (ras) (aoWm))
Instatiation and lexical variants
VERB
AGENT:
THÉME: low-for
type!
type
type III
We removed non-target examples (e.g. those cases in which the PP does not represent an ar- gument of the verb, but it is embedded in the nominal phrase, as in This bulb scatters dim light with green hue around; ID: AMG Enthusiast; AI9JTX5ZX0II), Care was also taken to vary the types of sentences as much possible (e.g. im- peratives, relatives) and pronominal entities (e.g., counter the fact that good portion of Amazon reviews involving these verbs are indeed in first person singular).
Figure 5: Pipeline process. In red, an example of the steps with the verb to spritz is provided.
The structure of the answer set is presented in Figures 3 and 4 for each context. Relevant exam- ples are given in section 3.
3 Creating the dataset
To instantiate the templates, we follow a pipeline process comprising several steps.
The extracted examples were then used in fill-mask task. Every naturally occurring example (seed) had in all six different maskings. Each of the three arguments (AGENT, ThEME and Loc) was masked in two contexts, one aiming at eliciting an indefinite noun (without the article, c.g. I spritz [MASK] with low fat oil) and the other at eliciting a definite noun (preceded by the definite article the, eg. I spritz the [MASK] with low fat oil). We used the DistilBERT uncased model (Sanh et al., 2019), as want to use different language model for data augmentation from the one we will use for learning (section 4) avoid bias. From the proposed items, we manually selected five elements that, when combined, pro- duced grammatically and semantically acceptable sentences (for example, they belonged to similar semantic field as the seed). We also took definite- ness into account. Pronouns were limited to those
Masked augmentation
1. Find relevant verbs in naturally occurring seed sentences.
2. Augment these sentences to create large dataset.
3. Recombine the seed and augmented items to further augment the size of the dataset in controlled way.
Each step is discussed in the following para- graphs. Figure 5 illustrates the workflow.
Natural instances of class verbs Dedicated rich repositories grouping verbs into semantically co- herent classes based on shared syntactic behavior exist (Stowe al., 2021). We retrieved the lex- ical verbs belonging the same class as spray² and the verb load from VERBNET (Kipper Schuler
'In alphabetical onder: baste, brush, drizzle, hang, load, plaster, pump. rub, seatten, seed, sew, shower, smear. smudge, sow, spatter, splash. splatter, spruy, spread, sprinkle, spritz, spurt, squirt, stick, strew, string, swab, swash, wrap. "https: //spik/s..///////////.. apps aops.alless allenai or gagaattssss wwl.eaaiorg/datee...ee.ee....eee
'https: htal:s:://vvb....e colorado adu//vebb-idex//vll spray-9.7 www.pppppppaay....mmmm 9.

---

CONTEXT
of lexical entries of the arguments and verb. Type II and Type III comprise 15,000 contexts/answers sets. Examples of type context and type III are shown in Figures and 7. Remaining examples are shown in the appendix.
The buyer can the tools bags.
The tools loaded by the buyer
The tools were loaded in bags by the buyer
The tools were loaded in bags
Bags were loaded by the buyer
Bags were loaded with the tools by the buyer Bags were loaded with the tools ???
Each of the three subsets of datasets is split 90: 10 into train and test subsets, which are provided with the data. 20% of the train data is used for develop- ment. Section presents the experimental settings.
Figure 6: Example of Type I context sentences.
CONTEXT
pumped the pipes with regular air
Experiment
2. The grain was sown by the monks
3 Regular air was pumped into the stove by me
We present two baselines to investigate the chal- lenge of detecting the underlying patterns of com- plex syntactic-semante mapping in transformer- based sentence embeddings (Hewitt and Manning, 2019; Yi et al., 2022). Our BLMs for the spray/lodd alternation are a case study of this kind of problem. To provide a benchmark for the task, we choose architectures that are simple, but effective three- layer feed-forward neural network, and three- layer convolutional neural network proven by their numerous applications in NLP. They are de- scribed in detail in section 4.2.
4 The contents were stuck in box
5 The suitease was loaded by the buyer
6 The dragon world was scattered with monsters by the wizard
7 The surface was splattered with the stuffing
8 ???
ANSWERS
1 The tomatoes splatter water all the grill
The scientists were seeded sulfur into the earth Farmers have squirt tiny hit oil everything
My child smeared with the flour all over the
People baste different types of marinades of scafood brushed mascara under the surface
The yummy almond butter spread someone over the pastry
The stuff sticks Bob in the truck
The cart can load the supplies in the buyer
4.1 Data
Figure 7: Example of Type IIII context sentences and answer set.
We use the BLM data described above, and 1x768 sentence embeddings generated using RoBERTa (Liu et al.. 2019) and Electra (Clark et al., 2020) pretrained models. For each sentence in a BLM
present in naturally occurring examples.
instance, we use the embedding of the [CLS]/ <>>>>>>>>>>>>> token as its representation.
Two of the authors individually validated each item. A consensus decision was then reached for each item. By the end of this process, then, each of the 30 verbs was related to five AGENTS, five THEME and five LOC. We refer to this group of sentences as the lexical seed ser.
4.2 Baseline systems
We aim detect patterns that encode linguistic information concerning verb alternations in se- quences of sentences. We choose two baseline systems - a feed-forward neural network (FFNN) and a convolutional neural network (CNN). Each has characteristics that will allow it to find pat- terns either more localized (the CNN) more distributed (the FFNN) in the input sequences.
Instantiation and lexical variants The con- stituents belonging to the lexical seed set were merged together build templates and answer sets. We constructed three types of contexts and answer sets, each of increasing lexical difficulty. Type I refers to problems generated directly with seed in- put segments and their variations: the same lexical elements for the verb, AGENT, THEME, and Loc are found for every sentence in each context. The combination of five arguments (125) 30 verbs creates 3750 contexts/answers sets. In Type II the templates are built with the same verb, but the argu- ments vary lexically. Finally, Type III results from a fully random reshuffle of Type II: for each con- text and answer set, each sentence varies in terms
The FFNN receives input as a concatenation of sentence embeddings in a sequence, with a size 7 x 768. This input is then processed through 3 fully connected layers, which progressively compress loyer1 3.5 768 logor?, 3.5
the input size (7 x 768
768 lager3, 768) to obtain the size of a sentence representation. The FINN's interconnected layers
RRoBERTa: anl-roberta-base Flectra: google/electro-
discriiicttn

---

enable it capture patterns that are distributed throughout the entire input vector.
the FFNN perfomed much better, we then per- formed experiments with Electra sentence embed- dings only with the FFNN.
The CNN takes as input an array of embeddings with a size of (7 x 768). This imput undergoes three consecutive layers of 2-dimensional convolutions, where each convolutional layer uses kernel size of (3x3) and a stride of 1, without dilation. The resulting output from the convolutional process is then passed through fully connected layer, which compresses to the size of the sentence representa- tion (768). By using kernel size of (3x3), stride=1, and no dilation, this configuration emphasizes the detection localized patterns within the sentence sequence array.
Figure 13 shows the heatmaps of Fl scores for ATL-ALT (calculated as averages over 5 runs). Heatmaps of ALT-ATL can be found in the ap- pendix. On the left side of the panel are results all data, the right side results where type II and type III training data are sampled to a sample size comparable to type
We did not detect any particular asymmetries in terms of performance between the two groups, suggesting that there is no a favoured alternate. While the results for all configurations are quite high, the most difficult setup is when the model is trained on data with the least amount of lexical variation (type I), and tested in the data with the highest lexical variation (type III), However, when using Electra sentence embeddings, even this set- up has a performance of F1 > 0.80, showing that in Electra sentence embeddings, lexical variation is not deterring factor neither in finding relevant patterss for verb alternations, nor in applying them.
Both networks output a vector representing the sentence embedding of the correct answer. The objective of learning is to maximize the probabil- ity of selecting the correct answer from a set of candidate answers. To achieve this, we employ the max-margin loss function, considering that the incorrect answers in the answer set are intention- ally designed to have minimal differences from the correct answer. This loss function combines the distances between the predicted answer and both the correct and incorrect answers. Initially, calculate score for each candidate answer's em- bedding & in the answer set A with respect to the predicted sentence embedding Cpred- This score is determined by the cosine of the angle between the respective vectors:
Overall, Electra performs better, confirming the results Yi al. (2022).
4.4 Error Analysis
Error analysis can help determine which are the more problematic patters detect in sentence embeddings. We perform this analysis for the more challenging setting: models trained on type III data, and tested on all the others. The corresponding plots are presented in Figures 9 and 10.
score(Cs, Cpred) = cos(ci, Cprea)
The loss function incorporates the max-margin concept, taking into account the difference between the score of the correct answer & and each of the incorrect answers e;:
Errors common to both groups Across mod- els, the highest error is mainly the SSM (e.g.. The wall sprayed the girl with the paint), which repre- sents an argument structure mistake, as the map- ping from semantic roles to syntactic functions is mistaken. This is indeed difficult aspect of the alternation to learn, as the surface syntactic struc- ture is correct, and the incorrect alternatives can be detected only if the underlying lexical semantic rule is mastered (Agent=subject, theme=object), namely, if the model has somehow acquired knowl- edge of semantic macroroles, together with world knowledge about what can be Agent or Theme for the verb spray, which corresponds to semantic microroles.
loss, = - score((( tccoree,,, epea))
During prediction, the answer with the highest score value from the candidate set is selected as the correct 5
4.3 Results
We performed experiments with RoBERTa sen- tence embeddings using both baselines. Because
'All systems used a learning rate of 0.001 and Adam op- timizer, and batch size 100. The training was done for 120 epochs. The experiments were run on an HP PAIR Worksta- tion 24 G4 MT, with435 an Intel Xcon W-2255 processor, 64G RAM, and a MSI GeForce RTX 3090 VENTUS 3X OC 24G GDDR6X GPU.
ALT (e.g., The girl sprayed the wall the paint; the girl sprayed over the wall with the paint) is also a very common error observed when testing

---

CNN-Roberta
FFNN-Roberta
FFNN-Electta
ALL TRAINING DATA
type_ H
8:985
0.891
type_
0.971
0.963
type_
0.996
0.957
Test on
Test
type_ll II
0.759
0.791
0.759
type_ll II
0,795
0.918
0.767
Test
type_
0.869
0.935
0.951
type_ll |I|
0.602
0.603
0.747
0.604
0.613
0.774
type_
0.804
0.812
0.953
type
type
type
type_
type
type
typee
sype
type III
Train
Train on
Train on
SAME TRAINING DATA
type_
0.989
0.972
0.861
type_
0.979
0.992
0.899
type_F
0.987
0.984
0.976
Test on
Test
type_ll
0.751
0.961
0.759
type_ll
0,725
0.977
0.798
18sT
type_1
0,881
0.982
0.932
type
0.537
0.593
0.754
type
0.575
0.649
0.786
type_1l
0.784
0.821
0.934
type
type
type.
type
type_
type II
type
type
type III
Train
Train
Train
Figure 8: Fl results for ATL-ALT (given AGENT-THEME- guess AGENT-LOC-THEMEN
0.200
Roberin, INN
0.175
Roberta_FF FFNN
0.150
Electra_FFMM FFNN
J SSM
0.125
0.100
0.075
0.050
0.000
AgentAct
Alt
NoEmb Type
LexPrep
SSM
AgentAct
NoEmb LexPrep Type
SSM
AgentAct Att
NoEmb LexPrep Type III
Figure Error analysis for ATL-ALT (given AGENT-THEME-LLOC predict AGENT-LOC-TIIEEE For example, given spray the paint onto the wall predict spray the wall with paint, aggregated trainings)
0.200
Roberta,CCm
probabiity
0.150
Bioctra, FFNN
6.125
0.100
0.075
0.025
0.000
NoEmb LexPrep Type
LexPrep
Type
Type III
Figure 10: Error analysis for ALT-ATL (given AGENT-LOC-THEME predict AGENT-THEME-LOC. For example, given spray the wall with paint predict spray the paint onto the wall. aggregated trainings)

---

the model on type III data. This means that the word order of semantic roles of the alternations are learned, but the syntax is not.
of arguments continues pose significant chal- lenges. In particular, we observed that the high- est number of errors is related to the mapping be- tween semantic roles and syntactic functions. We believe that mastering this aspect of the alternation proves to be challenging because the surface syn- tactic structure of incorrect alternatives is mostly correct, and identifying the correct answer requires that the underlying lexical semantic properties be identified in the distributed and continuous sen- tence representations, which the baseline models are not able to do.
AGENTACT mistakes are also frequent (The girl was sprayed the wall with the paint). Here the sentence is ungrammatical, but it is in the passive voice, like the majority of sentences in the context, so it is close the context by structural similarity. This suggests that the underlying rules are not de- composed, but learnt by analogy and similarity in a global space.
NOËMB is less problematic (the girl sprayed the paint for the room). This contrasts with the high number of mistakes in lexical choice of the prepo- sition in some cases. LEX seems problematic for type III of group ATL-ALT (the girl sprayed the paint under the wall). This mistakes shows that the models can detect correct from incorrect sentences -this is afterall a plausible sentence- but it shows that the general pattern of the alternation has not been identified at all.
Related work
The current paper does not have a direct compar- ison, since it is the first proposal of dataset for verb alternation adopting a BLM scheme. It is directly connected to the works on BLM on agree- ment, studies on verb classification and syntactic rules and, partially, on structured datasets adopted in computer vision (Wang and Su, 2015; Zhang et al., 2019).
Errors most prominent in one group There are asymmetries between the two groups regarding spe- cific types of errors. The occurrence of errors re- lated to the LexPrep rule is more prevalent when the target sentence follows the AGENT-LOC-THEMM structure, for example, the girl sprayed the wall under the paint. The limited number of errors of this kind in the other group suggests that the encod- ing of the preposition information for the locative element, when it is the target, is well preserved (e.g. the girl sprayed paint under the wall).
BLM for agreement The proposed structured dataset enriches the typology of linguistic phenom- ena that can be tested with BLMs. Previous studies focused on investigating agreement patterns be- tween the subject and verb in French (An et al., 2023; Merlo et al., 2022). One of the difficulties of the experiments was provided by the increas- ing number of attractors between the subject and the verb, which could potentially interfere with the detection of the agreement relation between them. Another difficulty was that the agreement patterns followed themselves global pattern in the matrix that had to be identified. Two exploratory baselines showed that despite the simplicity of the phenomenon, deep learning system do not manage to identify the underlying rules.
As mentioned earlier, ALT is a highly frequent error observed during testing on type III. Asymme- tries emerge in the nature of these errors, with two distinct types occurring. The first type involves the verb being followed by two noun phrases (the girl sprayed the wall the paint), while the second type involves the verb being followed by two prepo- sitional phrases (the girl sprayed onto the wall with the paint). Although the latter error is more prominent across the three models in both groups, there are distributional differences between the two groups. Moreover, in the ATL-ALT, clear dif- ference in performance can be observed between Roberta and Electra models.
Verb Alternations Previous work has focussed on understanding the automatic learning of verb al- ternations in terms of syntactic and semantic prop- erties of the verbs and their argument structures (allowing, or disallowing verb alternation) (see Ma- jewska and Korhonen 2023 and reference therein). These properties have been explored in relation to their representation in Large Language Models (Kann et al., 2019; Yi al., 2022), encompassing various dimensions of performance across differ- ent models. In particular, Yi et al. (2022) suggest that LLMs with contextual embeddings encode lin-
4.5 Discussion
According to the error analysis, the observed pat- terss of successful generalization predominantly concern syntactic aspects. The semantic mapping

---

guistic information on verb alternation classes, at both the word and sentence levels. Specifically, for the spray/load alternation, Yi et al. (2022) probed whether a verb entered this class by building two binary classifiers. The classifier predicts whether a verb can participate in both syntactic frames of the alternation (frame intended as syntactic envi- ronment, see also Kann et al. (2019)). The binary classification task done based token embed- dings makes the targeted phenomenon explicit, but does not provide insights into how this linguis- tic property of verbs is encoded, or how it interacts with the verb's ' arguments.
Human upper bound We do not have, this stage, comparable results with human behaviour, due to the type and size of the task.
Ethics Statement
To the best of our knowledge, there are no ethics concerns with this paper.
Acknowledgements
We gratefully acknowledge the partial support of this work by the Swiss National Science Foun- dation, through grants HINF40_180880 (NCCR Evolving Language) and SNF Advanced grant TMAG-1_209422 to PM.
6 Conclusion
In this paper, we have introduced BLM-s/IE, a BLM-structured dataset for the English spray/load verb alternation. This is complex case study for the larger problem of predicate-argument structure.
References
Aixiu An, Chunyang Jiang, Maria A. Rodriguez, Vivi Nastase, and Paola Merlo. 2023. M-M-- A French benchmark to investigate generalization of agreement in neural networks. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 1363- 1374, Dubrovnik, Croatia.
The dataset comprises two groups of templates. Each group presents one of the alternants, while the other alternant needs to be inferred hased on the information provided by the context the BLM. The answer set is designed to exhibit minimal su- perficial variations from the correct answer but significant differences in the underlying structure. Through error analysis, we gain better under- standing of the types of information that baseline models manage to learn, and what aspects instead will require specifically developed models and ar- chitectures.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. 2020. FElectra: Pre- training text encoders as discriminators rather than generators. In ICLR, pages 1-18.
Samuel C D'Elia. 2016. The Syntuctic and Semantic Atoms of the Spray/load Alternation. Ph.D. thesis, University of Kent.
Kristina Gulordava, Piotr Bojanowski. Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 1195-1205, New Orleans, Louisiana.
Our findings indicate that, on the one hand, cu- rated and structured datasets, such as the one pre- sented in this study, have the potential to lead to articulated understanding of learning. On the other hand, we conducted experiments using two dif- ferent architectures, and the results highlight the usefulness of the task, whose solution requires de- tecting underlying linguistic properties and rules. Overall, these findings provide further evidence supporting the use of our BLMs as an innovative benchmark to reach better qualitative understand- ing of learning in current neural network models.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4129 4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Limitations
Paul Hopper and Sandra A Thompson. 1980. Tran- sitivity in grammar and discourse. language, pages 251-299.
Additional alternation phenomena As this work serves as an initial blueprint, our decision has been to concentrate solely on single verb al- ternation. However, will be interesting for the future to aim for a broader spectrum of verb alter- nations and in multiple languages.
Katharina Kann, Alex Warstadt, Adina Williams, and Samuel R. Bowman, 2019. Verb argument structure alternations in word and sentence embeddings. In Proceedings of the Society for Computation Lin- guistics (SCiL) 2019, pages 287-297.

---

Karin Kipper Schuler. 2005. VerbNet: A broad- coverage, comprehensive verb lexicon. Ph thesis, University of Pennsylvania.
Meeting of the Association for Computational Lin- guisties (Volume 2: Short Papers), pages 784-789. Melbourne, Australia. Association for Computational Linguisties.
Malgorzata Kisielewska, Mariusz Urbański, and Katarzyna Paluszkiewicz. 2016. Abduction in one in- telligence test. types of reasoning involved in solving raven's 's advanced progressive matrices. In L Magnani and D Casadio, editors, Model-Based Reasoning in Science and Technology: Logical, Epistemological, and Cognitive Issues, pages 419-435.
Malka Rappaport and Beth Levin. 1988. What do with theta-roles. In Wendy Wilkins, editor, Thematic relations, pages 7-36.
John C. Raven. 1938. Standardization of progressive matrices. British Journal of Medical Psychology. 19:137-150.
Beth Levin. 1993. English verb classes and alterna- fions: A preliminary investigation. University of Chicago Press.
Sehastian Ruder. 2021. Challenges and Opportuni- ties in NLP Benchmarking. http://ruder... io/ illp-benchmarking.
Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annual Review of Linguistics, 7(1):195-212.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax- sensitive dependencies. Transactions of the Associa- tion for Computational Linguistics, 4:521-535.
Micah Shlain, Hillel Taub-Tabib, Shoval Sadde, and Yoav Goldberg. 2020. Syntactic search by example. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 17-23, Online, Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 11692.
Kevin Stowe, Jenette Preciado, Kathryn Conger, Su- san Windisch Brown, Ghazaleh Kazeminejad, James Gung, and Martha Palmer. 2021. SemLink 2.0: Chas- ing lexical resources. In Proceedings of the 14th In- ternational Conference on Computational Semantics (IWCS), pages 222-227, Groningen, The Netherlands (online). Association for Computational Linguistics.
Olga Majewska and Anna Korhonen. 2023. Verb classi- fication across languages. Annual Review of Linguis- tics, 91):3133-33.
Paola Merlo 2023a. Blackhird language matrices (BLM), new task for rule-like generalization in neural networks: Can large language models pass the test? In Findings of the 2023 Conference on Empirical Methods in Natural Language Processing.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpost language understand- ing systems. In Advances in Neurul Information Processing Systems, volume 32. Curran Associates, Inc.
Paola Merlo. 2023b. Blackbird language matrices (BLMs), a new task for rule-like generalization in neural networks: Motivations and formal specifica- tions. arXiv cs. CL. 11444.
Ke Wang and Zhendong Su. 2015. Automatic genera- tion of raven's progressive matrices. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 903-909 AAAI Press.
Paola Merlo, Aixiu An, and Maria A. Rodriguez. 2022. Blackbird's 's language matrices (blms): new bench- mark to investigate disentangled generalisation in neural networks.
Paola Merlo, Chunyang Jiang, Giuseppe Samo, and Vivi Nastase. 2023a. Blackbird Language Matrices Tasks for Generalization. In Proceedings of the GenBench workshop.
David Yi, James Bruno, Jiayu Han, Peter Zukerman, and Shane Seinert-TThelkkee 2022. Probing for un- derstanding of English verb classes and alternations in large pre-trained language models. In Proceedings of the Fifth BlackboxNL.P Workshop on Analyzing and Interpreting Newral Networks for NLP, pages 142-152, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguisties.
Paola Merlo, Giuseppe Samo, Vivi Nastase, and Chun- yang Jiang. 2023b. Building structured synthetic datasets: The case of Blackbird Language Matrices (BLMs). In Proceedings of the Ninth Itaiian Confer- ence on Computational Linguistics (Clic-li 2023).
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. RAVEN: A dataset for re- lational and analogical visual reasoning. In IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2019, Long Beach, CA, USA, June 16-20,
Pranav Rajpurkar, Robin Jia, and Percy Liang, 2018. Know what you don't know: Unanswerable ques- tions for SQUAD, In Proceedings of the 56th Annual

---

2019, pages 5317-5327. Computer Vision Founda- tion IEEE.
A
Appendix
A.1 Answer set type
TYPE
EXAMPLE CONTEXT
The buyer load the tools hags.
The tools loaded by the buyer
The tools loaded in bags by the buyer
The were loaded in bags Bags were loaded by the buyer Bags were loaded with the tools by buyer Bags were loaded with the tools ???
EXAMPLE OF ANSWERS
The buyer can load bags with the tools
The buyer was loaded hags with the tools The buyer can load bags the tools
The buyer can load in bags with the tools The buyer can load bags sale The buyer can load bags under the tools Bags can load the buyer with the tools The tools can load the buyer in bags Bags can load the tools in the buyer
Figure Example of Type context sentences and
A.2 Type II: example of context sentences and answer set
TYPE II
EXAMPLE CONTEXT
load bags
food
Clothes were loaded by me
Packages were loaded in the cart by visitors
Packages were loaded bags The cart was loaded by users The box was loaded with clothes by me The cart was loaded with the food ???
EXAMPLE OF ANSWERS
Visitors can load supplies in the box was loaded clothes in bags Visitors can load the food the cart
Visitors can load with the food in bags Users can load the tools for experimentation Users can load the food under bags Supplies load the customer the suitcase
The tools can load the buyer bags The cart load the supplies in the buyer
Figure 12: Example of Type context sentences and answer set.

---

A.3 Detailed F1 scores
CNN-Roberta
FFNN-Roherta ALL TRAINING DATA
FFNN-
400
648
435
Booken
SAME TRAINING DATA
AMI2
SANOKT
CTM4
0411316
Figure 13: F1 results for ATL-ALT group (given AGENT-THEME-LOC predict AGENT-LOC-THEMEL
CNN-Roherta
OFNNrRoberta ALL TRAINING DATA
FFNN-Flectra
280
400
63
05.001
SALDIC
323197
AM0
-44
SAME TRAINING DATA
400
Figure 14: Fl results for ALT-ATL group (given AGENT-LOC-THEME predict AGENT- CHENT-THEME-LLC)


Reviews:
[{"review_id": "qkLr3t8OFn", "reviewer": "Reviewer_fcXT", "summary": "", "questions": "- Footnote 2 links to one sense of the verb _spray_ which is relevant.  Did you also use a sense of the verb _load_ and, if so, which?\n- How exactly did you extract sentences in SPIKE (paragraph starting line 227)?  More details should be provided here or in an Appendix, in the interest of full reproducibility.\n- Figure 7: \"The contents were sticked in a box\".  I think this should be \"were stuck in a box\", i.e. _stick_ has an irregular past participle. I doubt this makes a huge difference to anything, but am curious at which step the passivization occurs.\n- Figures 9 and 10: can you clarify exactly what these plots are showing?  I _think_ it looks at all mistaken choices by the models, and shows which sentence was chosen instead of the correct one.  I think it would also be useful to have a real example mistake for each category; it was hard to determine what \"ALT\" meant at the bottom of the first column on page 6.  Relatedly, the discussion of ALT errors from lines 423 onward confused me a bit: are you saying that the model tends to select sentences of these two types?  The wording almost makes it sound like it's generating those, but I don't think that's the task.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The reviewer asks about data extraction.  We clarify here the extraction of the seed examples from SPIKE. We performed the search of different inflected forms of the verbs and analysed manually the naturally occurring examples in the output. We removed non-target examples (e.g. the PP does not represent an argument of the verb, but it is  embedded in the nominal, as in 'This bulb scatters dim light with a green hue around' (ID: AMG Enthusiast; AI9JTX5ZX0OJG)). Care was also taken to vary the types of sentences as much as possible (e.g. imperatives, relatives) and pronominal entities (e.g., to counter the fact that a good portion of Amazon reviews are in the 1st person singular). In the final version, we will take advantage of the extra page to explain in detail and integrate additional information, such as the enrichment of the discussion on the link of footnote 2.\n\nWe respectfully disagree that we do not have baseline comparisons. This paper concentrates on the description of the data and its benchmarking with basic architectures. We think this work represents a baseline in and of itself, the architectures we propose are standard, simple and do not integrate any novel insights or complex computational mechanisms. We also do not claim any preference for one architecture over another. \n\nIn work described in a different paper under submission, we compare these baseline generic architectures to more tailored and more articulate VAEs and do indeed show an improvement on several BLM problems.\n\nAs for a random baseline on this dataset, the task is multiple-choice, where, by the construction clearly described in the paper, 9 equally frequent options are available in the answer set, then the random baseline is 1 over 9.\n\nAs for comparison to humans, this is indeed planned work but it requires a whole dedicated paper due to the nature of the task. This task is not a natural task that all speakers will perform equally well. This is analogous to an IQ task, Raven's progressive matrices. The output from the psychological literature on RPMs shows that problem-solvers fall into distinct groups, with some achieving high scores and others achieving lower. This means that the establishing human performance needs a correlation with some other measure of abilities and clear control of the constraints in the experimental procedure (size of the dataset, carry-over effects, priming effects from easier tasks). \n\nWe have performed informal tests on BLMs. They show that speakers easily grasp type I, can still see the patterns in type II data, but find type III hard.\n\nAs for prompt-based settings, we are currently exploring prompt-based settings in a different paper. Again, given the nature of the task, a detailed qualitative evaluation is needed to establish if, but mostly how large language models solve the BLM task. Initial results show that, decomposing the solution of this task into the subskills that humans have been argued to deploy, large language models do well in the individuating subtasks (identifying the objects to manipulate and their attributes), but do not do so well in the integrative substasks that allow to eventually solve the task.\n\nAll lexical materials, among which irregular past tense, have been manually reviewed and corrected when needed. We will verify again the validation. Passivization is automatically created during the template constructions, since dedicated lexical entries of the lexical seed set involve the NPs associated with the past tense of the verb 'to be' and the past tense of the verb under investigation.\n\nThe plots in Figure 9 and 10 show the error analysis of our models, in terms of distributions. We added examples in the running text (we acknowledge that the examples for ALT are located relatively late in the previous manuscript, lines 426-431), but we can add them in the caption if needed. In lines 423-onwards, we comment on the fact that ALT represents an error that occurs more in type III than in the other two types, across templates, but it is always based on a multiple-choice task and not based on a generation task.\n\nFinally, the presentation improvements proposed by the reviewer and correction of the typos will be taken into account in the new version of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Application of a new diagnostic task method (Blackbird's language matrices) to a subtle linguistic phenomenon (spray/load alternation) at the syntax-semantics interface.  This seems like a very valuable contribution, showing that we can do linguistically motivated analysis work in a richer setting than normal classification-type tasks.\n- Detailed error analysis of models, showing that e.g. they suffer with selectional restrictions on arguments (animacy).  The main results and this analysis refine earlier probing work on verb alternations.\n- Data will be made publicly available when non-anonymized; I can imagine this being used by other researchers.", "reasons_to_reject": "- Some details of dataset construction were not described in sufficient detail.\n- Lack of baseline comparisons for the BLM task / dataset.  There are at least three things that would be welcome here: What's random chance performance on the task?  How well does a simple baseline (e.g. bag of word2vec vectors) do on the task?  Most crucially, but most difficult: how well do _people_ do on this task?  Just working through the examples, I personally find the task fairly difficult.  It's not obvious to me that we should expect a normal native speaker of English to get 100% on this task; it would be really useful to know how well people do, as well as what kinds of errors they make.  (The authors acknowledge the lack of human comparison in the limitations section.)\n- It would be nice to evaluate some _large_ language models in a prompting-based setting both as a comparison and to show that the BLM method can be used with models of that type.", "questions_to_authors": "- Footnote 2 links to one sense of the verb _spray_ which is relevant.  Did you also use a sense of the verb _load_ and, if so, which?\n- How exactly did you extract sentences in SPIKE (paragraph starting line 227)?  More details should be provided here or in an Appendix, in the interest of full reproducibility.\n- Figure 7: \"The contents were sticked in a box\".  I think this should be \"were stuck in a box\", i.e. _stick_ has an irregular past participle. I doubt this makes a huge difference to anything, but am curious at which step the passivization occurs.\n- Figures 9 and 10: can you clarify exactly what these plots are showing?  I _think_ it looks at all mistaken choices by the models, and shows which sentence was chosen instead of the correct one.  I think it would also be useful to have a real example mistake for each category; it was hard to determine what \"ALT\" meant at the bottom of the first column on page 6.  Relatedly, the discussion of ALT errors from lines 423 onward confused me a bit: are you saying that the model tends to select sentences of these two types?  The wording almost makes it sound like it's generating those, but I don't think that's the task.", "paper_topic_and_main_contributions": "This paper uses a recently introduced sentence pattern-matching task---Blackbird Language Matrices---to do detailed probing of language model understanding of a core syntax-semantics-interface phenomenon: the spray/load alternation.  The task consists of choosing a sentence which correctly continues a pattern (defined by different syntactic and semantic structures of arguments to a verb) from a set of alternatives which do not continue the pattern.\n\nThe paper introduces a rich dataset of instances of this task applied to the spray/load alternation (e.g. that \"She sprayed the wall with the paint\" and \"She sprayed the paint on the wall\" are both grammatical) and then probes RoBERTa and ELECTRA embeddings for their ability to complete the task.  The results suggest that the models are not bad at this task, but that they make a consistent type of error, usually relating more towards the semantics (e.g. not respecting animacy constraints) than to the syntax.  \n\nThe data and results are both interesting and do seem to represent an advance both in what's known about these alternations in particular in language models and also in the methodology of linguistically-informed probing with the new task.  Because the task is so new, there are a few places where the paper could benefit from more detailed exposition and where more baseline comparisons would be helpful for knowing how to interpret the results we do see.  But I do suspect this paper to be of interest to many in the field of interpretability and analysis.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In the paragraph first illustrating the spray/load alternation (line 86 and following, page 2), colors are used for the different semantic roles in the two sentences.  I think I would also similarly color-code the names \"Agent\", \"Theme\", and \"Location\", just to make the parallel especially clear.\n- An example of a BLM from earlier work in Section 2 (i.e. before 2.1) would be welcome, since this is a relatively new task that will be unfamiliar to many readers.\n- Similarly, I think Figures 2 and 3 would benefit a lot from additionally being instantiated with examples of the template, e.g. using the example alternation from the Introduction.  (These examples come up as separate figures 6 and 7, but I wanted to see those earlier.)\n- The open parenthesis on line 154 has no closing parenthesis.\n- Figure 5 has quite low resolution.\n- Line 346: I'm not familiar with the $[ \\cdot ]^+$ notation, and would personally prefer $\\max (0, \\dots)$.\n- Section 4.2: \"Figure 13\" I think should be \"Figure 8\".  Also, this paragraph says \"left handside\" and \"right handside\" for what I think should be \"top row\" and \"bottom row\"."}}, {"review_id": "bxpiTpRYqw", "reviewer": "Reviewer_uWko", "summary": "", "questions": "a) How do you construct the answer set? For some sub-patterns, it seems straightforward, but I’m not sure how to get NoEmb and LexPrep options from the heuristic process.\n\nb) How do you select the 15k examples for Type 2 and 3? Is it by random sampling (as I understand the total possible combinations are much more than that)?\n\nc) How do you get embeddings from RoBERTa or ELECTRA models? Is it from the first token, or mean pooling, or further trained with some recent sentence representation objectives (like SentenceBERT)? Also, do you keep the encoder weights frozen during training, or is it updated with the FFNN/CNN jointly?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We respectfully disagree that the data collection is performed \"heuristically\". We use naturally occurring examples as seeds for the automatic masked augmentation, which is then validated manually by expert linguists, a level of control that we believe is necessary. \nNoEmb and LexPrep represent errors with respect to the lexical preposition introducing the prepositional phrase (PP), but they aim to explore two different rules of the alternation. NoEmb aims to understand whether the PP is an argument and not a sub-phrase of the NP (e.g. the paint for the room). LexPrep aims to understand how much the lexicon, in particular the type of preposition, plays a role in the alternation. From the data creation perspective, LexPrep is generated by editing the PP with a preposition that does not appear in the original seed sentence (e.g. I spray the paint onto the wall; $=$ $\\neg$ onto).  NoEmb follows the same rule but additionally involves a fill-mask task to retrieve lexical materials (e.g. 'the paint for the [MASK]'). All pertinent components are present in the lexical seed set, provided in the supplementary file.\n\nThe type II and type III datasets were indeed randomly sampled.\n\nWe use the embedding of the [CLS] (for BERT) and 
 (for RoBERTa/ Electra) token (which is token 0) from the output of the transformer. There is no additional processing on this token. The reason for using this is that we did not want to operate on something fine-tuned for a specific task, but rather we wanted to investigate the raw transformer output. There is no further tuning for the transformer. We use the pre-trained model to get the sentence embeddings. This is separate from what happens with the baselines (which are trained).\n\nIf the paper is accepted, we will incorporate the reviewer's suggestions to improve the clarity of our presentation (e.g. an early presentation regarding the types of errors), taking advantage of the extra page.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a) The paper is well written and clearly explains the motivation and the complicated procedures to construct the dataset.\n\nb) Comprehensive empirical results that might shed lights on future development of better models.\n", "reasons_to_reject": "Some experimental details are not clearly stated or questionable, see my questions below.\n", "questions_to_authors": "a) How do you construct the answer set? For some sub-patterns, it seems straightforward, but I’m not sure how to get NoEmb and LexPrep options from the heuristic process.\n\nb) How do you select the 15k examples for Type 2 and 3? Is it by random sampling (as I understand the total possible combinations are much more than that)?\n\nc) How do you get embeddings from RoBERTa or ELECTRA models? Is it from the first token, or mean pooling, or further trained with some recent sentence representation objectives (like SentenceBERT)? Also, do you keep the encoder weights frozen during training, or is it updated with the FFNN/CNN jointly?\n", "paper_topic_and_main_contributions": "This paper introduces a new benchmark dataset to help understand how sentence embeddings from pretrained language models capture complex properties of predicate-argument structures. The dataset follows the BLM (Blackbird’s Language Matrices) framework, and focuses on the spray/load verb alternations describing an event where an agent causes motions of a theme to a location, where both the theme and the location can function as the syntactic direct object of the verb. The task is to select one alternate from a set of answer options, given the other alternate and a sequence of context sentences indicating some shared properties of the argument structure.\n\nThe dataset is collected heuristically by first identifying similar verbs to spray and load from VerbNet, along with some example sentences extracted from the web, and then augmented with a masked language model combinatorially. Three types of datasets are constructed with increasing lexical difficulty, varying by whether the predicates and the arguments in the context and the answer set use the same lexical variant. The answer set is designed to exhibit different subpatterns of the argument structure that could be violated.\n \nExperiments with several popular language models like RoBERTa and ELECTRA show that they all perform decently well on all three types of the dataset, achieving over 50% F1 even when trained on the easiest type and tested on the hardest. However, detailed error analysis shows that while the models manage to learn some aspects of the predicate-argument structures, there are other aspects that are still challenging, for example, understanding lexical semantic rules.\n\nMain contributions:\n\na) Constructed a new benchmark dataset focusing on complex predicate-argument structures that can be used to study whether language models has a deep understanding of core syntactic and semantic properties of language.\n\nb) Extensive experiments and detailed error analysis to show what type of knowledge the baseline models are capable of and what challenges still remain.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "a) Section 2.3, it might be helpful to include some examples for each subpattern that could be violated. Otherwise the readers will wait until Figure 7 to get a sense of what the incorrect answers look like, which also makes it hard to associate each answer example with their corresponding structures in Figure 3 and 4.\n\nb) Figure 5 looks very blurry.\n"}}, {"review_id": "KagDJyV84g", "reviewer": "Reviewer_YHa3", "summary": "", "questions": "a) With growing number of available datasets for probing understanding of LLMs, how do the authors foresee extending and maintaining this dataset? If such a test were to be used for benchmarks, how might they be extended for other languages?\n", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The current dataset is one instance of a problem task that can be applied to any linguistic problem in any language. \nIn this respect it is similar to the developments we have seen for example, in corpus annotation, for UD.\n\nThere are many ways in which this approach can be extended. First, a specialised corpus on a given phenomenon in one language, might be explored for the very same phenomenon in another languages, especially for low-resource languages, given that the underlying linguistic phenomenon is the same.\n\nSecondly, templates for different problems can all be collected in a multi-problem dataset (we are currently exploring this testing method in another paper). Also, different templates for different problems can be merged in a single multi-problem template (we are also exploring this approach in a prompt-based setting).\n\nFinally, in comparison to other datasets, we think that the task itself is compelling and controllable, mostly thanks to the carefully crafted answer set which gives rise to controlled experiments.  The answer set supports error analyses whose variables are defined a piori, a proper experimental procedure, while other datasets instead  rely on error analyses that are done a posteriori. Our dataset leads to sound fine-grained investigations of large language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors note the need for new benchmarks to probe language understanding of LLMs. To this end they present a case of alternations of verb argument structure for the English spray/load. Leveraging linguistic insight to examine the 'understanding' of LLMs is crafty. \n", "reasons_to_reject": "The paper would benefit from further proof reading. There are instances where it is a little hard to understand.  \n\nThe authors note that the presented experiments and dataset can be treated as a blueprint for further phenomena. While I see the transferability of the framework utilised, I wonder how diverse the use of a highly specialised corpus can be. This does not necessarily detract from the value of such corpora, but it would help if the authors elaborate on how future benchmarks should be built 'to detect additional layers of comprehension of natural languages'. ", "questions_to_authors": "a) With growing number of available datasets for probing understanding of LLMs, how do the authors foresee extending and maintaining this dataset? If such a test were to be used for benchmarks, how might they be extended for other languages?\n", "paper_topic_and_main_contributions": "This paper presents a new dataset for the spray/load verb alternations in English. The dataset aims to facilitate investigation of LLMs understanding of argument structure of a verb.  The authors also present a qualitative analysis of the errors produced by the 3 models considered. \n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "-Figure 5 is low resolution, resulting in a blurry image. \n-line 389 Under ALT errors, for both clarity and consistency,  it would be great the authors give an example. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper probes sentence embeddings from two pretrained language models, RoBERTa and ELECTRA, to better understand how well these models capture and generalize properties of spray/load verb alternations in English. It uses a sentence pattern-matching task (Blackbird Language Matrices) to do so, and shares the dataset and findings from resulting experiments, which suggest the models are adequate at the task but commit a consistent type of error. All in all, reviewers praised the novelty of the task and the leveraging of linguistic insights to investigate model “understanding”, as well as the detailed qualitative error analysis. There was some concern, however, that experimental and dataset construction details were missing and unclear (e.g. how were the embeddings acquired from RoBERTa and ELECTRA, how were sentences extracted in SPIKE, Figures 2, 3, 9, 10 could use clarifying details, etc.). One reviewer also requested a more detailed exposition of the maintenance and extensibility of this dataset to other phenomena. Lastly, there was some disagreement over the inclusion of baseline comparisons for this task, e.g. random chance, bag of word2vec vectors, human performance. The authors defended their choice to exclude these comparisons, and while I find their rebuttal satisfactory, it would perhaps be beneficial to include the points raised in the rebuttal in the revised paper.

In summary, quite a few minor revisions are needed to address all reviewers’ comments and concerns, which would benefit the Soundness (two 3s, one 4) and clarity of the paper. Barring one reviewer, the Excitement scores were quite high (two 4s, one 2); I would advise the authors to consider the concerns of that reviewer though, i.e. the maintenance and extensibility of the dataset. Given how novel and specific the task is, understanding how it contributes to the wider field of language model analysis and interpretability would raise the impact and interest of this paper.
Decision: Accept-Findings