How Well Do Text Embedding Models Understand Syntax?
Authors: Yan Zhang
Zhaopeng Feng
Zhiyang Teng
Zuozhu Liu
Haizhou Li
Venue: EMNLP 2023 Findings (2023)
Keywords: sentence embedding
compositional understanding

Abstract: Text embedding models have significantly contributed to advancements in natural language processing by adeptly capturing semantic properties of textual data. However, the ability of these models to generalize across a wide range of syntactic contexts remains under-explored. In this paper, we first develop an evaluation set, named SR,  to scrutinize the capability for syntax understanding of text embedding models from two crucial syntactic aspects: Structural heuristics, and Relational understanding among concepts, as revealed by the performance gaps in previous studies. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges, and such ineffectiveness becomes even more apparent when evaluated against existing benchmark datasets. Furthermore, we conduct rigorous analysis to unearth factors that lead to such limitations and examine why previous evaluations fail to detect such ineffectiveness. Lastly, we propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios. This study serves to highlight the hurdles associated with syntactic generalization and provides pragmatic guidance for boosting model performance across varied syntactic contexts.

How Well Do Text Embedding Models Understand Syntax?
Yan Zhang -1
Zhaopeng Feng*? Zhiyang Teng 3 Zuozhu Liu Haizhou Li
National University of Singapore "Zhejiang University NNnyynng Technological University "Angelalign Inc., China 'The Chinese University of Hong Kong, Shenzhen, China
chihyangtengggmail..cc 1 @ intl.zju.edu.cn (haizhou. eepelli,eeleeeee eleyanz Genus, edu. sg
Abstract
SBERT 98%
Text embedding models have significantly con- tributed to advancements in natural language processing by adeptly capturing semantic prop- erties of textual data. However, the ability of these models to generalize across wide range of syntactic contexts remains under-explored. In this paper, first develop evaluation set, named SR, to scrutinize the capability for syn- tax understanding of text embedding models from two crucial syntactic aspects: Structural heuristics, and Relational understanding among concepts, revealed by the performance gaps in previous studies. Our findings reveal that existing text embedding models have not suffi- ciently addressed these syntactic understanding challenges, and such ineffectiveness becomes even more apparent when evaluated against ex- isting benchmark datasets. Furthermore, conduct rigorous analysis to unearth factors that lead such limitations and examine why previous evaluations fail to detect such inef- fectiveness. Lastly, we propose strategies to augment the generalization ability of text bedding models in diverse syntactic scenarios. This study serves to highlight the hurdles as- sociated with syntactic generalization and pro- vides pragmatic guidance for boosting model performance across varied syntactic contexts.
chase mice.
SimCSE
94%
Figure 1: A probing example by using SBERT (all- MiniL.M-L.--y and SimCSE (Roberta-Large) com- pute the semantic similarity between "Cats chase mice" and "Mice chase cats". Both models exihibit high simi- larity socres at this simple task.
However, as we delve deeper into different hier- archies of language comprehension, the question that naturally arises is: How well do these text em- bedding models understand syntax? For example, could the state-of-the text embedding models understand and distinguish the difference between "Cats chase mice" and "Mice chase cats"? Syntax, constituted by set of rules that define sentence structures, forms pivotal aspect of natural lan- guage. It integrates both heuristics and composi- tional elements, establishing the bedrock for the expansive and intricate nature of human language (Manning and Schütze, 1999; Jurafsky and Martin, 2000). thorough comprehension of syntax es- sential for a text embedding model to effectively ascertain the relationships among words, thereby facilitating a level of language understanding that mirrors human cognitive processes. Moreover, text embedding models are increasingly deployed in LLM- agents and real-world applications, ensuring that they maintain a solid understanding of syntax critical to guarantee their reliability and efficacy.
1 Introduction
Text embedding plays significant role in a va- riety of natural language processing applications including language understanding (Du et al., 2020), information retrieval (Thakur et al., 2021) and ques- tion answering (Ni et al., 2021). Recently, many methods (Reimers and Gurevych, 2019; Gao et al., 2021; Ni et al., 2021; Neelakantan et al., 2022; Su et al., 2023), targeting at converting textual data into vector representations, have demonstrated remarkable performance across massive text em- bedding benchmarks (Muennighoff et al., 2023).
In this study, we address this question by in- troducing new evaluation set called SR desig- nated to probe the ability of text embedding mod- cls from three syntactic aspects (Partee, 1995; Gib- son, 1998: Gildea and Jurafsky, 2000; Mitchell and Lapata, 2010; McCoy et al., 2019; Linzen and Baroni, 2020): 1) Structural heuristics: the rules and patterns that govern sentence structures, and 2) Relational understanding among concepts: the
Equally Contributed. Corresponding author.

---

models' capability infer relationships between different concepts in text. These dimensions repre- sent the multifaceted nature of syntax.
We adopt the STS benchmark (Cer et al., 2017). which comprises a selection of STS tasks organized the context of SemEval between 2012 and 2017. The dataset include 1,379 sentence pairs from image captions, news headlines and user fo- rums. They provide range of sentences with vary- ing complexity and structures, making them an ideal starting point for our SR benchmark.
STS:
We take this inquiry a step further by conducting an incisive analysis to uncover the underpinnings of these limitations. By examining various mod- els and their responses to syntactic challenges, delineate the factors that contribute the mani- festation of these limitations and the reasons why they have eluded detection in conventional bench- marks. Recognizing these challenges is a precursor to addressing them. Finally, we show that simply augmenting the training Data with SR-like exam- ples, which can be generated through ChatGPT with designed prompts, can significantly enhance the generalization capabilities of text embedding models in syntactically diverse settings.
SICK: SICK(Sentences Involving Composi- tional Knowledge) (Marelli et al., 2014) includes 9927 sentence pairs that are rich in the lexical and syntactic phenomena. They provide range of sen- tences with diverse compositional structure for our SR benchmark.
CQADupStack: CQADupStack (Hoogeveen et al., 2015) is a dataset derived from Stack Ex- change and is geared towards the identification of duplicate questions in community Question- Answering forums. It brings in diversity from real- world queries and their paraphrased versions.
This research aimed at shedding light on the challenges of syntactic generalization text em- bedding models. By establishing a more rigorous evaluation set and proposing strategies for enhance- ment, we contribute to the advancement of text embedding models that are capable of nuanced understanding and high performance across var- ied syntactic contexts. Our work provides valu- able guidance and sets the stage for future research aimed at achieving more syntactically aware and robust text embedding models. SR benchmark and code are released
The Twitter dataset (Xu et al., 2015) consists of pairs of tweets together with a crowd- annotated score if the pair is a paraphrase. Includ- ing this dataset allows the SR benchmark to ac- count for the informal and concise nature of social media text, which often involves unique linguistic constructs.
Twitter:
BIOSSES: BIOSSES (Sogancioglu et al., 2017) is dataset designed evaluate biomedical seman- tic similarity with 100 testing pairs. By includ- ing BIOSSES, the SR benchmark encompasses the domain -pecifc language used in biomedical texts.
2 SR Benchmark
Typical evaluation sets have often been inadequate in rigorously assessing model's understanding of syntax. They tend to focus high-level perfor- mance metrics, while overlooking the finer aspects of syntactic understanding. SR was designed ad- dress the limitations and specifically evaluate text embedding models from two important syntactic aspects: Structural heuristics, and Relational un- derstanding. These dimensions were chosen due to their fundamental roles in natural language un- derstanding. In this section, we introduce the con- struction process of the SR benchmark.
AskUbuntu: AskUbuntu (Lei et al., 2015) is collection of user posts from the technical fo- rum AskUbuntu. The inclusion of AskUbuntu al- lows for the SR benchmark to encompass complex and technical questions typically encountered in community driven platforms.
By utilizing these datasets, the SR henchmark is well-equipped to evaluate text embedding models' syntactic understanding across varying domains and compositional structures. More important, the recent text embedding models have already demon- strated remarkable performance on these bench- marks. This configuration not only ensures holis- tic evaluation but also effectively pinpoints the models' strengths and weaknesses in diverse set- tings.
2.1 Foundation Corpus for SR Construction
For constructing the SR benchmark, it was vital to base it on rich and diverse foundational corpus that encompass different domains and compositional structures. These corpus include:
'RR Benchmark, code, prompts, and other details are pub- licty available at https://ginnhh.om//7z04444444

---

Structural Heuristies
Innput:
Here are some examples of turning the sentence Into an Inverted one.
Structural Houristics
S1: The horse ealing grees
CQADupStack
Changat: I: Eating the grass, the harse is.
performing on stage.
Original Sentence: (orignne= Sentence) Change:
Performing on stage,
Twitter
BIOSSES
Output: <hnswer>
ChargPT
Relational Understanding of changing the Here subject relation sentence
AskUBuuntu
Input:
Understanding
Original Sentence
1:: The horse cating the grass.
Change 1: The ealing the horse.
band is performing on stage.
band
Original Sentence: (Original Santence)
stage performing
Change:
Output:
ChaigPT
b)
Semandic Toxtual Similantty
Input: scoring.
similarity scoring
standard
Structural Heuristics
The two sentences completely equivefent, they mean the thing. Scoring
performing stage.
g. Sentence ilsell in the water besin.
is bathing the sink. Sentence 2: Birdie washing
Performing
sentences completely dissimilar. Scoring e.g. Sentemce 1: The black dog is running through the snow. Sentence 2: race car driver is driving his through the mud.
Understanding
SR
score similanity
performing stage.
Senience 1: (Original Sentence)
(Original Variation)
performing
Scoring:
Output:
<Score>
Figure Workflow of SR benchmark construction. The workflow begins with the selection of foundation datasets, followed by the generation of data probing structural heuristics and relational understanding among concepts. Subsequently, the generated data is annotated with score to represent semantic similarity of sentence pairs. Finally, the assembled SR benchmark is utilized evaluate the syntactic understanding capabilities of text embedding models.
2.2 Assessing Structural Heuristics
ognize that these sentences convey the same action but with different focus and structure.
To assess how well text embedding models compre- hend structural heuristics, we focus on the transfor- mation of sentences into different syntactic struc- tures (Gibson, 1998). In particular, we emphasize the exchange between active and passive voice as well as creating inverted sentences or partially in- verted sentences.
Inverted and Partially Inverted Sentences: In- verted sentences involve reversing the canonical subject-verb-object order, often for emphasis stylistic purposes. Partial inversion involves only a segment of the sentence being inverted. Assessing text embedding models on their ability to under- stand these structural changes critical for gauging how well they can adapt to varied syntactic con- structs. For instance, standard sentence like "The team has the championship." can be converted into an inverted sentence, such "Won the cham- pionship has the team.". while partially inverted example might be, "The championship. the team has won.". The models should be able to under- stand that, despite the alteration in structure, the
Active and Passive Voice Exchange: In the ac- tive voice, the subject of the sentence performs the action, whereas in the passive voice, the subject is acted upon. We transform sentences from the foundation dataset into their passive or active coun- terparts. For instance, an active sentence such as "The chef cooked the meal" can be transformed into its passive equivalent, "The meal was cooked by the chef." " Text embedding models should rec-

---

core information remains the same.
CQADepStuck
Original Samples
Moditfed Sumples
Medification Rate
Sinctural Houristis
400
2.55
2.3 Assessing Relational Understanding
Rebtional Undestondinng
Table 3: Statistical breakdown of the revision rate for modified sentences in CQADupStack.
The arrangement and relationship of concepts within sentence contribute significantly to the meaning and interprctation of the text. Changing the order or relationship of these concepts may radically alter the sentence's 's meaning.
2.4 Annotation and Validation
As presented in Figure 2, we first utilize ChatGPT to generate sentence pairs that probes the above three syntactic dimensions with specific designed prompts. Afterwards, ChatGPT is also utilized annotate the sentence pairs with semantic similarity scores. Through process of duplication and filter- ing of sentences that are too short, we manage to assemble a dataset comprising 9, 9,424 sentence pairs for each syntactic dimension in the SR benchmark.
Concept Order Manipulation: Concept order within sentences often plays a critical role in con- veying meaning. We manipulate the order of con- cepts in sentences from the foundation dataset to create new sentences. The objective is to exam- ine whether text embedding models can recognize how these manipulations impact the meaning of sentences. For example, consider the sentence: "Tom likes Taylor Swift.". By altering the order the concepts, we get: "Taylor Swift likes Tom.". Though structurally similar, these sentences have completely different meanings. The models should recognize the shift in the subject and object and the corresponding change in the meaning.
To verify the generation and annotation quality, first conduct additional validation by randomly sampling 800 sentence pairs from the SR bench- mark. Any sentences that are inaccurately gener- ated will be corrected as needed. Any sentences that are inaccurately generated will be modified as needed. Table 1, Table 2, and Table 3 present the statistical breakdown of the revision rate for these corrected sentences. Moreover, ChatGPT to assign similarity scores to the standard STS-B evaluation set, and we compute the correla- tion between these scores and the STS-B standard annotated scores.
Conceptual Relationships Perturbations: Fur- thermore, we evaluate the models' ability to under- stand various relationships among concepts, such cause-effect, part- phole, and synonymy etc. For instance, in sentence like "Rain causes floods.". the model should identify the cause-effect relation- ship between "rain" and "floods". As another ex- ample, consider the sentence "He read the book because he was interested in history.". By chang- ing the order, e.g., "Because he was interested in history, he read the book.", the meaning remains the same, but the structure has changed. The model should be able recognize the constancy in the meaning despite the alteration in syntax.
utilize
Upon completion of these exercises, we find that the revision rate is no more than 3% and the cor- relation scores between ChatGPT annotations and human annotations stand at 83.8%. This indicates that ChatGPT is proficient in generating syntax var- ied sentences and annotations that closely resonate with human evaluations, thereby affirming the reli- ability and authenticity of the SR benchmark.
Original Samples
Modiried Sumples
Medification Rate
Evaluating Test Embedding Models on SR
Strictural Houristis
400
0.758
Rebiional Uncbmtonling
400
100%
Table I: Statistical breakdown of the revision rate for modified sentences in STS.
In this section, we employ the SR Benchmark to evaluate the syntactic understanding capabilities of five state-of-the-ar text embedding models, includ- ing SentenceBERT (Reimers and Gurevych, 2019), SimCSE (Gao et al., 2021), Sentence-T5 (Ni et al., 2021), One embedder (Su et al.. 2023) and OpenAl Embedding (Neelakantan al., 2022). Detailed information on these models can be found in their original papers. Following previous works, we employ Spearman's correlation as the evaluation
Twiter Moditedd Samples
Otiginal Samples
Modificttion Kate
Structural Houristes
1.25%
Understading
Table 2: Statistical breakdown of the revision rate for modified sentences in Twitter.

---

Model
STS
SICK
AskUbuntu
CQADupStack
Twitter
BIOSSES
Avg
Structural Heuristics
SimCSE- BEE--BRRT-RRR sus
19.14
21.82
35.38
14.17
19.68
37.27
SimCSF-BERT- Lrggos
19.93
23.52
33.27
15,60
18.59
35.13
24.34
SinCSEE-RBERTa sup
22.97
26.88
37.87
19.20
15.88
35.42
26.37
PitiRT-all-MiiilM---...
13.08
14.83
36.36
11,69
116.9
T3.17
T8.05
SBEERT-all-mpnet--use--
18.35
24.62
37.62
24.39
13.14
Sentence-TS
15.60
21.55
35.34
23.31
20.80
35.45
25.34
Sentence-TS-Laage
17.03
22.83
36,20
24.30
21.83
35.90
26.35
Sentence-TS- TS-XL
17.86
23.25
37.02
26.93
21.24
35.79
27.01
Instructor- Base
17.48
18.73
33.94
21.06
15.95
23.31
21.75
Instructor-Large
20.96
26.35
36.36
22.09
19.34
29.79
25.81
Instructor-XL.
20.46
18,32
34.89
19.97
20.28
26.55
23.41
OpenAT (iext-embeeddin-aad--000)
15.35
21.55
36.25
26.50
17.90
22.40
23.32
Relational Understanding
BInCSE-BEKK-BBse--ee
30.39
38.82
44.87
30.15
38.07
41.79
37.35
SimCSF-BER'' Large-sup
38.8
43.48
44.13
30.64
40.09
48.80
41.16
BAnCSE-ROBERT: Lup
48.34
48.16
46.51
31.28
40.22
50.43
44.16
ABERT-all-MiniL---LL
-2.72
18.30
47.53
25.14
25.70
11.69
20.94
SBEIR--lll-pnnet-uss--Y
30.39
40.42
50.54
30.29
26.09
0.26
29.67
Sentence- -rreencc-T--Baee
33.61
39.36
52.59
35.72
47.63
Sontence-TS-Large
$1.08
50.20
51.09
39.85
44.80
53.82
Sentence-TS. -T-Ln
58.64
$3.34
55.12
42.10
53.41
Instructor- Base
1.35
28.06
48.64
31.46
36.75
18.68
27.49
Instructor-Lane Large
21.88
40.88
49.41
38.50
40.71
41.74
38.84
Instructor-XL XL
28.97
37.67
48.94
39.30
42.34
35.24
38.74
OpenAl fext-emmbdddin--aaa-0002
19.10
33.20
48.70
24.60
33.20
29.90
31,45
Table 4: Results of five text embedding models on the SR Benchmark. Spearman's correlation is reported.
metric assess how well the relationship between the cosine similarities of the sentence pairs and the annotated scores.
provide valuable syntactic understanding that isn't captured through mere parameter scale or breadth of training data. Sentence-T5, which is trained on large scale of web-based quvestion-answeringg datasets, demonstrates most robust performance. This might suggest that the diversity and complex- ity found in web-based question-answering data could be beneficial for embedding models in cap- turing syntactic aspects. We hypothesize that the web-based guestion-answeriin data, often encom- passing wide range of topics, styles, and struc- tures, are likely to present a richer and more varied set of syntactic compositions compared to other data types.
We present the evaluation results on the SR benchmark in Table 4. Though these models have previously exhibited remarkable performance on foundational test datasets, they all perform poorly with low correlations on the SR benchmark. This suggests that existing text embedding models have not been optimized sufficiently address the syn- tactic understanding challenge.
Interestingly, observe that, despite be- ing trained solely on natural language inference datasets, SimCSE outperforms SBERT, Instructor and OpenAl embedding models. These competing models often have more diverse supervised pair- wise training datasets or a greater number of pa- rameters, or both. For instance, SimCSE achieves performance with an average Spearman's 's correla- tion of 44.16% on the benchmark for assessing relational understanding, while SBERT, Instruc- for and OpenAl embedding models achieve av- erage Spearman's correlation of 29.67%, 38.74% and 31.45%, respectively. This could indicate that training on natural language inference tasks may
We use tree kernel (Collins and Duffy, 2002; Yu and Sun, 2022) as a measure of sentence structure diversity. The central idea of tree kernel is to count the number of common subtrees between two con- stituency
We compare the syntactic diversity of two corpora: WebQA (233k sentences selected) and NLI (275k sentences selected). We Stan- fordCoreNLP to obtain constituency parse trees of sentences. Then, we randomly select 1,000 parse trees and use the tree kernel to calculate their simi- larity. The results are shown in Table 5. The lower

---

Corpus
Tree Kernel Similority
Sentence
WebQA
0.064
watering flower.
NLI
0,072
pruning
Table 5: Comparison of tree kernel similarity between WebQA and NLI corpora.
Order Parturbation
Relational Parturtation
flower watering.
flower is
man is pruning flower.
A man is pruning flower.
the score, the more diverse the sentence structures are. As we can sec, sentences in WebQA have lower tree kernel similarity than those NLI, in- dicating that WebQA has more diverse sentence patterns and structures. This diversity could expose the model to a more comprehensive spectrum of syntactic elements, instilling a deeper sense of syn- tax into the embedding models. Since prevailing text embedding models like SBERT, SimCSE, etc. are mainly trained on NLI, we argue that Sentence- T5, which is trained on WebQA, can capture more syntactic nuances of sentences. Additionally, we found that these models generally struggle in the CQADupStack domain but show relatively better performance in the Ask domain. This dis- crepancy may be due to the specific challenges and complexities associated with the CQADupStack domain compared to the UbkUbuntu domain.
Results Jefore and Alter Perturbetion
romurtedion
afher weber cesl iimittation
60-
Crdkr
Figure Retrain SBERT under syntactic perturbations such as randomized word order and exchanged relation- ships among concepts. Despite sigmificant alterations in syntactic structure, the models continue exhibit high similarity scores, highlighting their reliance on semantic content over syntactic understanding.
Moreover, a noteworthy trend across all the text embedding models we assessed was the consis- tent pattern of comparatively better performance on the benchmark for relational understanding, as op- posed to benchmarks for structural heuristics. This suggests that the current text embedding models may be more adept capturing relations among concepts in sentences than they are at grappling with the finer points syntax, such as sentence structures. This superior performance in captur- ing relations could be a result of the training data, which often tends to focus semantic relation- ships. However, it might also indicate that the mod- els have an innate aptitude for discerning semantic relations compared understanding complex syntactic structures.
tactic information is not properly encoded. The rich semantic information contained in the large- scale corpora on which these models are trained allows them to make reasonably accurate predic- tions about the similarity between sentence pairs based shared content words, irrespective of their syntactic structure.
Traditional evaluation paradigms in semantic matching tasks often lack sensitivity to syntactic nuances. Specifically, they tend to reward models for correctly identifying semantic content overlap, but do not penalize them adequately for failing to recognize distinct syntactic constructions. As such, model might receive high similarity score for two sentences that share similar content but have different syntactic structures, thus effectively ignor- ing syntax.
4 Analysis
To empirically demonstrate that models can achieve high performance on semantic embedding matching tasks without proper syntactic under- standing, we conducted an experiment where we manipulated the input data to the text embedding models to remove or alter syntactic information while preserving semantic content. The models' performance was then evaluated in terms of their ability to accurately measure semantic similarity the manipulated data.
4.1 Shortcomings of Traditional Evaluation Paradigms
It is noteworthy that many text embedding mod- els, despite exhibiting poor syntactic understand- ing, have consistently shown high performance on semantic embedding matching tasks. primary reason for this that these models tend to cap- ture semantic content effectively, even when syn-

---

those in the SR benchmark, which is specifically designed to probe syntactic understanding. The pro- cess involves creating additional training examples that emulate the two facets of the SR benchmark: structural heuristics and relational understanding among concepts. Such new sentences can be syn- thesized by altering the original sentences through ChatGPT with designed prompts. Similar to Sec- tion 4.1, we fix sentence 1, do syntactic changes on each sentence and ask ChaiGPT(gpt-3.55 0301) to score the semantic similarity between the fixed sentence 1 and its perturbed one based on similarity scores with explanations and English examples from (Cer et al., 2017). We manually verified the generation reliability and annotation rationality. Each sentence pair can be written [S1, Perturbed S1, re-score].
Results Before and After Augmentation
Augmentation After Augmentation
80-
Speamants cornlation
60
33.9
24.9
19.6
Original
STS
Figure 4: Enhancing syntactic proficiency of SBERT through strategic data augmentation.
As presented in Figure 3, we created two variants of perturbations on the STS dataset one with ran- domized word order, and another with exchanged relationships among concepts - both of which sig- nificantly transform the syntactic architecture of the sentences. STS training dataset consists of sentence 1, sentence 2, and its semantic similarity score is marked by humans. We denote it as [S1, S2, score]. In this section, we fixed sentence 1 and did the perturbation on sentence 2. Each sentence pair is changed into [S1, Perturbed S2, score]. Re- markably, after retraining SBERT with perturbed input while keeping the original annotations unal- tered, we discerned that the text embedding mod- els continued to yield high similarity scores on the standard STS evaluation set. To illustrate, the corre- lation score following relational perturbation stood at 85.2%, nearly on par with the original score of 86.2%.
For our experiment, we adopt SBERT as the base model and conducted experiments by enhancing the STS training dataset with 10,000 SR-like ex- amples for each of the syntactic dimensions (struc- tural heuristics and relational understanding among concepts). We then retrained SBERT on the aug- mented dataset. We choose microsoff/mppet base as our raw model which is also the basic model of the best SBERT text embedding model SBERT-all- mpnet-base-v2. We follow Sentence-BERT train- ing settings (Reimers and Gurevych, 2019), use the regression objective function, a batch-size of 16, 4 training epochs, Adam optimizer with learning rate 2e-5, and linear learning rate warm-up over 10% of the training data. Our default pooling strategy is MEAN. We save the best parameters according the dev set the end of each epoch. At evaluation time, we compute the aosine-similarity between the sentence embeddings.
This experiment underscores the models' pro- clivity to latch onto semantic content, often over- looking syntactic structures when adjudicating the similarity between sentences. These findings prompt more rigorous and discerning evaluation paradigm that factors in both semantic and syntac- tic elements, paving the way for more holistic and accurate assessments of text embedding models.
Figure 4 presented the results. We can observe that the incorporation of the augmented training examples led to substantial improvement in the performance
the SR evaluation set, especially in the aspect of relational understanding. Specifi- cally, the Spearman's rank correlation coefficient in relational understanding witnessed a remarkable increase from 24.9% to 60.1%. This indicates that the augmented data effectively enabled the model to develop a better grasp of the syntactic relation- ships between concepts.
4.2 Simple Solution: Enhancing Syntactic Understanding through Targeted Data Augmentation
In light of the observation that contemporary text embedding models tend to exhibit weak syntac- tie understanding, one simple approach entails en- riching the training data with examples that mirror
However, while the performance improvement on the SR evaluation set is significant, it is also im- portant to observe how this augmentation impacts the performance on the original STS test set (STS-

---

B). We noticed slight decline in the performance on the STS-B test set, with the score decreasing from 86.7% to 84.4% after data augmentation. This slight decrease suggests that while the model be- came more proficient in understanding complex syntactic structures, it may have marginally com- promised on some aspects it had initially learned.
(Dyer et al., 2016; Linzen et al., 2016; Hupkes and Zuidema, 2017; Conneau al., 2018; Lin et al., 2019; McCoy et al., 2019; Shen et al., 2020; New- man et al., 2021). For example, Conneau et al. (2018) assess the syntactic generalization of mod- em sentence encoders through series of selected downstream tasks. Their approach is more geared towards understanding what properties are encoded in the sentence embeddings and how these embed- dings can be used for various NLP tasks. On the other hand, McCoy et al. (2019) developed the HANS dataset to specifically examine the perfor- mance of Natural Language Inference (NLI) mod- els, particularly focusing on whether these models are adopting superficial syntactic heuristics over a deeper semantic understanding. Their work is espe- cially relevant in the context of NLI tasks and aims to discern the methods that models use to arrive at decisions.
This experiment demonstrates the feasibility and potential effectiveness of targeted data augmenta- tion as a mcans to enhance the syntactic understand- ing of text embedding models. It also emphasizes the importance of careful data curation and bal- anced training to ensure that improvements in one area do not come at the expense of performance in another. Future work can focus refining this approach to optimize the trade-off's and achieve im- proved performance across different dimensions of syntactic understanding.
5 Related Work
In contrast, our work aims to construct a bench- mark for directly evaluating the syntactic capabili- ties of text embedding models, without being con- fined to specific task like NLI. The primary goal is to facilitate the selection of robust and effective text embedding models for training in various appli- cations by providing direct measure of their syn- tactic understanding. This benchmark thus serves as an essential tool for researchers and practitioners who are looking to employ text embedding models that are both syntactically sound and effective in real-world applications.
Text Embeddings Text representation learning is fundamental task in natural language processing. In recent literature, contrastive learning (Hjelm al., 2019; He al., 2020; Chen et al., 2020) has emerged the dominant paradigm for train- ing text embedding (Carlsson et al., 2020; Zhang et al., 2020; Gao et al., 2021; Yan et al., 2021; Ni et al., 2021; Chuang et al., 2022; Neelakantan et al., 2022; Su et al., 2023). These approachese typically involves the use of large pairwise pre- training datasets with rich compositional structures, wherein the model is optimized to distinguish be- tween similar and dissimilar text samples. Con- trastive pretraining is geared towards optimizing text embedding matching. As a result, most text embedding models (Reimers and Gurevych, 2019; Gao et al., 2021; Ni et al., 2021) are evaluated on the basis of their performance in semantic similar- ity matching tasks (Cer et al., 2017; Muennighoff al., 2023). However, these tasks not sufficiently encompass the complexity of syntax in natural lan- guage, which involve not only the arrangement of words but also their relationships and composi- tional semanties.
Conclusion
This paper highlights the shorteomings of current text embedding models in understanding syntax. We introduced the SR benchmark analyze mod- els across two syntactic dimensions: structural heuristics and relational understanding. Our find- ings reveal that while these models are adept at semantic tasks, they struggle with syntax. We pro- posed a data augmentation technique using exam- ples tailored for syntactic understanding, leading to notable performance gains on the SR benchmark. However, a slight performance dip was observed on the original test set. Future work could refine augmentation strategies to balance syntactic and se- mantic learning and create more holistic evaluation benchmarks.
Syntax Probing in NLP Syntactic Probing sceks to understand to what extent the learned rep- resentations in NLP models capture syntactic infor- mation and how such information can be effectively extracted and analyzed. There have been numerous studies focused on this topic, each offering unique insights and methods for syntactic generalization.

---

7 Limitation
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaud- hary, Onur Çelebi, Michael Auli, Ves Stoyanov, and Alexis Conneau. 2020. Self-training improves pre- training for natural language understanding. In North American Chapter of the Association for Compula- tional Linguistics.
One limitation is that the SR benchmark may not comprchensively cover all the intricacies of natural language syntax. Real-world text data can be vastly more complex and varied. Also, the enhancement in syntactic understanding was primarily based on simple data augmentations, while augmentation strategy is demand to ensure that enhancing syn- tactic understanding does not come at the expense of semantic comprehension.
Chris Dyer. Adhiguna Kuncoro, Miguel Ballesteros. and Noah A. Smith. 2016. Recurrent neural network grammars. In North American Chapter of the Associ- ation for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Lan- guage Processing (EMNLP).
A Acknowledgements
We would like to thank all the reviewers for their constructive comments. This research is supported by the Agency for Science, Technology and Re- search (A*STAR) under its AME Programmatic Funding Scheme (Project No. A18A2b0046); Zhiyang Teng is partially supported by CAAI- Huawei MindSpore Open Fund (CAAIXSILJJ- 2021-046A).
Edward Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition, 68:1-76.
Daniel Gildea and Dan Jurafsky. 2000. Automatic la- beling of semantic roles. In Annual Meeting of the Association for Computational Linguistics.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proc. of CVPR.
References
R.
Devon Hjelm. A. Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio, 2019. Learning deep representations by mutual information estimation and maximization. In Proc. of ICLR.
Fredrik Carlsson, Amaru Cuba Gyllensten, Evan- gelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2020. Semantic re-tuning with contrastive tension. In International Conference on Learning Representations.
Doris Hoogeveen, Karin M Verspoor, and Timothy Bald- win. 2015. Cqadupstack: A benchmark data set for community qpestion-answering research. In Proceed- ings of the 20th Australasian document computing symposium, pages 1-8.
Daniel Cer, Mona Diab, Eneko Aginre, Iñigo Lopez-
Gazpio, and Luciaa Specia. 2017. Semeval-2017 task Semantic textual similarity multilingual and cross-lingual focused evaluation. In Proc. of Se- mEval@A ACL.
Dieuwke Hupkes and Willem H. Zuidema. 2017. Vi- sualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hier- archical structure. Artif. Intell. Res., 61:907-926.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proc. of ICML.
Dan Jurafsky and James H. Martin, 2000. Speech and language processing.
Kung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljavei'o c, Shang- Wen Li, Wen tau Yih, Yoon Kim, and James Glass. contrastive learning for sentence embeddings. In North American Chap- ter of the Association for Computational Linguistics.
Tao Lei, Hrishikesh Joshi, Regina Barzilay, T. Jaakkola, K. Tymoshenko, Alessandro Moschitti, and Lluís Mårquez i Villodre. 2015. Semi-supervised question retrieval with gated convolutions. North American Chapter of the Association for Computational Linguistics.
2022. Diffese: Diffcrence--asee
Michael Collins and Nigel Duffy. 2002. New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In Pro- ceedings of the 40th Annual Meeting of the Associa- rion for Computational Linguistics.
Yongjie Lin, Yi Chem Tan, and Roberta Frank. 2019. Open sesame: Getting inside bert's linguistic knowl- edge. In BlackboxNLP@@.C..
Tal Linzen and Marco Baroni. 2020. Syntactic structure from deep learning. ArXiv, abs/2004, 10827.
Alexis Conneau, Germán Kruszewski, Guillaume Lam- ple, Loie Barrault, and Marco Baroni. 2018. What you can cram into a single S& * vector: Probing sentence embeddings for linguistic properties. In An- nual Meeting of the Association for Computational Linguistics.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of Istms to learn syntax- sensitive dependencies. Transactions of the Associa- tion for Computational Linguistics, 4:521-535.

---

Christopher D. Manning and Hinrich Schlitze. 1999. Foundations of statistical natural language process- ing. In SGMD.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang. Yushi Hu, Mari Ostendorf, Wen tau Yih, Noah A. Smith, Luke Zeullemoyer and Tao Yu. 2023, One embedder, any task: intrruttio-fiietuned text em- beddings. ArXiv, abs/2212.097411
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. 2014. A sick cure for the evaluation of com- positional distributional semantic models. In Inter- national Conference on Language Resources and Evaluation.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Tryna Gurevych. 2021. BEIR: A heterogeneous benchmark for sero-shot evaluation of information retrieval models. In Thirty-fifth Con- ference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntac- tic heuristics in natural language inference. ArXiv, abs/1902.01007.
Wei Xu, Chris Callison- Burch, and William B. Dolan, 2015. Semeval-2015 task I: Paraphrase and semantic similarity in twitter (pit). In International Workshop on Semantic Evaluation.
Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive sci- ence, 34 8: 1388-429.
Yuanmeng Yan, Rumei Li, Sirui Wang. Fuzheng Zhang. Wei Wu, and Weiran Xu. 2021. Consert: A con- trastive framework for self-supervised sentence rep- resentation transfer. ArXiv, abs/2105.11741.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. In Conference of the European Chapter of the Association for Computational Linguistics.
Mengying Yu and Aixin Sun. 2022. Dataset versus re- ality: Understanding model performance from the perspective of information need. Journal of the As- sociation for Information Science and Technology. 74:1293 1306.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas A. Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul. Girish Sastry, Gretchen Krueger, David P. Schnurr, Felipe Petroski Such, Kenny Sai- Kin Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. 2022. Text and code embeddings by contrastive pre- pre-training ArXiv, abs/2201. 100005... 10005.
Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim. and Lidong Bing. 2020. An unsupervised sentence embedding method by mutual information maximiza- tion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1601-1610, Online, Association for Computational Linguistics.
Benjamin Newman, Kai-Siang Ang. Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evalu- ation of language models. ArXiv, (abs/2004,09655.
Jianmo Ni, Gustavo Hern' andez Abrego, Noah Con- stant, Ma, Keith Hall, Daniel Matthew Cer, and Yinfei Yang. 2021. Sentence-t5; Scalable sentence encoders from pre-trained text-to-text models. In Proc. of ACL Findings.
Barbara H. Partee. 1995. Lexical semantics and compo- sitionality.
Nils Reimers and Tryna Gurevych, 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference Empirical Methods in Natural Lon- guage Processing.
Yikang Shen, Shawn Tan, Alessandro Sordoni, Siva Reddy, and Aaron C. Courville, 2020. Explicitly modeling syntax language models with incremen- tal parsing and dynamic oracle. In North American Chapter of the Association for Computational Lin- guistics.
Gizem Sogancioglu, Hakime Öztürk, and Arzucan Özgür. 2017. BIOSSES: semantic sentence simi- larity estimation system for the biomedical domain. Bioinformatics, 331445:44

---

transformed into "Two kids fighting path in the woods," through verb replacement, serves We also do additional research which call Lexi- cal Compositionality. refers how the individual meanings of words come together to form the com- posite meaning of the larger linguistic structure in which they are embedded. For evaluating text embedding models' adeptness in lexical composi- tionality, we create variants of the sentences in the foundation datasets through the insertion replace- ment of different types of words: nouns, adjectives, and verbs. The modified sentences retain the basic structure of the originals but feature an additional or changed word, either noun, adjective, or verb. This method dataset generation aims to examine whether text embedding models can adapt to and accurately capture the semantic changes brought by these insertions and replacements.
test the model's proficiency in redirecting focus to the altered semantic components the sentence. A Lexical Compositionality
We follow experiment settings in Section 3. The results are reported in Table 6.
B Generated Example
Table showcases an example of sentence pertur- bation.
Noun: The insertion or replacement of noun is anticipated to modify the subject matter the en- tities referred to within the sentence. For instance, the original sentence, "The cat jumped over the fence," could be transformed into "The cat jumped over the garden fence," by inserting the noun "gar- den". Similarly, by replacing the noun "cat" with "dog", rewrite the original sentence "The dog jumped over the fence". This method allows to gauge the text embedding models' ability to discern the introduction of new elements within the sen- tence's subject matter and evaluate their sensitivity and adaptability to these semantic alterations.
Adjective: The insertion replacement of an adjective adds changes descriptive aspect to the sentence, and imperative for text embed- ding model to recognize how these change the sen- tence's attributes or qualities. For instance, by mod- ifying the sentence "The car fast." " to "The lead- ing car is incredibly fast.", we can test the model's ability to comprehend the enhanced emphasis on the car's position and its speed. The robustness of embedding models can be verified.
verb can modify the actions or states conveyed in the sentence while the verb alter may radically trans- form its essence. Thus, it's crucial for text em- bedding models to detect how this influences the sentence's 's dynamics. For instance, changing "She reads books." " to "She reads and enjoys books." by inserting the verb "enjoys" examines the model's capability to understand the introduction an ad- ditional action. In a similar way, when the sentence "Two kids are walking on a path in the woods." is
Verb: The insertion or replacement

---

Model
STS
SICK
AskUbuntu
CQADupStack
Twitter
BIOSSES
Avg
Lexical Compositionality
SimCSF-BERT Base-sup
32.77
27.89
45.54
5.01
17.86
40.06
28.19
COnCSE-BERTT Large-sup
33.29
28.99
45.18
4.49
17.55
40.74
28.37
SinCSE-RoBERTa Laagge-e----ggsuuupp
38.24
35.32
48.06
6.59
19.64
42,69
31.76
30LRT-allMMiiLMAM
MML6-v22
25.00
19.21
46.93"
6.88
17.78
25.13
23.49
3BFRRTall-mpmeet--se-222
30.82
28.96
45.53
18.14
28.11
26.58
Sentence-T3-Base
34.24
23.90
48.30
6.39
22.41
36.00
28.54
Sentence-TS-Large
36.11
24.48
48.38
21.61
38.79
29.33
Sentence-TT-XL
37.49
25.94
48.81
21.46
36.31
29.62
Instructor Base
28.02
21.48
45.61
7.31
18.20
26.96
24.76
Instructor-Large
32.69
28.00
46.92
8.31
18.68
39.30
29.15
Instructor-Xe XL
28.57
20.86
49.97
7,81
20.38
31.18
26.46
26.43
26.70
49.77
6.67
20.73
26.37
26.11
Table 6: Results of five text embedding models on the Lexical Compositionality Benchmark. Spearman's correlation is reported.
Input: A group of people are nervous about crossing the water.
Structural Heuristies
Prompt: Change the active and passive voice of the sentence.
Score
Turn the sentence into inverted one.
Crossing the water makes a group of people nervous.
4
Nervous about crossing the water, group of people are.
Relational Understanding
Prompt:
Change the subject relation in the sentence, it needs you to change your thinking habits and say something anticonventional.
Score
The water is nervous about crossing a group of people.
Lexical Compositionality
Prompt:
Randomly add or replace nouns verbs / adjectives the sentence, but Score
keep the structure and other parts of the sentence unchanged.
A group of tourists are nervous about crossing the water.
4
A group of people are nervous about traversing over the water.
5
A group of anxious people are afraid of crossing the deep water.
3
Table 7: Example of generated perturbations given the input sentence and specific prompts.


Reviews:
[{"review_id": "On4UfEuBcU", "reviewer": "Reviewer_QKSJ", "summary": "", "questions": "A. Can you provide more details about how you retrain the SBERT model in section 4?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable insights and constructive feedback.\n\n> **Q1:** It is unclear to me why insertion and replacement of a noun/verb/adjective (or lexical compositionality as defined by the authors) can be considered syntactic changes. These operations do not change the sentence structure, but rather add or alter semantic content. I think concept order manipulation is in fact more related to lexical compositionality, as it studies how the order of subject and object affects the sentence-level semantics.\n> \n\n**A1:** Insertion and replacement of parts of speech (e.g., nouns, verbs, adjectives) present challenges for syntactic generalization. Such concepts about syntax have been discussed in related papers [1,2] or textbook [3], which is the insight we borrowed from.\n\nAs mentioned in chapter 2 of [1] and [3],  words in sentences often have hierarchical relationships, represented by tree structures in formal syntactic theories.  Inserting or replacing words can disrupt or modify these hierarchies.  For example, given a sentence: \"He shot the man with the gun.\"  and the replacement: \"He shot the man with the camera.\" In this example, replacing \"gun\" with \"camera\" creates syntactic ambiguity. In the original sentence, \"with the gun\" likely indicates the instrument used for shooting, while in the revised sentence, it can be interpreted as the man having a camera. This change introduces different parsing options and affects syntactic interpretation.\n\nMoreover, parts of sentences can work together as one group, while changing words can break up these groups or make new ones. For instance, given a sentence: \"I enjoy coffee.” and the  insertion example: “I enjoy making coffee.”. The  verb \"enjoy\" originally took a simple direct object, \"coffee.\" Now, it's taking a gerund phrase, \"making coffee,\" which has its own internal structure. Such insertion can alter the composition and boundaries of syntactic constituents. These constituents are pivotal in parsing and understanding sentence structure, as they represent the building blocks of sentences and play a crucial role in comprehending how words are grouped together.\n\n> **Q2:** It is important to include the performance on the original datasets, especially when the models compared are trained on different datasets/domains. For example, the authors claim that sentence-t5, which is trained on web-based QA datasets, is more robust than other baselines. However, it is unclear whether such improvement is a result of better syntactic understanding, or simply due to sentence-t5 being trained on the same domain as the benchmark dataset.\n> \n\n**A2:** The text embedding models studied in our research, such as Sentence-T5, SBERT, Instructor, OpenAI-ada, and SimCSE, have shown impressive performance on standard text embedding tests (the original datasets of our LSR benchmark). This demonstrates these models' strong ability to apply across different domains. You can find the detailed results in [4], and we also touch upon this point in our paper on line 41. Due to space limitations, we haven't included the performance results of the original benchmarks in this version. We will incorporate this information in the updated version.\n\nAmong these text embedding models, Sentence-T5 tends to outperform the others in our LSR benchmark, which includes syntactic variations not present in the original datasets. Hence, we propose that the training data used by Sentence-T5, which is sourced from web-based question-answering, might encompass a more diverse range of syntactic structures compared to other data types.\n\nWe use tree kernel [5] as a measure of sentence structure diversity. The central idea of tree kernel is to count the number of common subtrees between two constituency pars. We compare the syntactic diversity of two corpora: Web_QA (233k sentences) and NLI (275k sentences). We use StanfordCoreNLP to obtain constituency parse trees of sentences. Then, we randomly select 1,000 parse trees and use the tree kernel to calculate their similarity. The results are shown below. The lower the score, the more diverse the sentence structures are.\n\n| Corpus | Tree Kernel Similarity |\n| --- | --- |\n| Web_QA | 0.064 |\n| NLI | 0.072 |\n\nAs we can see, sentences in Web_QA have lower tree kernel similarity than those in NLI, indicating that Web_QA has more diverse sentence patterns and structures.\n\nSince prevailing text embedding models like SBERT, SimCSE, etc. are mainly trained on NLI, we argue that Sentence-T5, which is trained on Web_QA, can capture more syntactic nuances of sentences. This additional experiment supports our hypothesis.\n\n> **Q3:** In section 4 the authors retrain the sentence embedding model on syntactically-perturbed inputs with annotations unaltered, and claim that the model lacks syntactic understanding since the performance on the standard STS test set is on par with the original score. However, such settings somewhat encourage the model to ignore syntactic structures given unaltered labels, especially if the model is further finetuned from a trained SBERT model.\n> \n\n**A3:** Thanks for your comments. There might be some misunderstandings. In section 4, we aim to demonstrate that traditional text embedding evaluation paradigms lack sensitivity to syntactic nuances. To demonstrate this, we retrained the SBERT model  from the scratch with syntax-perturbed data while keeping the annotations unaltered. The results in Figure 3 show that the retrained SBERT model continued to yield good performance on the traditional STS evaluation set. **Specifically, the correlation score following syntax perturbation stood at 85.2%, nearly on par with the original score of 86.2%**. Such empirical results suggest that models can achieve high performance on traditional text embedding evaluation benchmarks without proper syntactic understanding. These findings prompt a more rigorous and discerning evaluation paradigm that factors in both semantic and syntactic elements, paving the way for more accurate assessments of text embedding models.\n\n> **Q4:** Can you provide more details about how you retrain the SBERT model in section 4?\n> \n\n**A4:** We retrain the SBERT model from the scratch using the same training objective and datasets. The only difference is the incorporation of syntax-perturbed data into the training dataset. STS training dataset consists of Sentence1, Sentence 2, and its semantic similarity score is marked by humans. We denote it as **[S1, S2, score]**.\n\nIn Section 4.1, we fix Sentence 1 and do the Order Perturbation and Relational Perturbation on each Sentence 2. Each sentence pair is changed into **[S1, Perturbed S2, score]**. So the model retrained can not learn proper syntactics at all, they can just learn how far the distance is in text embedding space between contents in two sentences.\n\nIn Section 4.2, for the simple data augmentation, we fix Sentence 1, do Lexical compositionality, Structural heuristics, and Relational understanding on each Sentence 1 and ask ChatGPT(gpt-3.5-turbo-0301) to score the semantic similarity between the fixed Sentence 1 and its perturbed one based on similarity scores with explanations and English examples from [6]. We manually verified the generation reliability and annotation rationality. Each sentence pair can be written as **[S1, Perturbed S1, re-score]**. The model retrained can learn proper syntactics variants with correct semantic scores.\n\nThrough enhancing the STS dataset, we finally obtain 10k training examples, 1.5k dev examples and 1.3k test samples.\n\nWe choose microsoft/mpnet-base as our raw model which is also the basic model of the best SBERT text embedding model SBERT-all-mpnet-base-v2. We follow Sentence-BERT training settings [7], use the regression objective function, a batch-size of 16, 4 training epochs, Adam optimizer with learning rate 2e−5, and a linear learning rate warm-up over 10% of the training data. Our default pooling strategy is MEAN. We save the best parameters according to the dev set at the end of each epoch. At evaluation time, we compute the cosine-similarity between the sentence embeddings. All systems are trained with 10 random seeds to counter variances. We will clarify these settings in our revised manuscript.\n\n[1] Barbara H. Partee. 1995. Lexical semantics and compositionality.\n\n[2] Edward Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition.\n\n[3] Andrew Carnie. 2012. Syntax: A generative introduction, volume 18. John Wiley & Sons.\n\n[4] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. In EACL 2023\n\n[5] Collins, M., & Du y, N. (2002). New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In ACL\n\n[6] Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic Textual Similarity. In Proceedings of *SEM 2013.\n\n[7] Reimers N, Gurevych I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow in general.\n2. The proposed benchmark identifies the limitations of syntactic understanding of existing sentence embedding models, and provides a valuable resource for improving sentence embeddings in terms of some syntactic aspects that are overlooked by previous work.", "reasons_to_reject": "1. It is unclear to me why insertion and replacement of a noun/verb/adjective (or lexical compositionality as defined by the authors) can be considered syntactic changes. These operations do not change the sentence structure, but rather add or alter semantic content. I think concept order manipulation is in fact more related to lexical compositionality, as it studies how the order of subject and object affects the sentence-level semantics.\n2. It is important to include the performance on the original datasets, especially when the models compared are trained on different datasets/domains. For example, the authors claim that sentence-t5, which is trained on web-based QA datasets, is more robust than other baselines. However, it is unclear whether such improvement is a result of better syntactic understanding, or simply due to sentence-t5 being trained on the same domain as the benchmark dataset. \n3. In section 4 the authors retrain the sentence embedding model on syntactically-perturbed inputs with annotations unaltered, and claim that the model lacks syntactic understanding since the performance on the standard STS test set is on par with the original score. However, such settings somewhat encourage the model to ignore syntactic structures given unaltered labels, especially if the model is further finetuned from a trained SBERT model.", "questions_to_authors": "A. Can you provide more details about how you retrain the SBERT model in section 4?", "paper_topic_and_main_contributions": "This paper studies how well sentence embedding models understand several syntactic aspects including lexical compositionality, structural heuristics, and relational understanding. To this end, this work proposes a new benchmark for evaluating syntactic understanding of sentence embedding models by applying syntactic transformations to existing datasets with ChatGPT. Results on this benchmark suggest that existing sentence embedding models do not perform well in terms of the three syntactic aspects, and can be improved with targeted data augmentation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0ARSD2F31p", "reviewer": "Reviewer_H1cQ", "summary": "", "questions": "A. In section 2.2, is it possible that the words being inserted or replaced are unrelated to the original semantics of the sentence at all? Take an example, is it possible the word \"cat\" in sentence \"The cat jumped over the fence\" is replaced by \"car\" rather than \"dog\"? would it reduce the reliability of the benchmark?\nB. Will the corpus be public？", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable insights and constructive feedback.\n\n> **Q1:** Thought the authors reported the correlation scores between the ChatGPT annotations and human annotations, the reliability of the generated sentences is expected to improve further.\n> \n\n**A1:** Thank you for the suggestions. As outlined in section 2.5, we have undertaken a quality validation process using human evaluators. Randomly selected pairs of sentences are presented to human reviewers who assess and verify their reliability. Any sentences that are inaccurately generated will be corrected as needed. The following table presents the statistical breakdown of the revision rate for these corrected sentences: \n\n| Dataset |  | STS |  |  |  | Twitter |  |  |  | CQADupStack |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Samples | Revised samples | Revise rate |  | Samples | Revised samples | Revise rate |  | Samples | Revised samples | Revise rate |\n| Lexical Compositionality | 600 | 28 | 4.67% |  | 600 | 21 | 3.50% |  | 600 | 27 | 4.50% |\n| Structural Heuristics | 400 | 3 | 0.75% |  | 400 | 5 | 1.25% |  | 400 | 9 | 2.25% |\n| Relational Understanding | 400 | 4 | 1.00% |  | 400 | 8 | 2.00% |  | 400 | 6 | 1.50% |\n\nThe data indicates a notably low revision rate within the sampled set, showcasing the high quality of sentences generated by ChatGPT. We will incorporate this updated table in the revised version of our work.\n\n> **Q2:** In section 2.2, is it possible that the words being inserted or replaced are unrelated to the original semantics of the sentence at all? Take an example, is it possible the word \"cat\" in sentence \"The cat jumped over the fence\" is replaced by \"car\" rather than \"dog\"? would it reduce the reliability of the benchmark?\n> \n\n**A2:** There is a procedure of scoring the two sentences after the sentence generation, and if “cat” is replaced with “car”, according to similarity scores with explanations and English examples from [1], the generated sentence will be scored low similarity with the anchor sentence. As explained in A1, human reviewers will also carefully examine and address any quality concerns. The statistics highlight that the sentences generated by ChatGPT exhibit a remarkably high level of quality and coherence.\n\n> **Q3:** Will the corpus be public？\n> \n\n**A3:** Yes, we indicated that LSR benchmark and code will be released.(line 103)\n\n[1] Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic Textual Similarity. In Proceedings of *SEM 2013.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of this paper is reasonable and intersting.\n2. They develope a new benchmark for syntac understanding.\n3. They propose a simple but effective strategy to improve model's understanding of syntax.", "reasons_to_reject": "Thought the authors reported the correlation scores between the ChatGPT annotations and human annotations, the reliability of the generated sentences is expected to improve further.", "questions_to_authors": "A. In section 2.2, is it possible that the words being inserted or replaced are unrelated to the original semantics of the sentence at all? Take an example, is it possible the word \"cat\" in sentence \"The cat jumped over the fence\" is replaced by \"car\" rather than \"dog\"? would it reduce the reliability of the benchmark?\nB. Will the corpus be public？", "paper_topic_and_main_contributions": "This paper argues that the existing text embedding models are inadequate for syntax understanding. To this end, the authors introduce a new benchmark LSR for assessing a model's understanding of syntax. LSR  is designed from three syntactic dimensions: Lexical compositionality, Structural heuristics and Relational understanding among concepts. They also propose a data augmentation strategy using examples tailored for  syntactic understanding, to improve model's performance on LSR benchmark. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fk4ycyysh8", "reviewer": "Reviewer_neEv", "summary": "", "questions": "A. The structural heuristic example in Figure 2-a didn't seem clear to me. Can you clarify that?\nB. Can you please clarify the ChatGPT model and date of use?\nC. You claimed, \"This configuration not only ensures a holistic evaluation but also effectively pinpoints the models’ strengths and weaknesses in diverse settings\" in line 166. There are some arguments in the discussion manifesting that. Can you please explain how you find models models’ strengths and weaknesses using the results of your benchmark?\nD. Verification was a bit vague to me. Can you please re-explain the data generation verification process? How do you make sure the correct verb is chosen in the Verb part of Lexical Compositionality?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable insights and constructive feedback.\n\n> **Q1:** Although the related work section discusses a range of studies, it overlooks some recent works on syntax generalization, including (https://aclanthology.org/2021.acl-long.289v2.pdf), which delves into systematic generalization with large language models by providing a benchmark. This work would be more compelling by direct comparison of their benchmark with others.\n> \n\n**A1:** There might be some misunderstandings.  Our primary goal is to emphasize the limitations of existing **text embedding models instead of large language models** in comprehending syntax. We achieve this by introducing our LSR benchmark. It's worth noting that the benchmarks proposed in the referenced study are designed to **assess large language modeling rather than text embeddings**. We appreciate the reviewer for bringing up the related work on syntax probing, and we intend to address this work in our revised version.\n\n> **Q2:** The paper does acknowledge the beneficial impact of data augmentation and proposes a simple method to bolster syntactic generalization. However, it falls short by not comparing this method with recent techniques, such as ([https://arxiv.org/pdf/2112.07610.pdf](https://arxiv.org/pdf/2112.07610.pdf)).\n> \n\n**A2:** Thank you for mentioning the related study. Our work focused on developing a new benchmark to test the syntax understanding of text embedding models. We have also shown that automatic data augmentation using LLMs effectively addresses these models' limitations. The cited study deals with data augmentation through example recombination to boost the compositional generalization of semantic parsers. Their method depends on the QCFG rules and may not work for all syntactic generalizations. We will discuss and address this work in our upcoming version.\n\n> **Q3:** While the authors claim to explore why prior evaluations failed to detect model ineffectiveness, they do not convincingly substantiate this with evidence or clearly elucidate how their benchmark differs from earlier works. Notable missing includes related works such as ([https://aclanthology.org/2020.acl-main.158.pdf](https://aclanthology.org/2020.acl-main.158.pdf)).\n> \n\n**A3:**  Our primary goal is to emphasize the limitations of existing **text embedding models**  in comprehending syntax. To the best of our knowledge, our work represents the first attempt to construct such a benchmark.  The referenced study is designed to **assess large language modeling rather than text embeddings**. We appreciate the reviewer for bringing up the related work on syntax probing, and we will discuss this work in our revised version.\n\n> **Q4:** The study reports an ~83% agreement rate between human and AI annotators, a figure which may not be sufficiently robust for evaluating large language models. This discrepancy could potentially compromise future evaluations of larger language models, undermining the contributions of this work to the field.\n> \n\n**A4:** Thank you for your comments. It's important to acknowledge that established human annotators frequently demonstrate inter-annotator agreement rates within the 80-90% range, which can vary based on task complexity and the subjectivity inherent in interpreting language. In this context, achieving an agreement rate of around 83% aligns with the inherent diversity in human annotation and establishes a practical benchmark for AI performance.\n\nFurthermore, we have taken steps to ensure the reliability of annotation and generation through human validation. As outlined in section 2.5, we have undertaken a quality validation process using human evaluators. Randomly selected pairs of sentences are presented to human reviewers who assess and verify their reliability. Any sentences that are inaccurately generated will be corrected as needed. The following table presents the statistical breakdown of the revision rate for these corrected sentences:\n\n| Dataset |  | STS |  |  |  | Twitter |  |  |  | CQADupStack |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Samples | Revised samples | Revise rate |  | Samples | Revised samples | Revise rate |  | Samples | Revised samples | Revise rate |\n| Lexical Compositionality | 600 | 28 | 4.67% |  | 600 | 21 | 3.50% |  | 600 | 27 | 4.50% |\n| Structural Heuristics | 400 | 3 | 0.75% |  | 400 | 5 | 1.25% |  | 400 | 9 | 2.25% |\n| Relational Understanding | 400 | 4 | 1.00% |  | 400 | 8 | 2.00% |  | 400 | 6 | 1.50% |\n\nThe data indicates a notably low revision rate within the sampled set, showcasing the high quality of sentences generated by ChatGPT. We will incorporate this updated table in the revised version of our work.\n\n> **Q5:**  Despite utilizing ChatGPT for data annotation, the paper does not specify the version and the date of the model employed for data generation, which may limit the study's applicability.\n> \n\n**A5:** The api version of ChatGPT is gpt-3.5-turbo-0301. We will update this information in the revised version.\n\n> **Q6:** Line 336 makes a claim about models performance on the foundational datasets, but it doesn't provides any results or citation for the claim,\n> \n\n**A6:** We have mentioned this claim and referenced the related study in the introduction (see line 41). We'll also add the citation to line 336.\n\n> **Q7:** The structural heuristic example in Figure 2-a didn't seem clear to me. Can you clarify that?\n> \n\n**A7:** In Figure 2-a, the sentence \"Her hair is being styled by the girl\" serves as a passive transformation of \"A girl is styling her hair.\" This transformation involves rearranging elements: the original object becomes the new subject, and the past participle of the verb is preceded by a form of \"to be.\" This shift in structure is a fundamental attribute of passive voice constructions in the English language. We will make this clear in the revised version.\n\n> **Q8:** Can you please clarify the ChatGPT model and date of use?\n> \n\n**A8:** The api version of ChatGPT is gpt-3.5-turbo-0301. We will update this information in the revised version.\n\n> **Q9:** You claimed, \"This configuration not only ensures a holistic evaluation but also effectively pinpoints the models’ strengths and weaknesses in diverse settings\" in line 166. There are some arguments in the discussion manifesting that. Can you please explain how you find models models’ strengths and weaknesses using the results of your benchmark?\n> \n\n**A9:** The text embedding models examined in our study, such as sentence-T5, SBERT, Instructor, OpenAI-ada, and SimCSE, have demonstrated impressive performance on conventional text embedding assessments (the original datasets of our LSR benchmark). This underscores these models' robust capacity to be applied across diverse domains. You can find comprehensive results in [1], and this point is also touched upon in our paper on line 41. Due to space constraints, we have not included the performance outcomes of the original benchmarks in this version. However, we intend to integrate this information in the revised version.\n\nHowever, all of these models exhibit weak performance with low correlations on the LSR benchmark. This indicates that existing text embedding models have not been adequately fine-tuned to effectively address the challenge of syntactic comprehension.\n\nFurthermore, among the aforementioned text embedding models, sentence-T5 consistently outperforms the others in our LSR benchmark, which incorporates syntactic variations not present in the initial datasets. This leads us to posit that the training data utilized by sentence-T5, derived from web-based question-answering sources, likely encompasses a more diverse array of syntactic structures in comparison to other types of data.\n\n> **Q10:** Verification was a bit vague to me. Can you please re-explain the data generation verification process? How do you make sure the correct verb is chosen in the Verb part of Lexical Compositionality?\n> \n\n**A10:** Thank you for the comments. As outlined in section 2.5, we have undertaken a quality validation process using human evaluators. Randomly selected pairs of sentences are presented to human reviewers who assess and verify their reliability. Any sentences that are inaccurately generated will be corrected as needed. The following table presents the statistical breakdown of the revision rate for these corrected sentences:\n\n| Dataset |  | STS |  |  |  | Twitter |  |  |  | CQADupStack |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Samples | Revised samples | Revise rate |  | Samples | Revised samples | Revise rate |  | Samples | Revised samples | Revise rate |\n| Lexical Compositionality | 600 | 28 | 4.67% |  | 600 | 21 | 3.50% |  | 600 | 27 | 4.50% |\n| Structural Heuristics | 400 | 3 | 0.75% |  | 400 | 5 | 1.25% |  | 400 | 9 | 2.25% |\n| Relational Understanding | 400 | 4 | 1.00% |  | 400 | 8 | 2.00% |  | 400 | 6 | 1.50% |\n\nThe data indicates a notably low revision rate within the sampled set, showcasing the high quality of sentences generated by ChatGPT. We will incorporate this updated table in the revised version of our work.\n\n[1] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. In EACL 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The manuscript proposes a robust benchmark featuring a well-reasoned testing hypothesis to appraise the syntactic capabilities of LLMs. Also, their benchmark comes from a variety of sources which make provides a mixture of real-world sentence structures. \n2. Their benchmark covers 3 aspects that are vital for syntactic generalization. \n3. The authors provide a thorough analysis of the results, strengthening their hypotheses and delivering a comprehensive understanding of the results.\n\n", "reasons_to_reject": "1. Although the related work section discusses a range of studies, it overlooks some recent works on syntax generalization, including (https://aclanthology.org/2021.acl-long.289v2.pdf), which delves into systematic generalization with large language models by providing a benchmark. This work would be more compelling by direct comparison of their benchmark with others.\n2. The paper does acknowledge the beneficial impact of data augmentation and proposes a simple method to bolster syntactic generalization. However, it falls short by not comparing this method with recent techniques, such as (https://arxiv.org/pdf/2112.07610.pdf).\n3. While the authors claim to explore why prior evaluations failed to detect model ineffectiveness, they do not convincingly substantiate this with evidence or clearly elucidate how their benchmark differs from earlier works. Notable missing includes related works such as (https://aclanthology.org/2020.acl-main.158.pdf).\n4. The study reports an ~83% agreement rate between human and AI annotators, a figure which may not be sufficiently robust for evaluating large language models. This discrepancy could potentially compromise future evaluations of larger language models, undermining the contributions of this work to the field.\n5. Despite utilizing ChatGPT for data annotation, the paper does not specify the version and the date of the model employed for data generation, which may limit the study's applicability.\n6. Line 336 makes a claim about models performance on the foundational datasets, but it doesn't provides any results or citation for the claim,", "questions_to_authors": "A. The structural heuristic example in Figure 2-a didn't seem clear to me. Can you clarify that?\nB. Can you please clarify the ChatGPT model and date of use?\nC. You claimed, \"This configuration not only ensures a holistic evaluation but also effectively pinpoints the models’ strengths and weaknesses in diverse settings\" in line 166. There are some arguments in the discussion manifesting that. Can you please explain how you find models models’ strengths and weaknesses using the results of your benchmark?\nD. Verification was a bit vague to me. Can you please re-explain the data generation verification process? How do you make sure the correct verb is chosen in the Verb part of Lexical Compositionality?", "paper_topic_and_main_contributions": "This work focuses on the generalization ability of language embedding in different syntactic contexts in language models used for natural language processing. While these models have proved successful in capturing the semantic aspects of a text, their ability to adapt to a wide variety of syntactic contexts requires more exploration. To investigate, they developed an evaluation set named LSR, which explores three key syntactic aspects: \"Lexical compositionally,\" \"Structural heuristics,\" and \"Relational understanding\" among concepts. Their analysis revealed that current text embedding models are not meeting these syntactic understanding challenges effectively. Furthermore, they perform some analysis to identify the factors causing these limitations. In addition, they propose a simple data augmentation method to improve the model's performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The best results in the tables should be highlighted."}, "extra_text": {"Missing_References": "Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational Linguistics.\n"}}]

Author rebuttal:

Meta-review: The paper presents a novel dataset to evaluate ability of sentence embedding models (such as SentenceBERT) to distinguish semantic differences expressed using syntactical means: e.g. A dog chase a cat -> A cat chase a dog. 
The paper uses a semi-automatic process to expand an existing evaluation dataset and then benchmarks existing sentence models, showing their limitations. 
All three reviewers agree in their general assessment of the paper. Authors made significant efforts to further clarify their task during rebuttal period. Parts of their response should be included into the final version, especially the table with human assessment.
Decision: Accept-Findings