C-STS: Conditional Semantic Textual Similarity
Authors: Ameet Deshpande
Carlos E Jimenez
Howard Chen
Vishvak Murahari
Victoria Graf
Tanmay Rajpurohit
Ashwin Kalyan
Danqi Chen
Karthik R Narasimhan
Venue: EMNLP 2023 Main (2023)
Keywords: Conditional similarity
Sentence similarity
Dataset

Abstract: Semantic textual similarity (STS) has been a cornerstone task in NLP that measures the degree of similarity between a pair of sentences, with applications in information retrieval, question answering, and embedding methods. However, it is an inherently ambiguous task, with the sentence similarity depending on the specific aspect of interest. We resolve this ambiguity by proposing a novel task called conditional STS (C-STS) which measures similarity conditioned on an aspect elucidated in natural language (hereon, condition). As an example, the similarity between the sentences "The NBA player shoots a three-pointer." and "A man throws a tennis ball into the air to serve." is higher for the condition "The motion of the ball." (both upward) and lower for "The size of the ball." (one large and one small). C-STS's advantages are two-fold: (1) it reduces the subjectivity and ambiguity of STS, and (2) enables fine-grained similarity evaluation using diverse conditions. C-STS contains almost 20,000 instances from diverse domains and we evaluate several state-of-the-art models to demonstrate that even the most performant fine-tuning and in-context learning models (GPT-4, Flan, SimCSE) find it challenging, with Spearman correlation scores of <50. We encourage the community to evaluate their models on C-STS to provide a more holistic view of semantic similarity and natural language understanding.

C-STS: Conditional Semantic Textual Similarity
Ameet Deshpande" 1,2
Carlos E. Jimenez"*
Howard Chen'
Vishvak Murahari'
Victoria Graf
Tanmay Rajpurohit
Ashwin Kalyan?
Danqi Chen
Karthik Narasimhan'
Princeton University
The Allen Institute for Al asdêcs. princeton. edu
Georgia Tech
Abstract
Semantic textual similarity (STS), a corner- stone task in NLP, measures the degree sim- ilarity between a pair of sentences, and has broad application in fields such as informa- tion retrieval and natural language understand- ing, However, sentence similarity can be in- herently ambiguous, depending on the specific aspect of interest. We resolve this ambiguity by proposing a novel task called Conditional STS (C-STS) which measures sentences' sim- ilarity conditioned an feature described in natural language (hereon, condition). As an example, the similarity between the sentences "The NBA player shoots a three- porree"pinter." and "A man throws a tennis ball into the air to serve." is higher for the condition "The motion the ball" (both upward) and lower for "The size of the ball" (one large and one small). C-STS's advantages are two-fold: (1) it reduces the subjectivity and ambiguity of STS and (2) en- ables fine-grained language model evaluation through diverse natural language conditions. We put several state-of-the-an the-art models to the test, and those performing well STS (e.g. SimCSE, F5an-T5, and GPT-4) find CSTS challenging: all yielding Spearman correlation scores below 50. To encourage prehensive evaluation of semantic similarity natural language understanding, we make nearly 19K C-STS examples and code available for others to train and test their models.
A windsurfer skims the water with his outstretched hand.
The base of the object: Similar
The surfer riding wave with mountain in the background.
The way the object propelled: Dissimilar
Figure I: C-STS: Two sentences are judged by their similarities based on free-form natural language condi- tions. The two sentences are more similar when judged by condition 'The base of the object' (highlighted in yellow) as both windsurfing and surfing use similar board but are dissimilar when judged by the condition "The way the object is propelled" (highlighted in blue) because one is propelled by waves and the other by wind. Providing conditions reduces ambiguity of the sentence similarity task, and allows evaluation of grounded and multi-faccted notion of sentence similarity.
semantic similarity between two sentences. Several diverse STS datasets are popularly used, with prior work expanding the STS task to multiple domains and languages (Agirre et al., 2013, 2014, 2015, 2016; Cer et al., 2017; Abdalla al., 2021). STS tasks have been component of the popular GLUE natural language understanding benchmark (Wang et al., 2018) and are a key evaluation tool for centence-represeenttiie learning specifically (Con- neau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019; Gao et al., 2021, inter alia).
1 Introduction
Over the years, natural language processing (NLP) has progressed through the co-evolution of model design (e.g. architectures, training methods) and evaluation methods for language tasks (Wang et al., 2018, 2019: Hendrycks et al., 2021). common task used to evaluate NLP models has been Seman- tie Textual Similarity (STS) (Agirre et al., 2012), which evaluates the models' ability to predict the
Despite its popularity, STS may be inherently ill-defined. The general semantic similarity of two sentences can be highly subjective and vary wildly depending on which aspects one decides focus on. As observed several studies, ambiguity in similarity judgements of word or sentence pairs can be reduced with the help of context for both hu-
Equal Contribution
'Code: www. www.giie tiub. httpa//prnnnetoonnppccoom

---

Sentence Pair Data Collection
Condition Annotation & Verification
Sample Image- Caption
Candidate Filterlin
large green ball was bouncing on street.
bought small green avocado.
Image Encodor
Image
Provide two conditios:
C-Hligh for sentences
Text Encoder
Conditoo-Writing Workers
Image Embedding
C-Low to have low similatty
sentences
Cosinel
C-Hight The color
C-Low: The size
Cosine(
Select top-K
Thresholding
Veriiication Workers
Captions
Toxt
Text
(Similar)
(Dissimiar)
Figure 2: Illustrating the data collection process for C-STS-2023 (Left) We first show the sentence pair collection procedure ()..2.1). Step A: An image-caption pair is sampled (red) from the dataset and then fed into the image encoder to get the image embedding. The image embedding is compared against all other image embeddings in the dataset (blue) find the top-k similar images. The original caption is then paired with the corresponding captions of the top-k similar images to generate sentence pairs. Step B: The sentence pairs are filtered based on textual similarity. (Right) We illustrate the condition amnotation/verrficttoo procedure ($2.2.2). Once the sentence pairs have been collected, they are sent qualified Mechanical Turkers to get annotations and verify conditions.
mans (De Deyne et al., 2016a.b) and models (Veit et al., 2016; Ye et al., 2022a; Lopez-Gazpio et al., 2017: Camburu et al., 2018).
88.09 on STS-B validation splits for comparison). We believe that C-STS provides a testbed for potentially novel modeling settings and applica- tions. Toward this end, we propose and evaluate unique encoding setting (a tri-encoder) and ob- jectives (a quadruplet contrastive loss with hard negatives) that take advantage of C-STS's three- sentence inputs and paired high- and low-similarity instances.
Considering the importance of STS tasks for evaluating sentence representations, we propose new task called Conditional STS (C-STS), illus- trated in Figure 1, which seeks to disambiguate the similarity of two sentences by measuring similarity within the context of a condition sentence.
Our qualitative analysis shows that models find C-STS challenging when tested on different aspects of the same sentence pair rather than testing conditional and ambiguous notion of similarity. We hope that future work evaluates on C-STS in addi- tion to STS tasks to comprehensively benchmark semantic similarity in language models.
C-STS, uses free-form natural language condi- tions, enabling us to evaluate and probe natural language understanding for myriad fine-grained as- pects. Figure/ 1 illustrates two conditions ("The base of the object" and "The way the object is pro- pelled") which probes language models' concep- tion of similarity for different aspects concerning water sports and physical reasoning. Since condi- tions themselves are unconstrained sentences, they allow to evaluate precise, grounded, and multi- faceted notion of sentence similarity.
2 Methodology
The C-STS task requires sentence pairs, conditions which probe different aspects of similarity, and the similarity label for given sentence pair and con- dition. This section describes the technical details involved in creating the dataset.
To comprehensively test models on C-STS, we create the C-STS-2023 dataset which includes 18,908 instances containing sentence pairs, a con- dition, and a scalar similarity judgement on the Likert scale (Likert, 1932). We find that even state- of-the-art sentence encoders and large language task. Although Sim- CSE (Gao et al., 2021) and GPT-4 (OpenAl, 2023a) are among the best-performing systems, their rela- tively poor Spearman correlation of 47.5 and 43.6 respectively, points to significant room for improve- ment (SimCSE achieves a Spearman correlation of
2.1 Background: Semantic textual similarity
Semantic textual similarity (STS) (Agirre et al., 2012, 2013, 2014, 2015, 2016; Cer et al., 2017) is task which requires machines to make similarity judgements between a pair of sentences ((3), $2)).
models perform poorly
STS measures the unconditional semantic similar- ity between sentences because the annotator mak- ing the similarity assessment must infer which as-

---

pect(s) of the sentences are being referred to. For- mally, consider conditions (c; C) that refer to disjoint aspects of the sentences, then the similarity the two sentences may be represented as:
image pairs, or Py. Each pair, denoted by I,, I,. is such that 1 is amongst the top-k most similar im- ages to 7, determined by the cosine distance metric of their respective image representations obtained via an image encoder (-). After establishing Pı, we convert it into a sentence pair store (Ps) by re- placing each image in pair with its corresponding caption. When each image I, E DD is associated with set of sentences (s); we take all sentence pairs from the Cartesian product {8}; x {s}; for each image pair I, 1 E P/.
C
Σ mi sim 82) s.t. Σ wy -1
Here, wi is the weight assigned by the annotator to the condition G, and sime, ($1, s2) is the similar- ity of the sentences with respect to the condition. These weights are latent the task and each an- notator has their own interpretation of them which helps marginalize similarity, thus introducing am- biguity in the task. C-STS seeks to disambiguate the STS task by measuring similarity conditioned by a single aspect specified in natural language.
Candidate Filtering After acquiring initial sen- tence pairs through image retrieval, we perform ad- ditional filtering to climinate sentence pairs which are ill-suited for our task.
Specifically, we aim to include only pairs of sentences for which the unconditional similarity is somewhat ambiguous, as this incentivizes models to rely on the condition when reasoning about the conditional similarity.
2.2 Conditional semantic textual similarity
C-STS is a task comprised of quadruplets con- taining two sentences (a sentence pair), natural language condition, and similarity assessment (1s1, c2, v)). Crucially, strict constraints on c, allowing it be any rel- evant phrase. This allows us to probe potentially any possible aspect of similarity that may be con- sidered between sentences.
To this end, we avoid high similarity sentence pairs by filtering out those with a high bag-of- words intersection over union and avoid low sim- ilarity sentence by choosing sentences with mod- crate or high cosine similarity of their SimCSE embeddings (Gao et al., 2021), See Appendix A.2 for a full description of all filtering criteria used.
do not place any
2.2.1 Sentence Data Collection
Dataset sources For the construction of sen- tence pairs candidates, we use two Image-caption datasets: the train split from the 2014 MS COCO dataset (Lin et al., 2014) containing 83,000 im- ages, and Flickr30K (Young et al., 2014) contain- ing 31,000 images. Each dataset is processed separately and we do not intermix them during the retrieval stage. We use CLIP-VIT (Radford et al., 2021) to encode images and include the specific filtering criteria in Table of Appendix A.2.
The first stage of making the C-STS dataset is to acquire the sentence pairs that will later be used in eliciting conditioning statements from annotators.
We source sentence pairs {s1, 82) for our dataset from image-captioning datasets through two-step process: (1) generate candidate text pairs through dense retrieval from the corresponding image rep- resentations and (2) filter out candidates that are irrelevant or ineffectual for our purposes.
Image Retrieval Image-captioning datasets pro- vide a compelling data source because image pair similarity and caption (text) pair similarity encode different semantics (Parekh et al., 2021). Image- representations thus serve as an informative latent variable which can represent their captions in ways that are not captured by text retrievers.
2.2.2 Annotation Methodology
For each sentence pair in the store (Ps), we wish to collect conditions and semantic similarity anno- tations for each sentence pair and condition triplet. $2, c). As is free-form natural language sentence, the annotator is provided with high- level of control over which aspect to condition on. Human annotations are acquired through Mechani- cal Turk in a 3-stage process.
{s1, {21,82;c)
Since current sentence representation models overlook aspects of conditional similarity, uti- lize both the image and text to retrieve sentence pairs which form the foundation of our dataset.
Stage 1: Choosing a high-quality worker pool In the first stage, we design qualification test to select workers who excel at our task. Specifically, we test two skills: (1) The quality of conditions they
We aim to derive sentence pairs from an image- caption dataset D to aid in creating conditioning statements. To do this, we first generate a store of

---

write for a given sentence pair and (2) semantic similarity judgements for a triplet {si, 82, et. We choose pool of 271 workers who perform well on both tasks and restrict subsequent stages to include only workers from this pool. See Appendices C.1 and C.2 for an example of these tests.
1971) be 0.61 which implies substantial inter- annotator agreement. Average length of sentences and conditions is 12.6 and 5.3 words.
Qualitative analysis C-STS allows to evalu- ate the generally fuzzy notion of sentence similarity with greater fidelity. We illustrate this in Table 1, where precise and discriminative conditions allow a targeted, fine- fine-grained, and grounded definition of sentence similarity. The following is repre- sentative instance where the conditions tease out nuanced and hidden similarities and differences between the two lexically similar sentences on surf- ing: Consider $1: "A windsurfer skims the water... and $2: "The surfer is riding a wave...") While the sentences are significantly dissimilar based on the condition "the way the object is propelled they talk about windsurfing and surfing respectively (the former uses a sail whereas the latter depends on the wave), they are very similar in context of the condition "the base of the object* as both wind- surfing and surfing use similar board.
Stage 2: Condition annotation After sourcing sentence pairs { $2) using the strategy discussed in the Section 2.2.1, we instruct workers annotate each pair with one condition such that the similarity in its context high (C-High) and one such that the similarity in its context is low (C-Low). Example: s1 A large green ball was bouncing the street s2 bought a small green avocado
C-Hligh: The color of the object
C-Low The size of the object
We do not place any constraints on the conditions other than that they should be semantically unam- biguous phrases and relevant to the sentence pair (Appendix C.I).
Our diverse set of conditions provides broad sup- port over the distribution of conditions and enables holistic and multi-faccted evaluation of sentence similarity. For example, the conditions for the sen- tences on Tennis in Table 1 test similarity both on the sport being played (which requires understand- ing lexical and knowledge artifacts) as well as the number of people (which requires reasoning and commonsense capabilities).
Stage 3: Condition verification and similarity assessment The output of annotations from the previous stage are triplets {81, 82, c} with binary similarity assessment (high low). In this stage we ask new annotators to assign similarity Likert scale (Likert, 1932) (as an integer between and 5) as common with semantic textual simi- larity tasks (Agirre et al., 2012). In addition to as- signing a similarity, we also use this stage to verify if the conditions from the previous stage are perti- nent to the sentence pairs, filtering out potentially low quality examples. At the end of this stage, have {$1, $2, c, y) quadruplets which have passed layer of human verification (Appendix C.2).
4 Baselines
We evaluate our dataset on several baselines which can be categorized into (1) Fine-tuning baselines, which are pre-trained models finetuned on the C- STS training split, and (2) Large language models (LLMs) baselines, which are evaluated using in- structions and in-context examples.
3 Dataset Analysis
4.1 Fine-tuning baselines
Dataset statistics To ensure high- quality faith- ful, and diverse annotations, we collect a total of 20,000 instances and perform quality assurance (Section 5.3) resulting in a total of 18,908 instances part of the C-STS-200 2023 dataset. Following stan- dard practice, we create train, validation, and test splits in 60: 15: 25 ratio. We present the dis- tribution of similarity scores, which are discrete numbers between 51, in Figure 4. We also mea- sure the inter-annotator agreement on a random sample of 100 examples with three independent annotations and find Fleiss' kappa score (Fleiss,
We evaluate three sentence encoder models RoBERTa (Liu et al., 2019), supervised Sim- CSE (Gao et al., 2021) and unsupervised Dif- fCSE (Chuang et al., 2022). SimCSE and DiffCSE represent state-of-the sentence encoder models which are particularly strong on STS tasks. For both SimCSE and DifICSE, we use the RoBERTa pre-trained varieties.
Encoding configurations Encoder-only Trans- former models, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), initially performed

---

Sentence
Sentence 2
Condition and Similarity
An older man holding glass of wine while standing be- tween two beautiful ladies.
A group of people gather around table with bottles and
The people's demeanor: 5 The number bottles:
glasses of wine.
Various items are spread out on the floor, like bag has been emptied.
A woman with bag and its contents placed out before her
The arrangement of objects: 4 The surface the objects are on:
on a bed.
The surfer is riding with a mountain in the back- ground.
The base of the object: 5 The way the object is propelled:
A windsurfer skims the water with his outstretched hand.
A young lady dressed in white playing tennis while the ball girl retrieves tennis ball be- hind her.
The sport being played:
Female tennis player jumping off the ground and swinging racket in front of an audience
The number of people:
Table 1:
Four examples from the C-STS validation set. Under different conditions, the same sentence pair can separated into high similarity and low similarity. Scale from (dissimilar) to (similar).
regression finetuning for STS tasks by simply con- catenating the sentences and encoding them to- gether before generating a prediction; let us call this type of architecture a cross-encoder. Recent approaches instead opt to encode sentences sepa- rately and compare their similarity using distance metric, such the cosine distance Reimers and Gurevych (2019); which we will call bi-encoder. While DiffCSE and SimCSE were designed with the bi-encoder setting in mind, we observe that they work well in the cross-encoder setting as well.
the cosine similarity as cos (h (c; (c; 82)). We experiment with functions for h, an MLP and the Hadamard product.
Objectives In addition to the standard MSE loss for regression, we use quadruplet contrastive mar- gin loss which we denote Quad. Since each sen- tence pair in C-STS comes with two conditions (one with higher similarity and one with lower sim- ilarity) we represent the conditional encoding of each sentence in the and p2 and represent the conditional encoding each sentence in the lower similarity pair as nı and n2. The Quad loss is then defined as follows:
pair as pr
For our baselines, we evaluate each model in both settings. For the cross-encoder configura- tion, we encode the triplet containing the sentences and the condition (fs1, $2, e}), and the output is a scalar similarity score fo(s); e). For the bi-encoder configuration (Reimers and Gurevych, 2019), the sentences of pair are encoded indepen- dently along with the condition using Siamese network and their cosine similarity is computed cos(.f((1: c), fe (82; c)).
Qpi, P2; n1, n2) max + cos(nn (n1, n2)
cos(pI.P2). 0)
where À is a margin hyperparameter.
We train all of our tasks for regression using, alternatively, mean squared error (MSE), Quad, and a linear combination of the quadruplet loss and MSE (Quad MSE). Since we require a separate conditional encoding fore each sentence, the Quad and (Quad + MSE) objectives apply only the the bi-encoder and tri-encoder configurations.
In addition to the bi- and cross-encoder mod- els, we propose tri-encoder models which encode each sentence and condition separately. This con- ceptually resembles late-interaction contextualized retrieval approaches, such as Humeau et al. (2020) or Khattab and Zaharia (2020), but our approach is specific to C-STS. For this, first encode all sen- tences of the triplet separately, with encoder (-(-) s; fs:); where s; E Rd We then perform an additional transformation ates on the condition and one each of the sentences. We finally compute the conditional similarity using
We evaluate the baselines on the test split for C-STS. We perform hyperparam- eter sweep to select the best performing configura- tion and test using models trained with 3 random seeds, with further details in Appendix A.3. As a comparison for our training setting, we perform a similar hyperparameter sweep for the STS-B (Cer et al., 2017) dataset, with the validation split results
Hyperparameters
2d IRdd that oper-

---

and and best hyperparameters shown in Table 9, show- ing that our finetuned baselines achieve very strong performance on traditional STS tasks.
C-STS
STS-B
Model
Spear.
Pears.
Spear.
Pears.
DiSICS
0.9
0.5
84.4
85.1
RoBERTa TRRASE
-0.4
-0.1
35.2
48.2
4.2 Large language models baselines
RoBERTa AIEERTaLARCE BLARGE
-1.8
-2.4
7.3
15.1
For the generative setting, we evaluate two types of models (I) cnttrucctionfinetuenn encoder- decoder models, including Flan-T5 (Chung et al., 2022), Flan-UL2 (Tay et al., 2023), and Tk- INSTRUCT (Wang et al., 2022) and (2) propri- etary autoregressive LLMs including ChatGPT- 3.5 (OpenAI, 2022) and GPT-4 (OpenAI, 2023a). For ChatGPT-3.5 and GPT-4 we use the Ope- nAI API with versions 3pt-3..-turbo--e gpt-4-0314 respectively.
SimCSE BASSE
1.7
0.8
85.1
86.8
SimCSE
1.9
1.4
88.9
Table 2: Zero-shot bi-encoder models evaluation results on C-STS and STS-B validation data. These results ver- ify that strong performance STS tasks do translate to C-STS, suggesting substantial room for improvement fine-grained sentence embedding models.
Encoding
Model
Spear.
Pears.
When evaluating zero- or few-shot capabilities, each model input is composed of up to three parts: instruction (task definition), in-context exam- ples, and query. Models are evaluated with 0, or examples and using three different instruction prompts: no instruction, short instruction, which provides only a high-level description of the task, and long instruction, shown in Figure 6, which re- sembles the annotation guidelines and is similar to the instructions used for the STS-B classification task in Wang et al. (2022).
RoBERTASASE
39.2
39.3
MoBERTaLARGR
40.7+0.
40.8
Cross- encoder
DiffCSEBASE
38.8129
39.0
SimCSERAsE
38,6
38,9
SimCSE
43.2
43.2
RoBERTàgAsE
28.1
22.3
RoBERTaLAKC
27.4
21.33
encoder
DiffCSEsAsE
43.4
43.5 10.2
SimCSERAsE
44.8
44,9 10.33
SimCSE
47.5
47.6
RoBERTHBASE
28.0
25.2
ROBERTaLARGE
20.3
18.942.
Tri- encoder
DitiCSEaAs
28.9
27.8
For few-shot evaluation, we additionally always group a sentence pairs' two conditional similar- ity examples together, models will always see contrasting pairs in the examples, but won't see paired example for the query. We provide exam- ples of the formats used for the input and output for more settings in Appendix As we did for the finetuned models, we also evaluate these models on the STS-B validation split, shown in Table 12, with instruction finetuned models and ChatGPT achieving strong performance.
SimCSF
31.5
31.0
SimCSE
35.3
35.6
Table 3: We report fine-tuned model test split results in Spearman and Pearson correlations for three models (RoBERTa, DiffCSE, and SimCSE) in different encod- ing settings.
Fine-tuning baselines We finetune sentence encoder baselines on C-STS and show the test per- formance in Table 3. Again, the best models are SimCSE and DiffCSE in the bi-encoding setting. This suggests that the sentence representations learned in their contrastive learning phase facilitate learning for C-STS substantially, but still struggle with all Spearman correlation below 50.
5 Results
5.1 Evaluating sentence encoders on C-STS
Zero-shot bi-encoder performance As an ini- tial comparison, we evaluate bi-encoder models without finetuning, on both C-STS and STS-B. B. As shown in Table 2, we see that strong performance STS-B does not translate to good performance C-STS, suggesting that these models fail entirely incorporate the provided conditioning statement. These results suggest that current approaches to training sentence encoders may be too specialized to existing tasks for evaluation, such as STS-B.
Performance on C-STS varies significantly de- pending on the encoding configurations, with the bi-encoder setting proving be the most effee- tive, especially for SimCSE and DiffCSE models. Performance of the tri-encoder model, introduced in Section 4. was generally poor, with mod- cls performing well below their bi-encoding and cross- cross-encoding counterparts.

---

Instruct.
Model
0-shot
2-shott
4-shot²
Spearman's Correlation
Cross Erooder
'SimCSE, ARM
47.5
40
BiFunde
Flan-T3xı.
1.7
11.3
16.4
Flan-T5xxu
10.1
12.8
30
None
Flan-UL2
5.1
18.8
14.9
Tk- Tk-Instructss
-1.8
1.1
2.8
20
Tk-Instruch
4.3
GPT-3.5
12.6
1.6
3.1
GPT-4
21.0
18.7
27.0
10
1000
3000
5000
7000
Flan-T3xı.
24.7
25.3
24.8
Dataset size
Flan-T5xxa
30.6
29.7
29.2
Flan-UL2
20.7
22.4
23.2
Figure 3: Model (SimCSELAACE) performance scaling as the dataset size increases. Across encoder types, Spearman correlation increases the dataset scales.
Short
Tk-Instructy
-0.3
3,9
4.9
Tk-Instruct
10.1
21.9
17.1
GPT-3.5
15.0
15.6
15.5
GPT-4
39.3
42.6
43.6
Flan-T5xx.
26.6
26.3
26.0
5.3
Analysis
Flan-T3xxu
30.5
30.1
30.6
Flan-U1.2
21.7
22,9
23.5
Scaling laws for C-STS We evaluate the effect of the quantity of C-STS data on sentence-embedding methods for SimCSE (Figure 3). We no- tice that for all three encoding strategies, perfor- mance monotonically increases as we increase the size of the training dataset. For example, for the SimCSE bi-encoder, the Spearman correlation in- creases from 30 when using train set of 1,000 examples to 45 for 7,000 examples.
Long
Tk- Tk-Insiructan
-0.9
3.9
3.9
Tk-Isstruct
12.0
20.7
17.8
GPT-3.5
9.9
16.6
15.3
GPT-4
32.5
41.8
43.1
Table 4: Few-shot Spearman correlation the test split. Models perform much worse than their finetuned coun- terparts, with GPT-4 being the only evaluated model that achieves comparable performance to some fine- tuned baselines. Fine-tuning on the full train set.
There is almost a linear increase the perfor- mance the models, especially the bi-encoder as we increase the amount of data. This quantitatively enforces the quality of the dataset, but also retroac- tively makes that point that rather than relying on more data, require better modeling strategies.
5.2 Evaluating pre-trained LLMs
We show performance of generative models eval- uated on C-STS in various prompting settings in Table with some additional results for smaller Flan-T5 models Table 11 in the Appendix. No- tably, the state-of-the-art language model. GPT-4, performs substantially better than all competing models and systems (UL2, Flan-T5, CharGPT-3.5) and is competitive with finetuned SiRCSELARGE model, the best performing sentence-encoder. For example, in most settings, GPT-4 outperforms ChatGPT-3.5 and Flan models by over 10 points. This suggests existing large language benchmarks may correlates with C-STS as GPT-4 has shown to be the most proficient wide variety of evalua- tion settings AI, 2023b).
Qualitative analysis We present predictions from different models Table 5 to illustrate sys- tematic pitfalls. For instance, Flan-T5 makes incor- rect predictions even for straightforward instances and falsely predicts that both sentences talk about the same dish, even though the sentences clearly talk about sandwiches and pizza respectively. Ad- ditionally, ChatGPT-3.5 incorrectly predicts that the two sentences are completely dissimilar when talking about the types of plants, even though both sentences mention flowering plants. Note that our annotation, unlike ChatGPT-3.5, captures the nu- ance that the first sentence talks about both shrub- bery and flowers, while the second sentence talks only about flowers, and therefore assigns conser- vative similarity score of 3. The most proficient model on C-STS, GPT-4, is much better at cap- turing these nuances and accurately predicts, for instance, that the height of the giraffe's head (refer to the fourth example), high in one sentence and
Between suites of models of different sizes (viz. Flan-T5, Tk-Instruct), we observe a strong corre- lation between model scale and performance. We also find that providing instructions improves per- formance substantially for C-STS and that this per- formance is robust to different instructions lengths and the number of in-context examples.

---

Model
Sentence
Sentence 2
Condition
Output
A man taking bite out of a sandwich at table with some- one else.
man sitting with pizza in his hand in front of pizza on table.
Flan-T5-
Pred: 4.5
Type of dish.
Base
Label: 1.0
A wooden bench surrounded by shrubbery and flowers on the side of house.
A scene displays vast array of flower pots in front decorated building.
The type of plants.
Pred: 0..0 Label: 3.0
GPT-3.5
A football player preparing football for a field goal kick, while his teammates coach watch him.
Football player jumping to catch the ball with an emply stand behind him.
The game be- ing played.
Pred: 3.0 Label:
GPT-4
giraffe in zoo bending over the fence towards where impalas are grazing.
The height of the giraffe's head.
A giraffe reaches up his head on a ledge high up on a rock.
Pred: 1.0
GPT-4
Label: 1.0
Table 5: Examples of model predictions evaluated C-STS in the in-context setting (K 2 with instructions). We choose examples with different levels of accuracy, showcasing different failure cases of model behavior.
low in another. GPT-4 is far from perfect though, and we outline a negative prediction (refer to the third example), where the model does not predict that the two sentences talk about the same game, even though they are very clearly about "Football".
ambiguity problem by collecting conditions and hence reduce subjectivity. To alleviate ambiguity, explanations play an important role in identifying the differences between the two sentences either in their syntactical structure (Lopez-Gazpio et al., 2017) or in natural language (Camburu et al., 2018), but the post-hoc nature of explanations prevents it from being used prior to the similarity judgement, rendering it supplemental component opposed to a paradigm change in the task setup. Beyond STS, works that leverage conditioning to enhance sentence representations obtain improved perfor- mance for retrieval (Asai et al., 2023) and embed- ding qualities (He et al., 2015; Su et al., 2023; Jiang et al., 2022), which corroborates the observation that conditioning as a form of disambiguation ben- efits similarity measures.
More broadly, C-STS provides a lens into model's s ability to understand and reason over spe- cific parts of each sentence and is well-suited to revealing systematic modeling issues.
6 Related Work
Historical perspectives of semantic similarities Measuring semantic similarities is a long-standing problem spanning cognitive science (Miller and Charles, 1991) to psychology (Tversky, 1977) where early attempts are made to quantify the sub- jective similarity judgements with information the- oretical concepts. More recently, interest in seman- tic similarity has gained popularity in the context of machine learning, with works in computer vision recognizing that the notion of similarity between images varies with conditions (Veit et al., 2016) and can therefore be ambiguous (Ye et al., 2022b).
7 Conclusion
In this work, we propose conditional semantic tex- tual similarity (C-STS), novel semantic similarity assessment task that resolves the inherent ambigu- ity in STS, Given the importance of STS and its importance in sentence representation evaluation we believe that C-STS timely and necessary addition to the language model evaluation land- scape. Rather than testing unconditional semantic similarity, the diversity of conditions in our dataset allows fine- -naiiid evaluation. The same sentence pairs can be tested on variety of different aspects represented by conditions, with similarities often varying significantly. C-STS poses challenging hurdle to both encoder-only and state-of-the-an generative language models which struggle to cap- ture the high-dimensional manifold similarity.
Textual similarity tasks Capturing textual simi- larity is also considered a fundamental problem in natural language processing. Works such as Agirre et al. (2012, 2016) define the textual semantic simi- larity tasks (STS), which is widely used in common benchmarks such as GLUE (Wang et al., 2018). Extensions to the STS setting have been proposed such making the task broader with multilingual- ity (Cer al., 2017) or incorporating relatedness (Abdalla et al., 2021). However, the loose defini- tion of similarity has not been acknowledged as an issue explicitly. In contrast, our work tackles the

---

We believe that a combination of improved model- ing and fine-tuning strategies are required to push the boundaries on C-STS and we hope that C-STS can enable innovative future work in language un- derstanding and representation learning.
Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012).
Volume 2:
Eneko Agiree, Daniel Cer. Mona Diab, Aitor Gonzalez- Agime, and Weiwei Guo. 2013. *SEM 2013
shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semanties (*EEM), Volume I: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 32-43, Atlanta, Georgia, USA. Association for Computational Linguistics.
Limitations
We propose the novel task of conditional semantic textual similarity (C-STS), Given that this is new task, we collect a dataset of over 19,000 instances, but one limitation that this size can be increased to ensure sentence embedding style models have additional data for fine-tuning. Further, we use two different sources to collect our sentence pairs, and future studies, motivated by STS follow-ups, can collect data from other sources.
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Ha- jishirzi, and Wen-tau Yih. 2023. Task-aware retrieval with instructions. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3650- 3675, Toronto, Canada. Association for Computa- tional Linguistics.
Oama-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018, e-snli: Nat- ural language inference with natural language ex- planations. Advances in Neural Information Processing Systems, volume 31. Curran Associates. Inc.
References
Mohamed Abdalla, Krishnapriya Vishnubhotla, and Saif M. Mohammad, 2021. What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study. ArXiv:21 10.04845 [es].
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of the I 1tth International Workshop on Semantic Evaluation (SemEval-20117) pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, lñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-20155 task 2: Semantic tex- tual similarity, English, Spanish and pilot inter- pretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252-263, Denver, Colorado. Association for Computational Linguistics.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2214 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81-91, Dublin, Ireland. As- sociation for Computational Linguistics.
Yung- Chuang. Rumen Dangovski, Hongyin Luo. Yang Zhang, Shiyu Chang, Marin Soljacic, Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. DifICSE: Difference-basee contrastive learning for sentence embeddings. In Proceedings of the 2022
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agiree Rada Mihalcea, Ger- man Rigau, and Janyce Wiebe. 2016. SemEval- 2016 Task I: Semantic Textual Similarity, Monolin- gual and Cross-Lingual Evaluation. In Proceedings of the 10th International Workshop on Semantie Evaluation (Sem6)vll-0116, pages 497-511, San Diego, California. Association for Computational Linguistics.
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207-4218, Seattle, United States. Association for Computational Lin- guistics.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling instructionn-inetume language models. arXiv preprint
Eneko Agirre. Daniel Cer, Mona Diab, and Aitor Agirre, 2012. 2emEval-2012 Task 6: Pilot on Semantic Textual Similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics Volume Proceedings of the main conference and the shared task, and
221.22210..11
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from

---

natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680, Copenhagen, Denmark. Association for Computa- tional Linguistics.
Empirical Methods in Natural Language Processing. pages 8826-88333 Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR "20, page 39-48, New York, NY, USA. Association for Computing Machinery.
Simon De Deyne, Daniel J Navarro, Amy Perfors, and Gert Storms. 2016a. Structure at every scale: A semantic network account of the similarities be- tween unrelated concepts. Journal of Experimental Psychology: General, 145(9):1228.
Simon De Deyne, Amy Perfors, and Daniel J Navarro. 2016b. Predicting human similarity judgments with distributional models: The value of word associations. Proceedings of coling 2016, the 26th international conference on computational linguistics: Technical papers. pages 1870.
Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dol- lár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European Conference on Computer Vision.
Jacob Devlin, Ming-Wi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Asso- ciation for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
I. Lopez-Gazpio, M. Maritxalar, A. Gonzalez-Agirre G. Rigau, I. Uria, and E. Agirre. 2017. Interpretable semantic textual similarity: Finding and explaining differences between sentences. Knowledge-Basd Systems, 119:186-199 186-199.
Joseph L. Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological Bulletin. 76:378-382.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SiMCSE: Simple contrastive learning sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6919 6900, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
George Miller and Walter G Charles. 1991. Contex- tual correlates of semantic similarity. Language and Cognitive Processes.
OpenAl. 2022. Introducing ChatGPT. https://
openai. coomai.com/blog/// com/blog/chatgpt.
OpenAL. 2023a. Gpt-4, Accessed: 2023-05-23.
Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multi- Perspective Sentence Similarity Modeling with Con- volutional Neural Networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1576-1586, Lisbon, Portugal. Association for Computational Linguistics.
OpenAL. 2023b. Gpt-4 technical report.
Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Wa- ters, and Yinfei Yang. 2021. Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Simi- larity Judgments for MS-COCO. arXiv:2004.1500 [es]. ArXiv: 2004.15020.
Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 1.21. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural lan- guage supervision. In International Conference Machine Learning.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architec- tures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations.
Nils Reimers and Iryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese bert- networks, In Proceedings of the 2019 Conference on Empirical Methods in Natural I Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-JJCNLP)), pages 3982-3992.
Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang. Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. 2022. Prompt- BERT: Improving BERT sentence embeddings with prompts. In Proceedings of the 2022 Conference on

---

Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang. Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finctune beddings. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1102- I121, Toronto, Canada. Association for Computa- tional Linguistics.
semantic matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 666 16110-6619.
text em-
Han-Jia Ye, Yi Shi, and De-Chuan Zhan. 2022b. Identi- fying Ambiguous Similarity Conditions via Semantic Matching. ArXiv:2204.0444 [cs].
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions visual denotations: New similarity metrics for semantic ference over event descriptions. Transactions of the Association for Computational Linguisties, 2:67-78.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Gar- cia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations.
Amos Tversky. 1977.
Features of similarity. Psychological Review, 84:327-352.
Andreas Veit, Serge Belongie, and Theofanis Karalet- sOs. 2016. Conditional similarity networks. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1781-1789.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: stick- ier benchmark for general-purpose language under- standing systems. arXiv preprint 905.00537.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task task benchmark and analysis platform for natural language understanding. EMNLP 2018, page 353.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganch Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai. Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong General- ization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Shen. 2022. Super=-aaurlllnstruciooo
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Jamie Brew. 2019. Huggingface's 's transformers: otate-of-the--e art natural language processing. CoRR. abs/1910.03771.
Han-Jia Ye, Yi Shi, and De-Chuan Zhan. 2022a. Identifying ambiguous similarity conditions via

---

A Appendix
any of these criteria. These thresholds were lected based primarily manual inspection of sam- ples on their margins. Criteria such as ratio and length are used primarily to facilitate comparison. Sentences with very different lengths are more dif- ficult to compare, are sentences that are very short contain few details.
A.1
Distribution annotated similarity in the dataset
Distribution of Similarities
3000
2500
Frequency
2000
COCO
Flickr30K
1500
k
64
128
1000
IOU
0.12
0.2
dixt
2 0.4
0.4
500
ratio
0.7
2 0.7
length
2. 50
2 48
2
4
5
Similarity Score
Table 6: The list of filters criteria and values used for each dataset. Sentence pairs that violate any criterion are discarded.
Figure 4: The train split distribution of similarity judge- Likert scale between [] 5].
The distribution of similarities is equitably spread out over the Likert scale, depicted in Figure 4.
A.3 Evaluation Details
Implementation Details All models, with the ex- ception of the ChatGPT systems, are trained and or evaluated in PyTorrh using the Huggingface Trans- formers library (Wolf al., 2019) and pre-trained weights repository. We use the STS-B dataset as co/docs/
A.2 Sentence Pair Generation Details
Here include some further details about sourc- ing sentence pairs from image-caption datasets. As discussed in Section 2, we use a variety of metrics to quantitatively characterize the sentence pairs, and then to filter with the goal of removing pairs with excessively high or low unconditional similarity. The general criteria consider defined as follows:
distributed on ht https: httg///guggggeeeeeeeeeeeeeeggegt datasets as part the GLUE (Wang et al., 2018) evaluation benchmark.
Finetuned Baselines For evaluation of the fine- tuned baselines on C-STS, we perform a hyperpa- rameter sweep to select the best training settings for each model and encoding method before evaluating on the test split of C-STS. We show the hyperpa- rameter values used in the sweep in Table 7, and the final hyperparameter values chosen in Table 8. We evaluate random seeds using the best validation configuration to evaluate on the test data, with final results reported in Table 3.
IOU - This is computed by taking the intersec- tion union of the bag of words for each sentence, after stopword removal. It repre- sents the lexical similarity and overlap of sentence pair.
The cosine distance of the pair's Sim- CSE embeddings. We chose SimCSE due to its ubiquity and effectiveness.
diet
We additionally perform extensive evaluation of our models on STS-B. We perform comparable validation sweep as shown in Table 7, reporting the best performing hyperparameters and their perfor- mance in Table 9.
ratio This is the ratio of the shorter sen- tence's word count to the longer sentence's word count given pair.
Lastly, perform data ablation training ROBERTaRASE model alternatively on only the dition and only the sentence pair. The model trained to predict similarity based on the condition statement alone recovers non-trivial performance, but falls well behind the full-input baseline.
length This is the character length of the shortest sentence a pair.
Using these criteria, we filter the sentence pairs based upon thresholds (exact values shown in Ta- ble 6) where sentences are rejected if they violate

---

Batch Size
(32)
Figure 6 shows an example for long instructions and zero-shot setup.
Encoding Type
[Cross, Bi-, Tri-]
Epochs
[3] {le-5, 3c-5)
Learning Rate
Instructions
IRR Schedule
(linear) MSS.. Quad, Quad + MSE] ([CLS] w/ MLP, w/o MLP) [42]
Objective
On a scale between and how similar are the following two sentences with respect the condition provided? Respond only with a score between and 5.
Pooler Type
Seed
Warmup Ratio
[0.1] {0, 0.1
Weight Decay
Examples
Table 7: Hyperparameter sweep done for C-STS vali-
Input: Sentence 1: A bunch of blue buses parked in parking lot front of a housing community.
dation for finctuned models. The same sweep, with the exception of the Encoding Type and Objective hyper parameters are done for ST
Sentence 2: Two buses, blue and one red and white, are going to different desti- nations.
Generative Baselines We report more details of results of the generative baselines for the validation sets of C-STS and STS-B.
Condition: The type of transportation. Output: 4.0
Input: Sentence A bunch of blue buses parked in parking lot front of housing community.
For comparison validation performance of other models, we include the validation perfor- mance for C-STS in Table 11, which largely mir- rors performance on the test set. We notice, expect- edly, that models frequently output non-numerical responses in settings where there are no instruc- tions to do so, or no in-context examples to follow.
Sentence 2: Two buses, one blue and one red and white, are going to different desti- nations. Condition: The number of buses. Output: 1.0
Query
On STS-B validation performance, models gen- erally perform much better than on C-STS, with some models performing comparably to finetuned models. Since STS-B B is included as task Natu- ral Instructions v2 (Wang et al., 2022), it is likely to be recognizable to Flan-T5 models, which counts Natural Instructions v2 in its training data. Like- wise, STS-B B is comprised of long-existing and pop- ular datasets, which plausibly exist in the the cor- pora used to train ChatGPT models.
Input: Sentence 1: The skater is descend- ing the wooden wall beside the slope Sentence boy skateboards off ramp covered in graffiti.
Condition: The location.
Output:
Figure We show the full input for 2-shot setting with short instructions.
Processing Prompting Baseline Generations For parsing prompting model generations, we al- low for maximum of 20 generation tokens. The output is stripped of non-numeric characters and errant punctuation before being cast float. For example, the response "The Answer is 2.0." pro- cessed as 2.0 and counts as a valid prediction. If the cast fails, we mark the answer invalid and replace the predictions by number ~ U [1,5].
C Crowdsoureing Guidelines
C.1 Condition Annotation
We provide the complete condition annotation guidelines used for Mechanical Turk data collec- tion in Figure 7.
C.2 Condition Verification
We provide the complete verification guidelines used for Mechanical Turk data collection in Fig- ure 8.
B Prompt Examples
All prompts for the prompting baselines may con- sist of instructions, examples, and query, though we include evaluations for no instructions and no examples in our results. Figure shows an prompt example for the short instructions and K = 2 and

---

Modeling type
Leurning
Weight Decay
Transform
Dbyective
Trl-Eaeoder Op.
Speammnn
Peurson
Croes Encoder
3.0e-05
0.10
True
MSE
41.02
40.95
RoBERTTRasE
Ri Ficedler
3.0c-05
MSE
57.17
To Encoder
3.0e-05
0.00
False
Quad + MSE
Concez
28.70
27.30
Cross Enowler
1.0c-05
0.10
True
MSE Quad
40.21
40.49
RoBERTI, seer
35.81
Tei Encoder
L.e-05
0.00
True
MSE
Hadamard
21.82
21.16
DiTUSF
Ri Finender
3.0o0s
0.00
False
MSF
42.18
41.85
Encoder
3.0e-05
Quad MSE
Hadamard
30.60
29.59
Cross Encoder
3.0c-05
0.10
MSE
33.91
34.90
Bi Eneeder
3.0e-1t5
0.10
Palse
MSE
45.67
45.55
Cross Encoder
L5
0.10
True
MSE
44.31
44.42
SinCSEr
Bi Faceoler
Tri Encoder
Lite-05
0.000
Tiue
MSE
Hadamard
34.46
34.95
Table Fine-tuning models' results validation split. We show the best performing configuration selected the validation split which was the final configuration used to report each models' test performance.
Model
Encoding
Learning Rate
Transform
Objective
Spearman
Pearson
Cross Encuder
3.0e:05
True
MSE
90.54
90.55
RoBERTapAs
Bi Encoder
3.0c-05
False
MSE
87.23
86.73
Cross Encoder
3.0c-05
False
MSE
89.75
89.82
DiffCSE
Encoder
3.0e-05
False
MSE
88.08
87.66
Cross Encoder
3.0c-05
True
MSE
91.49
91.58
RoBERTaLAACC
Encoder
3.0c-05
False
MSE
87.79
87.25
Cross Encoder
3.0c-05
True
MSE
89.50
89.65
SimCSE
Encoder
3.0c-05
False
MSE
89.69
89.30
SimCSE
Cross Encoder
3.0e-05
True
MSE MSE
91.73
91.78
Enneuder
1.0e-05
False
90.70
90.56
Table 9: Validation performance best sweep setting on STS-B.
Data Ablation
Spear.
Pears.
Condition Only
28.21
28.62
Sentence Only
9.98
9.51
Baseline
40.11
40.21
Table 10: When finetuned only with condition statement, RoBERTa model can recover non-trivial performance, but falls well behind the baseline. Training on only the sentence pairs proves be even less informative. We report the best validation performance over the same hyperparameter grid described in Section 4.1.

---

O-shot Pears.
2-shot
4-shot
Instruction
Model
Invalid
Spear.
Invalid
Pears.
Spear.
Invalid
Pears.
Spear.
Flan-T55MALL
91.74
3.23
2.20
35.64
7.06
8.20
24.21
7.14
6.98
Flan-T5mase T5SS
97.18
-4255
-3.65
6.42
5.51
9.86
2.40
11.39
12.11
Flan-T5LARCE
98.69
-2.86
-1.47
13.37
13.27
13.26
2.68
13.98
12.74
Flan-T5xi.
86.27
-0.81
-0.69
8.29
11.21
12.81
0.53
18.15
17.05
Flan-T5xxL.
74.14
3.21
3.78
0.14
11.37
12.05
0.00
10.28
12.08
Flan-U112
83.87
0.53
4.39
3.03
16.32
18.97
0.28
15.32
17.69
None
Tk-Instructz
87.33
-2.06
-2.05
0.67
2.12
1.70
0.00
0.26
0.32
Tk-Instructus
22.37
2,36
5.58
0.21
8.00
8.43
2.65
3.15
3.76
ChaiGPT-3.5
65.24
5.80
11.21
17.57
3.91
2.43
6.49
6.31
GPT-4
59.17
9.01
16.69
4,98
16.10
15.56
0.60
26.74
26.59
Flan-T5sMALL
0.00
-5.02
-7.43
0.00
-4,99
-5.81
0.00
-4.44
-4.29
Flan-T5PAsB
0.00
5.78
6.24
0.00
6.03
6.51
0.00
$.44
5.75
Flan-T5 LLARRR
0.00
13.98
13.89
0.00
12.29
12.68
0.00
10.54
11.11
Flan-T5xx.
0,00
25.00
25.34
0,00
24.12
24.24
0,00
21.63
23,02
Flan-T5xxL.
0.00
29.95
29.50
29.69
30.39
28.95
29.49
Flan-UL2
0.00
20.83
20.24
0.00
21.98
21.70
0.00
22.52
22.62
Short
Tk-Instructan
76.36
0.34
1.62
0.04
5.23
5.19
0.00
2.80
2.68
Tk-Instruct
0.04
9.50
11.78
0.04
22.10
23.84
0.00
15.65
17.56
ChaGGPT-3.5
0.00
12.91
11.13
0.04
16.63
17.62
0.07
12.60
13.76
GPT-4
0.00
38.77
39.47
39.76
41.25
41.52
42.05
MannTTSMAALL
0.00
-1.48
-1.58
0.00
-2.71
-3.14
0.00
-4.00
-4.67
Flan- -5nAK
0.00
6.53
6.08
0.00
10.87
10.44
0.00
8.47
7.59
Flan-T5LARGE
0.00
11.21
10.64
0.00
11.37
11.08
0.00
10.68
10.25
Flan-T5xi.
0.00
24.97
25.01
0.00
23.76
23.86
0.00
23.59
23.72
Flan-T5xxi.
0.00
29.71
29.79
0.00
30.68
30.69
0,00
30.01
29,99
Flan-U1.2
0.00
21.27
21.07
0.00
22.08
21.64
0.00
21.91
21.56
Long
Tk-Instruct3n
0.14
2.10
1.88
0.00
4.29
3.84
0.00
0.95
0.93
Tk-Instruct
0.00
9.24
11.24
0.00
19.82
21.23
0.00
16.06
17.38
ChaiGPT-3.5
0.00
8.42
15.46
15.70
GPT-4
0.00
33.48
33.04
0.00
39.08
39.53
0,00
42.26
42.38
Table 11: Validation performance for prompting baselines C-STS.

---

O-shot Pears.
2-shot
4-shot
Instruction
Model
Invalid
Spear.
Invalid
Pears.
Spear.
Invalid
Pears.
Spear.
Flan-T55MALL
89.20
0.53
0.80
48.00
-0.90
-3,26
46.87
-2.00
4.21
Flan-T5mxs T5SS
92.87
-1.38
4.03
3.67
0.25
41.81
3.67
40.71
39.76
Flan-T5LARCE
90.07
-1.06
5.64
9.67
5.29
65.38
2.67
67.44
68.90
Flan-T5xi.
87.53
-1.71
73.70
72.77
76.41
Flan-T5xxL.
63.80
-4,34
13.41
0.00
65.44
67.50
0.00
70.87
71.60
Flan-U112
97.00
-0.41
2.74
6.60
73.17
75.51
1.53
80.01
81.66
None
Tk-Instructaa
69.20
3,90
5.22
0.07
8.04
7.97
0.27
6.65
8.34
Tk-Instruct
2.87
3.39
8.71
0.13
5.43
9.88
0.13
11,65
15.47
ChaiGPT-3.5
96.93
-4.17
1.61
64.83
74.96
76.15
GPT-4
63.20
-2,40
20.10
0.00
76.70
75.92
0.00
86.16
86.25
Flan-T5sMALL
0.07
18.44
18.43
0.07
19.09
19.21
0.00
19.97
20.41
Flan-T5$AsB
0.00
80.98
80.94
0.00
80.91
80.97
0.00
81.27
81.31
Mlan-T5LARGR
0.00
87.85
87.89
0.00
86.90
87.34
0.00
86.44
86.88
Flan-T5xx.
0.00
89,69
89.76
0.00
89.57
89.48
0.00
89.53
89.36
Flan-T5xxL
0.00
89.80
89.79
88.33
88.62
86.54
87.38
Flan-UL2
0.00
91.57
91.62
0.00
91.72
91.62
0.00
91.60
91.48
Tk-Instructan
63.93
5,00
20.17
0.00
49.86
51,24
0.00
48.08
48.58
Tk-Instructus
0.07
35.04
34.79
0.00
50.65
54.01
0.00
48.48
51,99
ChaGGPT-3.5
0.00
86.58
86.78
0.00
83.69
83.13
0.00
85.12
84.91
GPT-4
88.20
88.95
88.44
88.96
Flan-T5 T5
0.00
6.33
5.98
0.00
14.92
14.75
0.00
16.81
16.06
Flan- T5
0.00
82.32
82.22
0.00
82.42
82.49
0.00
82.14
82.16
Flan-T5LARGE
0.00
89.81
89.86
0.00
89.85
89.85
0.00
89.55
89.56
Flan-T5xi.
90.33
90.62
89.86
90.12
0.00
90.43
90.66
Flan-T5xx1.
0.00
90.75
90.97
0.00
91.58
91.51
0.00
91.50
91.35
Flan-U1.2
0.00
91.02
91.68
0.00
91.45
91.90
0.00
91.67
92.05
Long
Tk-Instruct3p
0.00
23,90
26.76
0.00
66.89
67.63
0.00
65.19
66.04
Tk-Instruct
0.00
64.09
65.20
0.00
67.93
69.06
0.00
61.38
63.65
ChaiGPT-3.5
86.59
86.90
GPT-4
0.00
89.57
89.76
0.00
90.01
89.95
0.00
90.73
90.65
Table 12: Validation performance for prompting baselines STS BB

---

Instructions
Evaluate the similarity between the two sentences, with respect to the condi- tion. Assign the pair score between 1 and as follows:
Definition:
1: The sentences are completely dis- similar with respect to the condition.
2: The two sentences are dissimilar, but are on a similar topic with respect to the condi- tion.
3 : The two sentences are roughly equiva- lent, but some important information differs or missing with respect to the condition. 4: The two sentences mostly equivalent, but some unimportant details differ with re- spect to the condition.
5: The two sentences are completely equiv- alent with respect to the condition.
Query
Input: Sentence 1: Elderly sitting on blue couch reading a paper. Sentence Older man riding public trans-
portation while reading a newspaper.
Condition: The location of the man. Output:
Figure 6: The full text input for the zero-shot evaluation with large language models, using 'long' instructions. Emphasis and section titles added for clarity.

---

We will be hosting more HITs in the future and we invite you to attend those. Please send any feedback you have to: placeholder@gmaillcom
Task summary
Our goal understand the similarity sentence pair based condition. Concretely, for a sentence pair (S1 and S2), provide one condition (C-High) such that S1 have high similarity, and one condition (C-Low) such that they have low similarity.
As example:
A large green was bouncing on the street.
S2: bought small green avocado.
C-High: The color (High Similarity because green in both sentences)
C-Low: The size (Low Similarity because large
second
sentence)
Conditions are English phrases which are used to choose an aspect of the sentence.
Guidelines for conditions
You allowed the internet to understand the sentences, but the conditions need written by you. The following guidelines need to followed.
1. Conditions should be grammatically correct English phrases or sentences.
2. Avoid conditions which reference missing information that cannot be inferred from sentences. For example, avoid following condition, because the color the animal in $2 unclear.
a. S1: Brown bears attacked people in the night.
b. 92: Some dogs were barking the road.
C. C-High: The color of the animal.
But the following
good condition because can inferred the game is chess:
d. S1: Black ultimately reached an endgame two pawns
92: Now the white king comes just in time to support his
C-High: The game being played.
Conditions should reference aspects or attributes of sentences not the
values. For example, the following ('The color green") is an incorrect condition because directly mentions "green', which is the value of the attribute "color": $1: A large green watermelon.
Figure 7: Annotation Guidelines

---

b. S2: A green avocado the basket.
C. C-High: The color is green.
Instead. the same condition correctly be written as: "The color of the fruit".
4. Avoid conditions which explicitly words like "sentences". For example, instead of saying "the color in sentence", just say
color".
Avoid vague conditions which do not help narrow down a specific aspect of the sentence. For example, avoid conditions which simply say "The activity". which does not help narrow down aspect. Instead hobby" as much as possible.
informative words like "the sport" or "the
Whenever possible, try to write conditions which refer to abstract similarity.
Consider the following sentences:
Two women calebrating goal.
couple is eating tasty meal.
condition which more abstract preferred:
c. Abstract condition: The sentiment of the people.
Although more literal condition valid, it is less preferred:
Literal condition: number people.
Examples
provide good and bad examples of conditions sentence pairs, along with the reasoning.
Good examples
All the following conditions are valid because they follow guidelines.
Sentence 1
Sentence 2
Condition
Similarity
Explanation
The moon looked incredible!
The car was completely covered in snow.
The color of the object.
High
The color is white in both cases. This is a good condition because it references the color of the object without explicitly mentioning it.
group people wearing helmets and riding bikes.
group of bikers gathered together and taking pictures.
The speed of Low the eyclists.
The group of cyclists moving sentence whereas they not the second. Hence their speeds are dissimilar.
Three people are holding ladder while another climbs
Three people are listening to music
The number of people.
Low
There are four people first sentence but only three second sentence.

---

Bad examples
the following conditions are invalid because they ignore one or more of our guidelines.
Sentence
Sentence 2
Condition
Reason for invalidity condition
Egyptians appeased gods with offerings and prayers.
People this faith specific gods protect their lives.
The culture involved.
violates guideline
The culture the second sentence cannol inferred and is missing information.
An adult elephant is playing river.
boulder rolling down the hill.
The size the object is large.
violates guideline 3.
The condition should have been 'The size the object", without explicitly referring to being "large'.
A guitarist playing on bench.
man green hat playing the guilar on the road.
The instrument in the sentence.
violates guideline 4.
The condition would be good "in the sentence" was removed so that it is just "The instrument".
middle-aged is helping construct grass hut.
Three work roof.
The activity.
This condition is too vague and specific aspect. better condition would be: "The type of construction".
reference
man on top of a partially completed roof laying down more shingles.
man hard hat safety gear stands a construction site.
The number of people.
While this condition valid, violates guideline 6, which says that an abstract condition should considered wherever possible.
better condition would have been. "The occupation of the man", which "construction worker" both cases.

---

We will be hosting more HITs the future and we invite you attempt those.
Please send any feedback you have to: placeholder@ggaiilcco
Task summary
Our goal understand the similarity sentence pair based condition. Concretely, for a sentence pair (S1 and $2), and condition 'C', provide score which indicates the similarity of S1 and S2 with respect to C.
As example:
A large green was bouncing on the street. S2: bought small green avocado.
C: The size of the object
Similarity: (Low Similarity because it is large the and small second sentence)
Guidelines for annotating similarity
Part
Given two sentences and condition, first check if the condition applies both the sentences, the condition does not apply even one of sentences, please check the provided to indicate the same. For example:
$1: A small dog happily runs across the street.
S2: bought small green avocado.
C: The sentiment
In the above example, the condition does not make sense for S2 because there no sentiment that can be inferred from it.
Part 2
the condition makes sense, given two sentences and condition, please ascribe similarity score for the sentences when interpreted with respect to the condition.
The score has
of following numbers: (1, 2, 5).
Tips:
Please avoid overusing the middle range score (3) as much as possible.
Feel free to use the extreme scores and if they make sense to you.
The following the meaning the different scores:
1. Score = two sentences are completely dissimilar with respect to the condition.
For example:
Figure 8: Verification Guidelines

---

A man cooks in kitchen.
S2: A woman is riding a bike on the road.
C: The gender (Man and woman dissimilar with respect to gender)
2. Score The two sentences dissimilar, but are similar topic with respect condition.
For example:
$1: A man plays the guitar.
S2: A littie girl listens the violin.
C: The instrument (Both are string instruments, similar but different instruments)
3. Score = 3: The two sentences are roughly equivalent, but some important information differs or is missing with respect to the condition. For example:
A small crowd gathered around injured person.
S2: A crowd jumps up and down to the tunes played by an artist.
C: Number of people
(While both are crowds, is important and unclear how many people there are.)
4. Score 4: The two sentences are mostly equivalent, but some unimportant details differ with respect the condition.
For example:
S1: The little girl plays the jazz guitar.
S2: The guitar looked nice and shiny.
C: The instrument (Guitar both cases, the exact type different unimportant)
5. Score
two sentences are completely equivalent as they mean the same thing with respect to the condition.
For example:
S1: boys play playground.
There are girls near the fountain.
C: The number of people (3 and three are strictly equivalent)


Reviews:
[{"review_id": "95BAlqbtRt", "reviewer": "Reviewer_cVEM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper and give useful comments.\nWe believe it is a good suggestion to increase the size of the dataset and are working on semi-automated ways to collect a large dataset without significantly increased human effort.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper can be accepted for the following reasons: the authors offer a new approach to measure a sentence similarity. The title and abstract are correctly selected and written. The paper is correctly structured, easy to follow. Pictures and tables are relevant and clearly explained.", "reasons_to_reject": "As an option the size of the dataset can be increase to ensure sentence embedding style models have additional data for fine-tuning. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a novel sentence similarity task (called conditional semantic textual similarity  C-STS) that resolves the problem of the inherent ambiguity in semantic textual similarity (STS). The authors create the C-STS-2023 dataset to test models on C-STS. C-STS-2023 dataset includes instances containing sentence pairs, a condition, and a scalar similarity judgement. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3dghnGo3M8", "reviewer": "Reviewer_4jPR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper proposes a new benchmark useful for evaluating the language models\n* The paper contains a nice set of baselines", "reasons_to_reject": "N/A", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new task called conditional sentence similarity assigning similarity scores to a pair of sentences and a condition sentence. The condition sentence makes the task less ambiguous and requires the model to capture more fine-grained semantic signals from the given two sentences. The experiments are conducted with a nice set of baselines: conventional fine-tuning-based methods with encoder-based models and prompting-based methods with fine-tuned and proprietary LLMs.\nThe paper is well-written and easy to follow.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NHky2pSzc4", "reviewer": "Reviewer_L1HG", "summary": "", "questions": "A. Given the step in the data collection procedure in which annotators are explicitly asked to generate both a high similarity and a low similarity condition for each sentence pair, I'm somewhat puzzled as to why the distribution in Figure 4 isn't more balanced between low and high similarity judgments (e.g. it seems like there are almost twice as many 4 and 5 responses (each) as there are 1 responses).\n\n\nB. There is a fair amount of past work at this point, primarily in the NLI literature (e.g. [here](https://aclanthology.org/Q19-1043/) and [here](https://aclanthology.org/2020.starsem-1.9.pdf)) that shows that individual annotators' response distributions (e.g. on a Likert scale) can vary considerably (e.g. some people will tend to cluster their responses in the 2-4 range, while others might tend to use more 1s and 5s), likely even when given criteria for each rating or admonitions to use the full scale like the ones provided in the appendix of this paper. This seems like a potential concern here, given that there is no redundancy in the annotation (except on the 100-example subset used for IAA evaluation, unless I'm mistaken). I would be curious to know if the authors plan to release (anonymized) annotator information along with each example for researchers interested in modeling this variability.\n\nC. A relatively minor point: the train/validation/test split ratio allocates fewer examples to train than I'm accustomed to seeing (I much more commonly see 80/10/10 or at least 70/15/15; perhaps this is an idiosyncratic feature of my research background). Yet, L252-3 says this ratio is \"standard practice.\" If this is so, it would be good to provide references; if not, it would be good to know how this ratio was decided.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review, we greatly appreciate your feedback and comments.\n\n*A. Given the step in the data collection procedure in which annotators are explicitly asked to generate both a high similarity and a low similarity condition for each sentence pair, I'm somewhat puzzled as to why the distribution in Figure 4 isn't more balanced between low and high similarity judgments (e.g. it seems like there are almost twice as many 4 and 5 responses (each) as there are 1 responses).*\n\nWe perform our annotation process in two steps (1) acquiring the C-High and C-Low conditions and (2) acquiring similarity judgements. Step 1 is done without explicit reference to a Likert scale or ordinal rating system, we simply ask for a high and low condition provided some examples. During step 2, similarity judgements are not primed with whether the condition was acquired as high or low and human annotators’ Likert rating may be slightly biased towards higher similarity judgements. Additionally, during step 2, we always have annotators rate whether the condition is high-quality and applicable to both sentences. We find that annotators are more likely to eliminate low-similarity conditions in this phase, which also results in instance pairs with slightly higher-skewed similarity scores. Overall, this results seems to be the result of our straightforward annotation process and natural annotator perceptions. You can see further details on the annotator guidelines in Appendix C.\n\nHowever, our quality assessment and inter annotator agreement scores can confirm that the quality of the dataset is high.\n\n\n*B. There is a fair amount of past work at this point, primarily in the NLI literature (e.g. here and here) that shows that individual annotators' response distributions (e.g. on a Likert scale) can vary considerably (e.g. some people will tend to cluster their responses in the 2-4 range, while others might tend to use more 1s and 5s), likely even when given criteria for each rating or admonitions to use the full scale like the ones provided in the appendix of this paper. This seems like a potential concern here, given that there is no redundancy in the annotation (except on the 100-example subset used for IAA evaluation, unless I'm mistaken). I would be curious to know if the authors plan to release (anonymized) annotator information along with each example for researchers interested in modeling this variability.*\n\nLike you pointed out, there is in fact redundancy when measuring the IAA, but the redundancy does not naturally exists for the rest of the data. But the additional motivation of our work was indeed to avoid the 2-4 gravitation that annotators naturally get into. \nBut just to make sure we understand correctly, could you elaborate on what kind of anonymized annotator information you are looking for? Are they demographics related?\n\n\n*C. A relatively minor point: the train/validation/test split ratio allocates fewer examples to train than I'm accustomed to seeing (I much more commonly see 80/10/10 or at least 70/15/15; perhaps this is an idiosyncratic feature of my research background). Yet, L252-3 says this ratio is \"standard practice.\" If this is so, it would be good to provide references; if not, it would be good to know how this ratio was decided.*\n\nRegarding C) our standard practice referred to having a significantly larger training set that the eval and test sets. The reason we went for a 60:15:25 split is that we wanted to thoroughly test the model with a large amount of data rather than provide a large training set and test on a small test set. This is because C-STS is a novel task and we believe that the community wil benefit from a thorough evaluation. We will change the writing to reflect our motivation. Thank you for pointing this out.\n\n*Given that the judgments are ordinal, Kendall's tau or Krippendorff's alpha seem like more appropriate measures of IAA than Fleiss's k.*\n\nThanks for this suggestion, we will incorporate this in the camera-ready version by providing both the judgements.\n\n\n*Section 7 is more of a \"conclusion\" section than a \"discussion\" section; it might be advisable to rename it to the former.*\n\nWe will rename this accordingly.\n\nWe’d also like to acknowledge your other comments/corrections and we shall be updating the final paper to address each of your suggestions. Thank you again for your time and feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The C-STS task seems well-motivated and potentially of interest to a broad subset of the NLP community, including the IR, NLI, and RTE communities, as well as folks working at the intersection of vision and language.\n- The experiments are quite thorough while leaving ample room for future work, and substantial details about both the data collection procedure and experimental setup are provided in the appendices.\n", "reasons_to_reject": "I do not have strong reasons to reject, but a number of more modest concerns, which I raise in the section below.", "questions_to_authors": "A. Given the step in the data collection procedure in which annotators are explicitly asked to generate both a high similarity and a low similarity condition for each sentence pair, I'm somewhat puzzled as to why the distribution in Figure 4 isn't more balanced between low and high similarity judgments (e.g. it seems like there are almost twice as many 4 and 5 responses (each) as there are 1 responses).\n\n\nB. There is a fair amount of past work at this point, primarily in the NLI literature (e.g. [here](https://aclanthology.org/Q19-1043/) and [here](https://aclanthology.org/2020.starsem-1.9.pdf)) that shows that individual annotators' response distributions (e.g. on a Likert scale) can vary considerably (e.g. some people will tend to cluster their responses in the 2-4 range, while others might tend to use more 1s and 5s), likely even when given criteria for each rating or admonitions to use the full scale like the ones provided in the appendix of this paper. This seems like a potential concern here, given that there is no redundancy in the annotation (except on the 100-example subset used for IAA evaluation, unless I'm mistaken). I would be curious to know if the authors plan to release (anonymized) annotator information along with each example for researchers interested in modeling this variability.\n\nC. A relatively minor point: the train/validation/test split ratio allocates fewer examples to train than I'm accustomed to seeing (I much more commonly see 80/10/10 or at least 70/15/15; perhaps this is an idiosyncratic feature of my research background). Yet, L252-3 says this ratio is \"standard practice.\" If this is so, it would be good to provide references; if not, it would be good to know how this ratio was decided.", "paper_topic_and_main_contributions": "This paper introduces a novel task, *conditional* semantic textual similarity (C-STS). The well-established semantic textual similarity (STS) task requires systems to assign a scalar value $y$ to a pair of sentences $(s_1,s_2)$ indicating the degree of semantic similarity between $s_1$ and $s_2$. C-STS modifies this task definition by soliciting semantic similarity judgments relative to a particular *condition*, $c$, which expresses some feature or aspect with respect to which the semantic similarity of $s_1$ and $s_2$ should be evaluated. The paper presents a 19K-example dataset of $(s_1,s_2,c,y)$ tuples, where $s_1$ and $s_2$ are image captions drawn from existing image captioning datasets, $c$ is a human-written condition, and $y$ is a human-annotated similarity judgment on $s_1$ and $s_2$ relative to $c$. Evaluations across a variety of different models and settings (zero-shot, few-shot, and fine-tuned) demonstrate that C-STS is challenging even for very recent, top-performing models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None that I am aware of, although I am not very familiar with the STS literature."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos\n- L232: missing space between \"pair\" and \"(Appendix C.1)\"\n- L317: \"for\" -> \"with\"\n- L544: \"corroborates with\" -> \"corroborates\"\n\nPresentation\n- Given that the judgments are ordinal, Kendall's $\\tau$ or Krippendorff's $\\alpha$ seem like more appropriate measures of IAA than Fleiss's $\\kappa$.\n- Section 7 is more of a \"conclusion\" section than a \"discussion\" section; it might be advisable to rename it to the former.\n- The ethics statement appears to be missing.\n- The principle item in the limitations section is that the dataset could be bigger. This is a fair point, but could surely be said of most datasets. It seems like there are potentially additional, more important limitations that others should be aware of (e.g. the apparent skew in the response distribution I raise above)."}}]

Author rebuttal:

Meta-review: This paper presents a novel task and dataset, enhancing the semantic textual similarity (STS) by introducing conditional elements. Even in the era of large-scale language models, predicting textual similarity remain vital, with applications like retrieval-augmented generation. We look forward a refined camera-ready version and updated dataset that takes into account feedback from reviews and discussions with reviewer L1HG.
Decision: Accept-Main