Description-Based Text Similarity
Authors: Shauli Ravfogel
Valentina Pyatkin
Amir David Nissan Cohen
Avshalom Manevich
Yoav Goldberg
Venue: COLM (2024)
Keywords: semantics, similarity, description based similarity, retrieval
TL;DR: We define the notion of description-based similarity, use LLMs to create a tailored dataset for this kind of similarity, and train a corresponding retrieval model.

Abstract: Identifying texts with a given semantics is central for many information seeking scenarios. Similarity search over vector embeddings appear to be central to this ability, yet the similarity reflected in current text embeddings is corpus-driven, and is inconsistent and sub-optimal for many use cases. What, then, is a good notion of similarity for effective retrieval of text?

We identify the need to search for texts based on abstract descriptions of their content, and the corresponding notion of \emph{description based similarity}. We demonstrate the inadequacy of current text embeddings and propose an alternative model that significantly improves when used in standard nearest neighbor search. The model is trained using positive and negative pairs sourced through prompting a LLM, demonstrating how data from LLMs can be used for creating new capabilities not immediately possible using the original model.

Published as a conference paper at COLM 2024
Description-Based Text Similarity
Shauli Ravfogel' Valentina Pyatkin Amir DN Cohen' Avshalom Manevich' Yoav Goldberg!2
BBar-Ilan University Allen Institute for Artificial Intelligence University of Washington Shtauli.ravfoged valpyatkin, amirdnc, avshalomman, yoav hww.ggoldberggee. (gmbbillrr@ggggllle @egmail..com
Abstract
Identifying texts with a given semantics is central for many information seeking scenarios. Similarity search over vector embeddings appear to be central to this ability, yet the similarity reflected in current text embeddings is corpus-driven, and is inconsistent and sub-optimal for many use cases. What, then, is a good notion of similarity for effective retrieval of text?
We identify the need to search for texts based abstract descriptions of their content, and the corresponding notion of description based similarity. We demonstrate the inadequacy of current text embeddings and propose an alternative model that significantly improves when used in standard nearest neighbor search. The model trained using positive and negative pairs sourced through prompting LLM, demonstrating how data from LLMs can be used for creating new capabilities not immediately possible using the original model.
1 Introduction
Searching for texts based on their semantics is important for knowledge seeking agents.
Such agents can be human users, or artificial ones: either LLM-based agents that are tasked with complex goal and need to locate information as sub-goal, or as components in retrieval augmented generation (Khandelwal et al., 2019; Guu et al., 2020; Parisi et al., 2022). Current semantic search solutions are based on dense encoders (Reimers & Gurevych, 2019; Gao et al., 2021) which learn a representation space such that "similar" documents are proximate in space. The notion of similarity in this context, however, is not explicitly defined but rather learned from vast datasets containing pairs of texts labeled as similar, often mixing various different kinds of similarity (Kaster et al., 2021; Opitz & Frank, 2022). This makes them sub-optimal for information seeking queries, as is hard to control or predict the results of given similarity-based query. What is good query representation and similarity definition for semantic-search use case?
In this paper, we suggest a consistent and well-defined relation between texts, which we believe to be a useful one to encode as a vector-similarity metric: the relation between abstract descriptions of sentences, and their instantiations. While LLMs can identify and operate on this relation, we find that the representation spaces that emerge using common text-encoding techniques are sub-optimal for encoding as a similarity metric. We show how to construct better embeddings for this purpose. Using LLMs, we create dataset that captures this specific notion for similarity, and use it to train an encoder whose representation space suppresses tate-oof-the-rt text encoders trained on orders of magnitude more data.
Our focus is in a common kind of information need which is mostly unachievable with current search techniques: retrieving texts based on a description of the content of the text. For example, in the domain of medical research, an agent might want to find sentences discussing the efficacy of specific drug in treating particular condition, such "the effectiveness of drug in managing hypertension", Or they can go more abstract, and look for "substance abuse in animals" "a transfer of dísease between two species". Outside

---

Published as a conference paper at COLM 2024
of the hard sciences, one may want to search the corpus for sentences related to a historical event, such as "an important battle fought during World War II" or "a significant scientific discovery in the field of physics". In international relations research context, the agent may want to scour a corpus for "one country threatening the economy of another country", in a trading context an agent may search for "a transaction involving precious metals", and pop-culture journalism an agent may search twitter for "a fight between two celebrities".
In all these cases, the agent is not interested in a definition or a single answer, but in sentences whose content is a specific instantiation of their query (for example, "The studies have shown that a sub-population of primates chronically consume intoxicating amounts of alcohol" for the "substance abuse in animals" query). In other words, we are interested in higher-order similarity reflecting the "instance-of" property.
Query:: book that influenced the development of ginre
Query: architect designing building
The book now creditd with inventing the ganrn today known as the
The axciitstt Edbiin Forris Durang designed the bu kding
Regarcy Hatorical.
Loon Eugène Ar the chief des grar for the and Rocts Magney
infuential
steampunk dieselourk
Tusle: designed
din..
penses.
design
ruo since become his beer bear-known known work and is considered important in
Local anchiect Fooir Chancler design oree the building.
The development 20th century science
peecursor
Burought'' (1817), which spowned the planetary romance and sword and planot genres
inseration
Princess Mars
The aechttct of the building is unleawn.
The builting win des deiigedd by anchitest
book considered infiuentall its geore Russion
Thes wrlisw the architet.
texts published
following decades,
Quary: company which is part of another company
developting ganree
The work presertss the history of the grene through as discussioo of the hass and works its important earry wetters
suboid ay Hutchipon did
CK Life Sciences Internaionall (Holiings) Inc., or CK Lite Sclonoes,
diveloppmnt that Subgeme Book: Like Gerea Book but focusing narroar sog ment of the full gatre.
Pocten (oompany)) subaldaryy of Sinopeo
bu
L-Hamrapin
EM M croskootroii Marin (ubsldaary of The Swatch Groupt
Query: change of career path
compeny, is company owns 51 %6 more voling stock another fimm for diey))
panent
He then moved a manager carser. Aflewwards. he became en agent fuli-time
When exteipg company sebibiaaees a new company and keepa companies buy minority
He practioe one your before moving Streatr start ustale business.
najorty
is called
company.
He briefly took on a job oo on actuary before embarking into pohor
Holding (in organisation who curs control of small group of other companies)
Otherwie, change profission secessery
decrease, and oducational opoortuniies expand workers may incrasdngly gly find themselves forted to futil the goas of one career and then adopt another here
Tranition that we chen there encing
adulhood the baggert change being the canser they Ultimonlly you have to take a difieent dirctton in your lif, in your caneer
indute
changes
Figure 1: Top retrieval results from the Wikipedia Index. Ours: the model developed in this work. Existing: all all-mpnet mmpnet-base-2,, a strong sentence-similarity encoder.
Such retrieval cannot be easily achieved through keyword-based retrieval, because the retrieved text is more specific than the description, causing very low lexical overlap. It is also not easily achievable by current "dense retrieval" systems that rely on vector similarity: generic sentence similarity methods (Reimers & Gurevych, 2019; Gao et al., 2021) tend to retrieve texts that are similar to the description, rather than instantiations of it (e.g., query like "an architect designing a building" should return a sentence like "The Fallingwater, remarkable architectural masterpiece located in rural southwestern Pennsylvania, was designed by Frank Lloyd Wright""" and not "The architect participates in developing the requirements the client wants in the building.", although the latter more similar under conventional sentence similarity models). Similarly, systems that are trained to retrieve passages that contain answers to questions (trained, for example, on SQuAD (Rajpurkar et al., 2016; 2018)), beyond being focused on questions rather than assertions, are also focused on specifics rather than abstract situations (questions are of the form "when did Columbus discover America" and not "a discovery on new land by an explorer"). Models trained on large data from search query logs may be more diverse, but are generally not available outside of a few large technology companies. We do not find an existing soltuion that fits our goal (Section 6).
We show that retrieval based on description is achievable: given training data consisting of <description, text> pairs, we can train descriptions encoder and a text encoder that learn to represent items such that the descriptions and the texts they describe are close in embedding space (S4). These vector encodings can then be used in standard similarity- based retrieval setting. Figure 1 shows four queries that did not appear in the training data, and their 4 retrieved items, over corpus of almost 10M wikipedia sentences.
2

---

Published as a conference paper at COLM 2024
To obtain the training data ($3), we observe that the reverse direction of the process, going from a text to its description, is a task that can quite easily be performed either by crowd- workers, or,
do in this work, by large language models such GPT-3 (Brown et al., 2020) and Codex (Chen et al., 2021). We thus use the davinci-text-Mi descriptions of sentences sampled from Wikipedia, and use the result as training corpus. Each sentence can accommodate many different descriptions, pertaining to different aspects of the text. We therefore produce five different descriptions for each text, in addition to incorrect descriptions, to be used as negative examples. We find the models trained on this data to excel in both human (Section 5.1) and automatic (Section 5.2) evaluation.
model to generate
Description-based Similarity
General similarity metrics in document retrieval capture broad range of lexical, syntactic, and semantic resemblances and relations, offering a foundational approach to similarity assessment across various tasks. However, their overarching nature often compromises task-specific relevance and precision. In contrast, we propose a specific relation that we want to be reflected by the similarity metric-the abstract description relafion-which explicitly models the relation between high-level descriptive queries and concrete instances within documents.
We start by defining the abstract description relation:
Definition 1 (The Abstract-Descriithee Relation) Given two texts, 1 T and D, we say that (T, D) satisfies the abstract description relation iff.
D describes? (some of) the content of
2) D contains less information and is less specific than T.
Note that this is many-to-mane relation, which is not reflexive, anti-symmetric and non- transitive. The abstract descriptions relation relates to, but is not the same, as other text-based semantic notions such as paraphrases, entailments and summaries. In particular, descriptions are not paraphrases, also more specific than summaries or entailments: while many descriptions are also participating in the entailment and summary relations, not all entailments summaries are abstract descriptions. One notable difference from summaries is that summaries attempt to capture the main events of the text, and do not abstract over it.
paraphrases are symmetric and non lossy. The description relation
Example To illustrate the abstract-description relation, consider the following text and the three valid descriptions of (taken from our dataset, Section 3):
Text: "On July 2, concurrent with the Battle of Gettysburg in neighboring Adams County, Captain Ulric Dahlgren's Federal cavalry patrol galloped into Greencastle's town square, where they surprised and captured several Confederate cavalrymen carrying vital correspondence from Richmond." Description 1: Military personnel thwarting an enemy's attempt to convey vital documents.
Description 2: The disruption of a communication exchange in rural area. Description A dramatic, unexpected event occurring in a town square during battle.
Clearly, the descriptions are highly abstract, contrast to conventional summary of the text; and they omit some key details, such as the country and exact conflict being discussed, or even the fact the event occurred during a battle (description 2); the date; and the specific units being involved. Additionally, the sentence "The town of Greencastle existed during the battle of Gettyburg" is entailed by the text, but does not describe it. See Section 6 for additional discussion of relation to previous work.
In the current paper, both and sentences, but this not part of the definition: T can be londer or shorter than a sentence.
We do not formally define describes and build on the intuitive, English language meaning of the term.
3

---

Published as a conference paper at COLM 2024
Utility We argue that abstract descriptions provide natural and efficient way to express information seeking needs: users can always describe the results they want in natural language. Importantly, these descriptions only need to cover some of the content, not every aspect (as some may be irrelevant to the user). A military historian might want to find dramatic events during battle (description 3) without specifying the time location.
Similarity Existing text encoders struggle with this relation because descriptions have little lexical overlap with the text and they all lack concrete details mentioned in the text. However, if we could create representation space where the text is close to each description, retrieval would be straightforward. Our goal is to learn embedding functions Ey and ED for texts and descriptions such that sim (T)) Ep (D)) correlates with the abstract desciprion relation by being higher for and D for which the relation holds than for all other pairs.
3 Obtaining Training Data
We use GPT-3 (text-davinci-00)) to generate positive and misleading descriptions for sentences from the English Wikipedia dataset." For each sentence, we generate 5 valid descriptions and 5 misleading descriptions, In total, we generate descriptions for 165,960 Wikipedia sentences. See the Appendix for the exact prompts we use.
Generating more abstract descriptions While the descriptions we generate do tend to be abstract, to augment the dataset with descriptions of higher abstraction, we randomly select a subset of instances, re-prompt GPT3 with three of the valid descriptions it generated, and ask it to generate abstract versions of them (this prompt is in-context learning one, the exact prompt appears in Appendix A.1). This results in 69,891 additional descriptions for 23,297 sentences (14.3% of the data). To illustrate the effect of this iterative generation, for the sentence "Civil war resumed, this time between revolutionary armies that had fought in united cause to oust Huerta in 1913-14.", one of the original descriptions generated was A conflict between opposing groups arising from the overthrowing of a political leader", while the iterative query resulted in the more abstract description "Conflict arose between two sides that had previously been allied.".
Final dataset Table 1 shows several examples of the generated data, including the original sentence and pairs of valid and misleading descriptions. The generated data includes a wide range both positive and misleading descriptions that align with the original sentence and the abstract description. The positive descriptions accurately capture the main meaning and key concepts of the sentence, while the misleading descriptions contain inaccuracies irrelevant information. We have randomly divided the data into 158,000 train, 5000 development and 2960 test instances, each composed a sentence, 5 invalid descriptions and 5-8 valid descriptions. We found the quality of the generated descriptions adequate for training, and for measuring progress during iterative development, which we also confirmed through human evaluation. We showed 229 valid descriptions and corresponding sentences to Turkers, asking them to rate on a scale of 4, how well the sentence fits the description. On average the instances were highly rated with a score of 3.69/ /4, which lies between The sentence covers most of the main points mentioned in the description and The sentence covers everything mentioned in the description.
However, some of the descriptions do not adequately capture our intended spirit of abstract descriptions of sentences that reflect an information need. Thus, for the purpose of human- evaluation of quality (Section 5), we manually curate a subset of 201 sentence descriptions from the test set, which we manually verified to reflect a clear information need that would make sense to a human. These were collected by consulting only the descriptions, without the associated sentences they were derived from, or any model prediction based on them.
Phttpss//hugggngffcc.oo/// wwtp:////ttattttaadeeaedeeeeeeee wikipedia
4

---

Published as a conference paper at COLM 2024
Sentence
Good Descriptions
Bad Descriptions
Intercepted by Union gunboats, over 300 of his men succeeded in crossing.
A large group of people overcom-
A group of people being inter- cepted while crossing desert. A neurotransmitter found in the stomach in high concentrations. soccer player selected for tournament in the Phillippines 2021.
ing a challenge.
Dopamine constitutes about 80% of the
A neurotransmitter found in the
catecholamine content in the brain. December 2021, Kammeraad was named Philippines 23-man squad the 2020 AFF Championship held in Singapore.
brain in high concentrations.
A sportsperson's is inclusion a squad for championship.
Around this time, MTV introduced static and single color digital on-screen graphic to be shown during all of its
A visual element was imple- mented to enhance the viewing experience.
MTV's use of dynamic graphic.
programming.
the signing, he quoted as having replied comment by John Hancock that they must all hang together: "Yes, must, indeed, all hang together, or most assuredly we shall all hang sepa- rately".
A historical event where sig- nificant figure made comment about unity.
joke about the consequences not working together.
It was said Democritus's father was from noble family and wealthy that he received Xerxes on his march through Abdera.
A description of wealthy fam- ily's involvement significant event.
description a famous
leader's 's family background.
Heseltine favoured privatisation of state owned industries, a novel idea in 1979 as the Conservatives were initially only propusing denationalise the dustries nationalised by Labour the 1970s
A political party's s plan to reverse a previous government's policy.
The effects of privatisation on the economy.
Table 1: Examples of generated data training data, including the original sentence, the good and bad descriptions
Encoder Training
In order to train our model for the task of aligning sentences with their descriptions, we utilize a pretrained sentence embedding model and fine-tune with contrastive learning. During the training process, represent each sentence and its corresponding valid descrip- tions using two distinct instances of the model: one as a sentence encoder and the other as description encoder.
Let represent a set of sentences, Ps represent the set of valid descriptions associated with sentence s, and N₂ represent the set of negative descriptions for that same sentence We encode each sentence and description via a model, resulting in a vector representation for each token. We
mean pooling over the token vectors of each of the sentence and description pairs to obtain vector representations in Rd. Specifically, we denote the vector representation of a sentence vs, the vector representation of valid description of it as Vp, and the vector representation of negative description as Vn.-
To train the encoder, we combine two loss functions: the triplet loss (Chechik et al., 2010)
and the InfoNCE loss (van den Oord et al., 2018)
The triplet loss, denoted as Liriple(s) (s), is calculated for each sentence as follows:
Σ
max m + ||v, -vpl1 ||vs - vn,l(2)
(1)
(px)
Here, represents the margin parameter that defines the minimum distance between the positive and negative descriptions. We take m = 1. This loss encourages the representation of each sentence to be closer to its valid descriptions than to its invalid descriptions.
The InfoNCE loss, denoted as (I),NC((s), is computed using a random collection of in- batch negatives (i.e., valid descriptions of other sentences in the batch, well sentences that correspond to those descriptions). Let N' represent the set of all in-batch negatives sampled from the valid descriptions of other sentences within the batch, including the

---

Published as a conference paper at COLM 2024
sentences themselves. The InfoNCE loss is given by:
exp V=:Vp
- log
(2)
exp(
R)
+ En exp( V.
Where cosine similarity and t is the temperature (we take T = 0.1).
The final loss used for training is combination of the triplet loss and scaled version of the InfoNCE loss:
Loss(s) = Lrrpple (s) + InfoNCE
We take a 0.1. An ablation study revealed a modest improvement when using the com- bined loss compared to using only the triplet component or only the Info-NCE component (Appendix A.5). We train for 30 epochs with a batch size of 128 and optimize using Adam (Kingma & Ba, 2015).
5 Evaluation
Traditional information retrieval (IR) benchmarks do not align with our focus abstract semantic similarity, matching generalized descriptions with explicit, concrete instances. As such, we construct set of test queries, and quantitatively evaluate our model in two ways. We perform human evaluation on the results retrieved from a large corpus (Section 5.1). Additionally, we perform automatic evaluation on addeesarially-construcce relevant and irrelevant sentences for the test queries (Section 5.2), to test the robustness of our model. We attach the training and test sets, alongside the code, in the supplementary material.
set of
Setting We sample a set of 10 million Wikipedia sentences (in addition to the set used for training and evaluation). We filter sentences shorter than 6 words, leaving a set of 9.55 million sentences. We encode them using the trained sentence encoder, resulting in an index called the Wikipedia Index henceforth. This is the set from which we retrieve in evaluation. Given query 9, we represent with the query encoder and perform exact nearest-neighbor search under cosine distance.
Evaluation set
We chose random set of 201 descriptions from the test set, which we man- person may be interested in. We then
ually verified to be reasonable description-queric performed crowd-sourced evaluation of retrieval based on these descriptions, comparing our abstract-similarity model to each of the baseline models.
Baselines We evaluate our model against strong sentence encoder models based on the MTEB (Muennighoff et al., 2022) leaderboard in the Sentence-Transformee framework (Reimers & Gurevych, 2020),* 4 all-mpnet-base-ve E5-base (Wang et al., 2022), Instructor (Su et al., 2022), GTE-large Li et al. (2023), EmBER-v1 5, BGE-en Xiao et al. (2023), and contriever Tzacard et al. (2021) 6. All models were finetuned by their creators on diverse sentence-similarity datasets, containing orders of magnitude more data than ours. Beyond the
models, our study incorporates additional baselines: BM25, HyDE and a SNLI-based model (Bowman et al., 2015b). BM25 (Robertson et al., 1995) uses term frequency and document length to estimate document's relevance specific query. BM25 has been shown to be a strong baseline for document retrieval (Izacard et al., 2022). HyDE (Gao et al., 2022) zero-shot model using GPT-3 to generate synthetic documents for a given query. The dense representations of these documents are averaged and fed as query to pretrained document retriever? Due to the similarity between descriptions and
#nttps://hugginngaac wtwaaacee...eeeeeeeeeeeeeeeeeeee /eteeeeeeeeeeeesstieeee Phttps://huggingffae
6GTF, EmBER and BGE the state-of-the-art in the time of the writing as per the benchmark, while the other models are highly popular.
7 Note that HyDE is different than our model and the other baselines in the sense that it calls the GPT-3 API once per query at inference time.

---

Published as a conference paper at COLM 2024
entailments, we also finetune a MPnet-based model for retrieval on the SNLI dataset. See Appendix A.2 for details on our baselines.
Our model, denoted Abstract-sim, is fine-tuned version of the pretrained MPnet model (Song et al., 2020). We do not use all-mpnet-base base-v,, which was further finetuned on simi- larity datasets, it yielded worse results in preliminary experiments. Fig. 1 shows the top results of four queries, alongside the top results from all-mpnet-banl -asttt-baeeeee for comparison.
5.1 Human Evaluation
We perform human evaluation over naturally occurring sentences, in a natural retrieval scenario, where abstract descriptions are likely to be used as queries. The human evaluation compares the top sentences retrieved with our method, and to the top sentences retrieved with the state-of-the-ar semantic sentence encoding models..
The evaluation setup is structured as follows.' 9 Crowdworkers are shown a query and results from search over the Wikipedia Index. Particularly, they are shown 10 sentences, 5 of which are the top-5 retrieved sentences from abstract-si and 5 of which are the top-5 retrieved sentences from one of the baseline (each experiment with another baseline). The 10 sentences are randomly shuffled, and crowdworkers are then asked to select all sentences that they deem a reasonable fit for the query. Each task is shown to three distinct annotators. We aimed at paying cworkrss $15 per hour on average. Each query instance is shown to 3 annotators.
Metrics We report the average number of results from each model that were selected as relevant (as a histogram), as well as the mean number of times a specific number of sentences from a given model was chosen (the mean of the histogram).
Abstract-sim (Ours)
MPNet
Model
# chosen
E5
HyDe
abstract-sim
3.89+0.073 55
Counts
Instructor
HyDE
2.2 / 5
1.89 /
Instructor-large
1.64 /
E5-base
1.61 /
Table 2: Human evaluation results (Sec- tion 5.1): number of sentences that crowd- workers deemed to be fitting the query, from set of retrieved sentences: Our model (abstract-sim) vs. the four baselines. The num- ber reported for abstract-sine is mean+std over the binary comparisons against each of the 4 baselines.
Number sentences chosen
Figure 2: Human evaluation results (Sec- tion 5.1): number of times a given num- ber of sentences was chosen per query instance: Our model (abstract-sim), aver- aged over all baseline evaluations, vs. the baselines.
Results For evaluation we only count sentences to have been selected as relevant, they were chosen by at least out of 3 annotators. In Table 2 we show the average number of valid retrieved sentences per method. The annotators have chosen significantly more sentences from
abstract-sim model compared to all 4 baselines, with our model having close to out of 5 sentences deemed as fittingg the query on average and the baseline models between 1.61-2.2 sentences. Fig. shows the complete distribution of the number of times given number of sentences was chosen from a given model (where the maximum is 5, that is,
We do not compare against NLI and BM-25 due to their very low precision and recall in the automatic evaluation (Section 5.2; Appendix A.4) and Fig. 4. Additionally, due to budget constraints, we only compare against the top-3 dense retreieval as per the automatic evaluation: E5, MPnet and Instructor.
Screenshots of the annotation interface can be found in the Appendix.
7

---

Published as a conference paper at COLM 2024
all the 5 results for the model were chosen). Notably, in 99/201 of the test cases, 5 sentences were chosen from abstract-sim's results; from the baselines all 5 sentence were only chosen between 14-28 times. That is, in many of the cases all top results were considered as relevant for the query. Conversely, the baselines show a large number of cases where only 0,1, or 2 sentences where chosen, while these cases (below 5 vs, at least 42 for the case of O relevant sentences). Overall, human inspection of top-retrieved results show large preference for our models compared with the baselines.
much rarer among abstract- sim results
5.2 Automatic Evaluation
We accompany the human evaluation with a manually-conssrrutee automatic evaluation dataset, focused on robustness to misleading results. We do not know how many relevant sentences exist in the Wikipedia index for each query (if any). To allow for an automatic evaluation in the face of this challenge, we use the following evaluation scheme. We used GPT to generate a set of valid sentences per description. To test robustness, we work under an adversarial setting, where for each query we generate both relevant sentences and distracting sentences. We measure the precision and recall of our model and the baselines mentioned above.
Generating sentences from descriptions We start with the 201 valid, manually-verified descriptions in the test set. We use GPT for the reverse task of our main task: mapping abstract descriptions to concrete sentences. We randomly choose one negative (invalid) description from the entry in the test set that corresponds to each valid description. We manually verify that the chosen description indeed topically similar but invalid. In case the description does not contradict the valid description, we manually change it. The process results in a complementary set of 201 invalid abstract descriptions. For example, for the valid test example "The existence of a river and town with the same name", we have the invalid description "The existence of a river and a county with the same name". generate set of 12 sentences that match
For both the valid and invalid descriptions, the given descriptions, ending up with 12 sentences that align with a description, and 12 sentences that align with contradicting description, that serves as distractor. These 24 sentences were then combined with the remaining 9.55 million sentences in the Wikipedia Index. The prompt used to generate these sentences is available in Appendix A.1.1. We use Mturk to verify the validity of the resulting set of sentences." The process results in an average of 11.2 valid sentences and 9.3 invalid sentences per test query. See Appendix A.3 for a sample.
Setting We follow 3 metrics: valid-recall@k, invalid-recall@. and precision@k. valid-recall@n measures the number of valid sentences captured within the first k retrieval results over the Wikipedia index. Similarly, irvalid-rreclllo measures the number of invalid sentences captured. Finally, precision@k calculated only with respect valid and invalid sentences (excluding the Wikipedia index, which might contain many more valid sentences): we calculate the similaríty of the description to the valid and invalid sentences, and count the number of valid sentences within the top k results.
The precision results shown in Appendix A.4 and the recall results are shown in Appendix A.5. Our models improves over all baselines in terms of precision@k. The gap is largest for precision@1, and gradually decreases. Our model achieves precision@1 = =8..4%, compared with 73.6% for the strongest baseline, E5, corresponding to 31 31/201 vs. 53/201 errors in the highest ranked result, respectively. The gap decreases with increasing (note that we have a maximum of 12 positive examples). As for recall, generally models that achieve high valid-recall@k also achieve high invalid-recallk.. Our model achieves relatively low valid- reekl but is better than all models in terms of invalid-recall@k (except SNLI, which has both low valid recall and low invalid recall), i.e., tends to avoid returning invalid sentences, at the price of missing some valid ones.
Results
For the set of valid sentences we filter out all sentences chosen as fitting the description by at least two annotators, For the set of invalid sentences we take all sentences chosen as not to be suitable fit for the description by at least two annotators.
8

---

Published as a conference paper at COLM 2024
1.0
0.9
0.8
0.7
Precision@k
0.6
Nfnat
0.5
Instuctor
0.4
HyDE
0.3
Abetmacssi (ound
0.2
2
10
12
Figure 3: Precision automatic evaluation results (Section 5.2): precision@k curve for abstract-sim and the baselines. Vertical lines represents 1 standard deviation.
Description-based Similarity vs. Previous Work
We compare description-based similarity with popular existing similarity-based retrieval methods, as well as the related NLP task of recognizing textual entailment (Dagan et al., 2005; Bowman al., 2015a).
Vs. Keyword-based Retrieval: Keyword-based retrieval methods rely on exact lexical matches, which makes them inherently weak for retrieval based on abstract descriptions. These methods require users to construct queries using specific keywords, resulting in laborious and suboptimal process. For example, to retrieve sentences related to "animals," " would need come up with exhaustive list of animal names, which impractical and may lead to incomplete results. Consequently, keyword-bar -baed retrieval is ill-suited for retrieving sentences based on abstract descriptions.
Vs. Dense Similarity Retrieval: This family of methods, exemplified by SBERT (Reimers objective that encourages sentences with "similar meaning" to have high similarity. Similar meaning, here, is determined by multiple сorpora such as Reddit comments (Henderson et al., 2019), SentEval (Conneau & Kiela, 2018) and SNLI (Bowman et al., 2015a). As such, the type of similarity captured by such models in practice emerges from the training corpus (Kaster al., 2021; Öpitz & Frank, 2022) and is not well understood. Our goal is similarity metric for the specific type of relation we define, between abstract descriptions and concrete instantiations of them. Moreover, while sentence similarity models aim to cluster sentences with similar meaning together, description does not have "similar meaning" to the text it describes, but rather other descriptions of the same text.
& Gurevych, 2019) encodes sentences based
Vs. QA-trained Dense Retrieval: These systems are trained to retrieve paragraphs based on a question, an open-QA setting (Karpukhin et al., 2020) The retrieved paragraphs are then run through reader component, which attempts to extract the answer from each retrieved paragraph. The training objective is encode paragraphs to be similar the questions to which they contain an answer. Question could be seen as similar to descriptions (e.g. "early albums of metal bands" can be served by retrieving for "which metal bands released an early album""), but they also differ in that: (a) it is often cumbersome for user to rephrase the information need as a question-in the above example, the move to question form not trivial; (b) questions the question, rather then on situation involving a relation or interaction between several entities; (c) the kinds of questions in current QA training sets tend to ask about specific,
often focused on single entity that is the answer to

---

Published as a conference paper at COLM 2024
rather than abstract, cases, e.g. asking "which metal band released album Painkiller?" or "what is the first album by Metallica?",
Moreover, in many cases that translation of descriptions to questions is altogether impossible. Often, there is no single question whose answer accurately fulfills the information need that can be expressed by simple description. Consider user interested in movie scripts where "A character is being rescued by another character". Formulating this abstract description is easy. On the other hand, while it is possible formulate several questions that resemble that description, such as "In what sefting is one character being rescued by another" or "What positive help does one character give another character?" none of them accurately captures the intent of the original description.
Vs Query-trained Dense Retrieval: These systems are trained on a collection of <query, document> pairs, which typically obtained from search engine logs." In sense, these subsume the description-retries not on sentences; (b) not focused on this task, may retrieve also results which not descriptions; and, most importantly (c) are mostly based on proprietary data that is only available within a handful of large companies.
task, but are (a) focused on documents and
Vs. Entailment/ / NLI Edescription, text> pairs adhere to the entailment relation be- tween positive <hypothesis, text> pairs the Textual Inference task (Dagan et al., 2005; Bowman et al., 2015a), which is a superset of the <description, text> relation. In theory, NLI based similarity models could perform well on this task. However, in practice they do not perform well, possibly due to the composition of existing NLI datasets. Additionally, the do not usually encode the hypothesis and the premise independently, making efficient indexing difficult.
Conclusions
We introduce the task of sentence retrieval based on abstract descriptions. We show that current centence-embedee to generate set of diverse valid and invalid abstract descriptions, and train retrieval model on that resulting data. We find that the model trained on the data that is tailored to this task is performing significantly better than standard sentence-similarity models. This disparity highlights that the notion of similarity captured by sentence transformers is vague, and that tailoring it to specific information seeking need may result in significant practical improvements.
methods are not good fit for the task. We leverage GPT-3
Acknowledgements
This project received funding from the European Research Council (ERC) under the Euro- pean Union's Horizon 2020 research and innovation program, grant agreement No. 802774 (iEXTRACT). Shauli Ravfogel is grateful to be supported by the Bloomberg Data Science Ph.D Fellowship.
Ethics Statement
As all language technology, the models and data are inherently dual use-they can be used both for good (e.g., . to advance human knowledge) or for bad (e.g., for surveillance that is aimed at depression of minority communities). We hope that the benefits outweighs the risks in our case.
In the context of academic research, the focus is on the MSMARCO dataset (Bajaj et al., 2016), which contains natural language questions extracted from query logs. However, query logs include many different query types beyond questions, and modern search systems have been reported to incorporate such embedding based results for general queries. For these, see "Vs. QA-trained dense retrieval" above.
10

---

Published as a conference paper at COLM 2024
According to the terms-of-servic of the GPT API, the API output (the collected data and the models we created based on it) should not be used to compete with OpenAL. We declare we have no such intentions, and ask the users of the data and models to also refrain from doing so.
References
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Bhaskar Mitra Andrew McNamara, Mir Rosenberg Tri Nguyen, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In InCoCo@NIPS, 2016.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, 2015a.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Conference on Empirical Methods in Natural Language Processing, 2015b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11(3), 2010.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. ar Xiv preprint ar xxXiv:2107.03374, 2021.
Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual en- tailment challenge. In Proceedings of the First international conference on Machine Learning Challenges: evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, pp- 177-190, 2005.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels, 2022.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, 2021.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training, In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Matthew Henderson, Pawel Budzianowski, Inigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrkšic, Georgios Spithourakis, Pei-Hao Su, Ivan Vulic, et al. A repository of conversational datasets. ACL 2019, pp. 1, 2019.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Towards unsupervised dense information retrieval with contrastive learning. Xiv preprint ar Xixx:21122099188 2(3), 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Unsupervised dense information retrieval with con- trastive learning, 2022.
11

---

Published as a conference paper at COLM 2024
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question an- swering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7766967817 Online, November 2020. Association for Computa- tional Linguistics. doi: 10.18653 /v1 /2000eemmlp-main.55000 URL https::/acclaattee org/2020. emnlp-main momppmmain.5555 550.
Marvin Kaster, Wei Zhao, and Steffen Eger. Global explainability of bert-based evaluation metrics by disentangling along linguistic factors. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9912-8925, 2021.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor language models. ar preprint 17Xiv:1911.00172, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://axiivv.ogr/b//////// 14t.s/1111.66.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. ar preprint arXiv:2308.03281 2023.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint ar Xrit:2210007316, 2022.
Juri Opitz and Anette Frank. Sbert studies meaning representations: Decomposing sentence embeddings into explainable semantic features. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, Pp. 625 625-638, 2022.
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. Xiv preprint ar Xrii:22205.12255. 2022.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016
Pranav Rajpurkar, Robin Jia, and Percy Liang, Know what you don't know: Unanswer- able questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992, 2019.
Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2020. URL https://arxiv.org// wwt.:oog//bbb///////// 2004..220000099 09813.
Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. Okapi at trec-3. In Overview of the Third Text REtrieval Conference (TREC-3), pp. 109-126. Gaithers- burg, MD: NIST, January 1995. URL https: https://www.micrrsst/ publication//okpii-a-rr-----
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33:16857-16867, 2020.
12

---

Published as a conference paper at COLM 2024
Hongjin Su, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, Tao Yu, et al. One embedder, any task: destruction-finetueed text embeddings. arXio preprint ar Xfio:2212.097411, 2022.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR,
2018. URL http:////axx wwt.w/o/////////// tabbs//87.. 0071807774550
Liang Wang. Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. Xiv preprint a<Xo::221..03533, 2022.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged re- sources to advance general chinese embedding, 2023.
13

---

Published as a conference paper at COLM 2024
A Appendix
Limitations
Our training data, models and experiment are all strictly English-based, More importantly, we observed the following limitation of the resulting similarity model. While it clearly is better than all existing models we compared against at identifying sentences given an abstract description, we also observed the opposite tendency: for some queries, faithful to the provided description. For example, searching for the query "The debut novel of a french author" returns results such as "Eugénie Grandet is a novel first published in 1833 by French author Honoré de Balzac" Houellebecq", either mentioning the first time the novel was published, instead of returning mentions of first novel published by author; or mentioning novels written by French authors, regardless of whether or not they are their debut novels.
not
"Lanzarote (novel), novel by Michel
A.1 Prompting
These are the prompts used to generate the sentence descriptions dataset. The "main prompt" was used to generate 5 valid descriptions and invalid descriptions per sentence. For approximately 14% of the sentences, we re-feed GPT with one of its valid generations and use the "Make-more-abstract prompt" to generate additional more abstract version of the descriptions. Finally, we the "Description sentence prompt" to generate set of sentences that align with the 201 test descriptions, used for evaluation.
Main prompt:
Let's write abstract descriptions of sentences. Example:
Sentence: Pilate 's role in the events leading to the crucifixion lent themselver to melodrama even tragedy and Pilate often has a role in medieval mystery plays
Description:
description of historical religious figure's s involvement significant event and its later portrayal in art.
Note: Descriptions can differ in the level of abstraction, granularity and the part of the sentence they focus on. Some descriptions neeed to be abstract, while others should be concrete and detailed.
For the following sentence, write up 5 good and stand-alone, independent descriptions and 5 bad descriptions (which may be related, but are clearly wrong). Output json file with keys 'good', 'bad'.
Sentence: (sentence)
Start your answer with curly bracket.
A.1.1 Make-more-abstract Prompt
Sentence: in spite of excellent pediatric health care several educational problems could be noted in this tertiary pediatric center
Description: Despite having advanced healthcare resources, certain deficiencies in education were identified at medical center that serves children.
A very abstract description: The provision of care at specialized medical center was not optimal one particular area, despite the presence of advanced resources.
14

---

Published as a conference paper at COLM 2024
Sentence:
(sentence)
Description: (description)
A very abstract description:
Description to sentence prompt
Create a JSON output with key 'sentences' containing 15 Wikipedia-style different sentences. The sentences should align with the given description, i.e., the description must be a valid characterization of the sentences. Notice: (1) You must avoid using words appearing in the description; (2) You MUST mention concrete entities such as names of people, places and events make the sentence sound natural; (3) you MUST make sure each sentence is relevant for the description; (4) IMPORTANT: you MUST make the sentences different from each other; they must not mention the same topics. Description: "(description)"
Be faithful to the description. Start your answer with curly bracket.
A.2 Baseline Models
HyDE We adapted HyDE to our scenario by: adding an appropriate prompt for sentence generation matching the description in the query and b. replacing the document retriever with sentence retriever hall-mpnet-i (oppnet-bssev2),
Instructor Instructor generates task-specific embedidngs by specifying the type of task in the prompt. We use the recommended prompt "Represent the Wikipedia document for retrieval" for the sentence encoder, and the closest prompt from Su et al. (2022)'s 's dataset, "Represent the Wikipedia summary for retrieving relevant passages:", for the description encoder; variations on the query prompt, such as "Represent the Wikipedia description for retrieving relevant passages:", yield similar results.
SNLI baseline The notion of description-based similarity is related to NLP task of recog- nizing textual entailment (Dagan et al., 2005; Bowman et al., 2015a) (see below in Section 6). As such, it is natural ask how do models trained on popular RTE datasets, such SNLI (Bowman et al., 2015b), fare on this task. We extract entailment and neutral pairs from the SNLI dataset, and finetune an MPnet-base model for 30 epochs with the objective of minimizing the InfoNCE loss Eq. (2), where hypothesis is the query, the negative pairs are taken from neutral premises while the positive the entailing premise. We then evaluate this model in the same way we evaluate the other baselines.
15

---

Published as a conference paper at COLM 2024
A.3 Automated evaluation data
Doseipten
Valid setilencc
Lovald senieax
A pasiod of dificully and somon for
The death his beloved mother was an extraticly ditlicult and somowiul time for Alber.
This individual building was diffkultt place live in.
individual.
A ahit in the way people are referred
In the current er3, more and more people are proferting by their given name, rather than tracitional tities.
Alice was reterred to ao iittes the same way she used to be in the qre-qaaaanine period.
to has occurred.
The honoring of an actor's legacy.
On 18 April 2000, cerememy was the TCheeCLChhame Chinese The- atre in Hollywood to commemo- rate
Thespian from all over the no- tion gathered Los Angeles to recogniae the immense infle- ence veteran director Stan
actor Peter TToole, who passed away 2013.
The act two individus's reaching a muthal inderstanding
The leaders different decided to aside their differemon and mech peaceful understanding.
Thrre school friends, Jack, Rachel, finally rencher mutual egrement over they' d
rib.
concept by
hogaly Allhooagh strongly possibility perpetual
scholar.
searcher
champi-
grand jury's investigation
grand jury into the devaststing floods that occurred acroes the nation.
political corruption
launched investigation scandal incolving promiment figures the govermmntt
polincal
He The offect of decrous the
The deelining taond the mumn- ber predators hus coused vere depletion in the paa lation.
Predeturs have ivolied lime, playing critical cole eodogy,
ber of poudators.
cillerent niches and competing with each other..
Table 3: Examples of generated training data, including the original sentence, the good and bad descriptions
Table 3 presents sample of descriptions from the 201 examples test set, alongside one invalid and one invalid sentence (generated by GPT3) per description. These were used in the automatic evaluation (Section 5.2).
16

---

Published as a conference paper at COLM 2024
A.4 Recall Results
1.0
1.0
Validrecaligee
Asrrebore rour
0.0
0.6
0.5
0.5
Craer
0.4
0.4
0.0
300
1000
1500
2000
500
1000
15000
2000
Figure 4: Recall automatic evaluation results (Section 5.2): valid-recall@k (left, higher is better) and rnvalid-recall@k (right, lower better) for abstract-sim and the baselines. Vertical lines represent 1 standard deviation.
Figure Fig. 4 presents valid-recall@k (higher is better) and invalid-recall@k (lower is better) for the automatic evaluation experiment (Section 5.2).
A.5 Ablation of Loss Components
0.7
0.7
Valddnecal@et
0.6
0.6
0.5
0.5
0.4
0.2
0.1
0.0
0.0
500
1000
1500
2000
10
12
Figure 5: Ablation results on the automatic evaluation (Section 5.2).
Fig. 5 presents the results of automatic evaluation when training models with the individual loss components (only the triplet loss, only the info-NCE loss) compared with using the combination of the two losses.
17

---

Published as a conference paper at COLM 2024
A.6 Human-evaluation Interface
This is the interface used for MTurk evaluation:
Instructions (clikk to expand/ccllepse)
Thanks for participating in this HIT! Please read the instructions carefully.
type sentence looking for Imagine the descript on something like a search query for search machine. The Sentences the result we obtained after searching for sentences that fit the description. allowed to contain additional information, not mentioned in the Description, as lang as covers what has been requested in the Description.
you shown Description
Description
you consider good
QueryiDescription.
submit responses that uninformed instructions.
Descriation:
$(desription1)
retrieved
description
4(sentence1)
4(sentence2)
((sentence3)
R(sentenced)
4(sentence5)
((sentence6)
S(sentence7)
4(sentence8)
((sentence9)
#benteence10
(Optional) if anything was you
experianced any issues, or you have any other fedback for us.
18

---

Published as a conference paper at COLM 2024
This is the interface with an instantiated descriptions and 10 retrieved sentences (5 from baselines and from our model, presented in random order).
Description:
A period of difficulty and sorrow for an individual.
Choose retrieved sentences that description
For some years in the period 1945-1974 there was an individual college championship.
was period great political difficulty Italy.
personal this time was filled with tragedy.
An sorrow;
Gauss plunged into depression from which he never fully recovered.
Attwater suffered for several days afterwards, though.
The time of suffering and illness
During this time, however, Charlesfort had fallen into despair.
An wed your sons sarrow; diffcculty properly assessing value individual gern quality diamond complicates situation.
This the interface we used for assessing the coverage of the GPT3 generated description and its corresponding sentence.
Description:
sidesorpttont
Retrieved
$:5entence1)
Coverage How well does the retrieved sentence the description? (Noti: The desaripians should capsure some aspect of the sensenoe,, but they donft need to fully describe all the lacers the sentencer Le. the sentence allowwd contain soditional informtion not mensioned the description and should not be panalized It)
1. The retrieved sentence hot relevvant all with respect the description.
2. The retrieved sentence contains minor elements mentioned the desoription.
3. The retrieved sencence cevers come of the points mentloned the desoription.
A. The retrieved sentence covers jostt of the main points mendoned the desoription.
5. The retrieved sentence cevers everyyting mentioned the description.
19


Reviews:
[{"review_id": "b4pLB0o0mb", "reviewer": "Reviewer_88m7", "summary": "The paper tackles the question of what would be a good similarity measure for effective text retrieval, suggesting that the similarity induced from word embeddings is corpus driven (trained on sentence pairs labeled as similar according to some notion of similarity, which is not always consistent) and not optimal for many use-cases.\n\nThe authors propose a new model that works better then current embeddings in nearest neighbor search. Such a model is trained using LLMs to annotate/describe pieces of text and generate positive and negative examples.\n\nThe key idea is to obtain an abstract description of a sentence, where the sentence participates in an instance-of relation with its description.\n\nCurrent dense retrieval methods tend to retrieve texts that are similar to the description, so a model that is specialized on retrieving instantiation of the description is needed.\n\nThe authors train an encoder model that maps <description, text> pairs close in the embedding space. The description of a piece of text is obtained with the davinci-text-03 model. The model produces different descriptions, and some that are incorrect are used as a negative example.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and thoughtful feedback. \n\n***Specialization and Fairness***: While our model is indeed specialized for the task of retrieving text by abstract description, this specialization underscores a critical point: existing models, trained on broader and often less specific definitions of similarity (I.e., many different similarity datasets), fail to capture this notion, as demonstrated in our experimental evaluation. The in-distribution test data was necessary to fairly evaluate the capability of our model in this specific context. Out main contribution is not the training of the specific model (indeed, it is very likely that better models can be found), but rather in advocating for this specific notion of similarity and showing that current models fall short in acquiring it. Please note that the test set was sourced from GPT-generated queries, but was further manually evaluated and refined by the authors. \n\n***Tying encoders weights***: We have tried that in preliminary experiments, but this resulted in somewhat decreased performance. Please note that in our setting, the queries and the results are very different in their surface form: queries are short and highly abstract, while results are concrete and detailed\n\n***Color Scheme in Figure 2***: We apologize for the oversight in our color selection. We will revise Figure 2 to use a more distinct and accessible color scheme.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of using LLMs to create special kinds of queries related to a piece of content and then train a specialized retrieval model on such data is interesting and it has a lot of potential.", "reasons_to_reject": "I think the ways to evaluate the proposed approach are tricky and the current system definitely has some shortcomings, one of them being too narrow and favorable to the proposed fine-tuned model.\n\nThe authors are defining a task that is focused on a subset of all possible search queries, they are extracting features (descriptions) of sentences via an automatic process (LLM), training a model on such data and testing using descriptions that are automatically generated too.\n\nTherefore, the model is by definition specialized for this task (and the other baselines are not) and in addition, test descriptions are in-distribution. Probably a manually designed set would have been a little better.\n\nMaybe expanding the type of queries and showing an approach that can improve over some types of queries without hurting the performance of the others would be more interesting.", "questions_to_authors": "Encoders for descriptions and sentences have separate weights because of the asymmetry of the task. Have you tried tying the weights?\n\nSuggestions: in Figure 2, the colors used for the Abstract-sim and Instructor bars are too similar and difficult to separate for some people. Please use a different color scheme.\n\nTypos: soltuion, desciprion", "extra_scores": {}}, {"review_id": "LCryRhiXAE", "reviewer": "Reviewer_Utoh", "summary": "This paper identifies the need to search for texts based on abstract descriptions of their content, and the corresponding notion of description-based similarity. An embedding model combining the triplet loss and the InfoNCE loss is proposed, and it can significantly improve the standard nearest neighbor search.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback. \n\n***Significance of Description-Based Similarity Task***: We envision the task of retrieving texts by their description to be directly useful on its own for many end-uses, in particular information seeking by experts in large document collections. This is true regardless of any relevance for an existing NLP end-task.  A second-order user-study can be useful, but is not in the scope of the current work.\n\nIn the introduction section, we discuss the use cases of our proposed approach in information-seeking scenarios. Its Usefulness is particularly highlighted in scenarios where experts aim to locate domain-specific relevant text in large corpora. For instance, a legal researcher can inquire about \"precedents for intellectual property disputes\". Natural language descriptions provide an effective way for domain experts to search over large corpora of natural language texts.\n\n***Fair Comparison with Fine-Tuned Baselines***: We focus on drawing attention to the need to retrieve by description, and showcasing the limitations of existing encoders in this task. We find it essential to test models on our data to showcase the limitations of SOTA general-purpose encoders in capturing the notion of similarity we argue is useful. Their failure is a nontrivial finding, as these models were trained on orders of magnitude more data. It is not possible to highlight this inherent limitation of existing models without evaluating them on data tailored for our notion of similarity. \n\n***InfoNCE Temperature Setting***: The choice of the temperature parameter in InfoNCE was based on preliminary experiments where a value of 0.1 yielded better performance for our specific task. We acknowledge the need for an ablation study to support this choice and will include such a study in the revised manuscript to provide a detailed justification.\n\n***Choice of GPT-3 over ChatGPT***:  Please note that we used text-davinci-003, an instruction-tuned version. We found that it worked better than ChatGPT, which generated highly verbose responses. However, the LM choice is pretty much orthogonal to our approach. \n\n***Dataset Statistics***: We will present detailed statistics of our dataset, including train/dev/test set sizes and the number of positive and negative examples, in a table for better clarity.\n\n***Missing References***: We appreciate the pointers to relevant literature. We will include the suggested references on LLM-based synthetic data generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tThis paper presents a new type of semantic textual similarity task: description-based text similarity. This new type of similarity benefits retrieval-augmented generation.\n•\tThis paper proposes a description-based text similarity dataset. This dataset is helpful for this research area.\n•\tBoth the human and auto evaluations suggest the effectiveness of the proposed model.", "reasons_to_reject": "•\tThe experiment is inadequate. The experiments focus on comparing different models' performance. However, there is no experiment to support the significance of the proposed new type of similarity task. It would be appreciated if you could provide some experiments on downstream tasks to quantitatively demonstrate the importance of the proposed description-based text similarity task. \n•\tThe experiment might be unfair. It is better to also compare the proposed model with baselines fine-tuned on the collected dataset.", "questions_to_authors": "•\tTo my knowledge, InfoNCE usually sets $\\tau$ to 0.05, but this paper sets it to 0.1. I haven't seen an ablation study to support this setting. Could you explain this further?\n•\tWhy do you use GPT-3 instead of more powerful LLMs like ChatGPT to generate datasets?\n\nIn addition, I put some suggestions and typo correction comments as follows.\n\nSuggestions\n•\tIt is better to provide an explanation for the hyperparameter selection.\n•\tIt is better to present the statistics, such as train/dev/test set size, positive and negative size, of the proposed data in a table.\n•\tThere might be some missing references in the section on obtaining training data. LLM-based synthetic data for text similarity or information retrieval have been explored [1][2]. It is better to also acknowledge their contributions.\n[1] Li X, Li J. Angle-optimized text embeddings[J]. arXiv preprint arXiv:2309.12871, 2023.\n[2] Wang L, Yang N, Huang X, et al. Improving text embeddings with large language models[J]. arXiv preprint arXiv:2401.00368, 2023.\n\nTypos:\nPage 3: iff -> if (in definition 1)\nPage 3: londer -> longer (in footnote 1)\nPage 14: ?? missing reference?", "extra_scores": {}}, {"review_id": "u8Mdcgk55M", "reviewer": "Reviewer_9ziQ", "summary": "This paper presents a new variant of information retrieval where the query is an abstract description and the goal is to return specific instances that match the description. For example, if the description is \"A neurotransmitter found in the brain in high concentrations.\" then an acceptable result will be the sentence \"Dopamine constitutes about 80% of the catecholamine content in the brain.\".\n\nThe method in the paper is to train special sentence encoders using a special dataset of triples of acceptable result, good description and bad description. LLMs (GPT-3) are used to generate the descriptions and a dual-encoder method to embed the abstract description which is the query, and the document using two separate models. The loss function was empirically determined to be a combination of triplet loss and info-nce loss.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback and the opportunity to address the concerns raised. Our task of retrieving text based on abstract descriptions introduces a new notion of similarity. It has no relation to TREC's long query tasks.\n\nUnlike TREC's long query tasks, which generally focus on detailed and verbose queries directly related to the content, our work addresses the challenge of aligning abstract, high-level descriptions with specific instances. This requires a different approach and demonstrates significant novelty in the context of information retrieval. We do not understand in what sense this is “incremental”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a variant of the fairly standard task of information retrieval given a long query. The paper is well written and readable and the results are plausible. This work critically relies on LLMs for bootstrapping its dataset and it might be helpful to practitioners working on this problem.", "reasons_to_reject": "The paper lacks technical depth and the methods and results are not very insightful. The task presented in this paper is claimed to be novel but it is quite similar to the long query retrieval tasks at TREC. Also the subject of the paper is only tangentially related to language modeling. The main use of LLMs in this paper is to generate the training dataset.\n\nOverall the proposed method does not seem to be very novel or technically significant and has only a little incremental value. Therefore I will not recommend to accept this paper for publication at COLM.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eJsGGBmpP2", "reviewer": "Reviewer_i7FB", "summary": "This paper argues that current embedding similarity approaches are inadequate for retrieval. Sentence embeddings do not necessarily abstract over the meaning of a text and hence during retrieval we cannot simply use similarity metrics over the actual embeddings of the text. We instead need to develop some summary / description / abstraction of this text and use this to guide our search. The authors construct a data set to test this hypothesis.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and recognition of the novelty of our work.\n\n***Dataset Size and Practical Implications***: We acknowledge that the current dataset may not be large enough to fully assess the practical implications of our approach in broader document retrieval scenarios, however, we put emphasis on the quality of the queries used in the evaluation, as we manually verified each of them, and edited them when it was necessary. While the dataset, consisting of 200 queries, is not very long, it is still long enough to showcase the limitations of existing approaches to the problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think that the paper merits acceptance. The hypothesis of the paper is intuitive, and there is enough novelty (including the construction of a data set) to support publication.", "reasons_to_reject": "The main flaw of the paper is in the assessment of the method. The data set is not large enough to assess how their approach could assist in more accurate document retrieval in more practical terms. Nevertheless, I understand that constructing such a data set is not an easy endeavour.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers generally like the idea proposed in this paper, which leverages LLMs to create new data for training a specialized retrieval model. Most of the criticisms center on the evaluation, including the size of the dataset, the impact on downstream tasks, etc. The authors should address these feedback in the next version of this paper properly.
Decision: Accept