Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness
Authors: Xinran Zhao
Tong Chen
Sihao Chen
Hongming Zhang
Tongshuang Wu
Venue: COLM (2024)
Keywords: retrieval, perspective, text embedding, retrieval-augmented generation
TL;DR: we propose a novel retrieval benchmark, PIR, to study if and how current retrievers can handle nuanced perspective changes in user queries from real-world scenarios.

Abstract: The task of Information Retrieval (IR) requires a system to identify relevant documents based on users' information needs. In real-world scenarios, retrievers are expected to not only rely on the semantic relevance between the documents and the queries but also recognize the nuanced intents or perspectives behind a user query. For example, when asked to verify a claim, a retrieval system is expected to identify evidence from both supporting vs. contradicting perspectives, for the downstream system to make a fair judgment call.
In this work, we study whether retrievers can recognize and respond to different perspectives of the queries --- beyond finding relevant documents for a claim, can retrievers distinguish supporting vs. opposing documents? We reform and extend six existing tasks to create a benchmark for retrieval, where we have diverse perspectives described in free-form text, besides root, neutral queries. We show that current retrievers covered in our experiments have limited awareness of subtly different perspectives in queries and can also be biased toward certain perspectives. Motivated by the observation, we further explore the potential to leverage geometric features of retriever representation space to improve the perspective awareness of retrievers in a zero-shot manner. We demonstrate the efficiency and effectiveness of our projection-based methods on the same set of tasks. Further analysis also shows how perspective awareness improves performance on various downstream tasks, with 4.2% higher accuracy on AmbigQA and 29.9% more correlation with designated viewpoints on essay writing, compared to non-perspective-aware baselines.

Published as a conference paper at COLM 2024
Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness
Xinran Zhao', Tong Chen', Sihao Chen', Hongming Zhang*, Tongshuang Wu' :Carnegie Mellon University, UUiiversity of Washington, UUiversity of Pennsylvania, *Tencent AT Lab, Bellevue [xinranz3, sherryw]@andrew. cmu. edu
Abstract
The task of Information Retrieval (IR) requires a system identify relevant documents based on users' information needs. In real-world scenarios, retrievers are expected not only rely on the semantic relevance between the documents and the queries but also recognize the nuanced intents or perspectives behind user query. For example, when asked to verify a claim, retrieval system is expected to identify evidence from both supporting vs. contradicting perspectives, for the downstream system to make a fair judgment call. In this work, study whether retrievers can recognize and respond to different perspectives the queries - beyond finding relevant documents for a claim, can retrievers distinguish supporting documents? We reform and extend six existing tasks to create benchmark for retrieval, where we have diverse perspectives described in free-form text, besides root, neutral queries. We show that current retrievers covered in our experiments have limited awareness of subtly different perspectives in queries and can also be biased toward certain perspectives. Motivated by the observation, we further explore the potential leverage geometric features of retriever representation space to improve the perspective aware- ness of retrievers in a zero-shot manner. We demonstrate the efficiency and effectiveness of our projection-based methods on the same set of tasks. Fur- ther analysis also shows how perspective awareness improves performance various downstream tasks, with 4.2% higher accuracy on AmbigQA and 29.9% more correlation with designated viewpoints on essay writing, compared to non-perspective-aware baselines
opposing
Introduction
Large Language models (LLMs) have now reshaped various real-world applications, be it creative writing (Yuan et al., 2022), natural-language search (Ziems et al., 2023), embodied agents (Wang et al., 2023), or web-based AT assistance (Yao et al., 2023; Xie et al., 2024). However, there is widely recognized limitation affecting these applications (Khattab & Zaharia, 2020): the knowledge embedded within the parameters of language models can often be opaque, static, and inefficient. To enhance the practical usability of LLMs for knowledge-intens intensiveeeiieeeeeee tasks, substantial efforts have been dedicated to retrieval-augmented generation (RAG, Lewis al., 2020) to fetch external knowledge using retrievers, and thereby augment LLMs as part of the context.
Retrievers, however, are double-edged swords. While they bring in richer contexts, they can also become bottlenecks when they miss the nuanced demands of users. For instance, consider the scenario in Figure where a user requests an LLM compose an essay opposing the claim, "African governments should enforce stricter animal protection measures." It crucial for the retriever to accurately grasp the user's 's stance of opposition; failure might result in the inclusion of documents that, while relevant, do not align with the user's 's
Our code is available https::/ //t:::///titit wtwp://cco//oll/housstttttoo//o/

---

Published as a conference paper at COLM 2024
Retrieved articles, ranked
RAG output from GPT-3.5
Stance
Support
Oppose
Articla I: A tougher approach neadad to prevent their natural habitats...
Articla Ni Creating tougher respanses will fai to deter poachers because...
Protecting endengered animalls is crucial, imposing tougher protections solely is not enough. Besides tougher protaction, other
User Instruction
Queey
Exirting retriever
Argumeett wrong side
Write an essay opposing the claim:
Afrcan Covermeent must implement taughier protactioos for anumais
Artile 1: Creating tougher tougherresponser responses will fail to deter poachers because
Artic i N: A tougher approach needeed to prevent their natural habitats...
Taugher protections the solution. Peachers operate driven by saciosconamic factors, the severity of consequences...
Quany
Perypection: Oppors
Parspactive-aware retriever
Argument from the right side
Figure 1: An example of how perspective-ware information retrieval differs from the current retrieval pipeline. Perspectives further specifying the intent, e-g., "Article that opposes", will influence the ranks relevant articles, hence influencing the downstream task performance.
perspective (e-g,, those that actually support the claim). Such misalignment could lead to lack of helpful information or even Ripple Effects (Cohen et al., 2023), which skew the whole LLM's output in a biased manner. However, existing RAG systems typically leverage fuzzy retrievers (Cross, 1994) that emphasize the overall lexical or semantic similarity, e.g., token- based BM25 (Robertson & Zaragoza, 2009) or embedding- 2020). It remains unclear whether retrievers can adequately capture the subtle perspective differences In Figure 1, the perspective is only reflected by single keyword "opposing." "
DPR (Karpukhin et al.,
In this work, we evaluate and improve retrievers on their sensitivity to the subtly differ- ent perspectives in queries. First, to enable systematic analysis of retrievers' perspective awareness, we create a I'erspective-wwar Information Retrieval benchmark (PIR). Each instance PIR is tuple of {base query (e.g., " Affrican government..""" per ve (e-g., support/ oppose), suppor/opppose), gold documents). We create PIR by repurposing six existing datasets² in various domains, such that the benchmark covers multiple wealdworld tasks that require retrieval (e-g,, news, argument mining, question answering, etc), as well as diverse perspec- tives (left or right-wing ideologies for news, supporting refuting claims for fact-checking, etc.). In total, we collect 7,494 diverse queries and 10,286 corpus candidates. Along with the data structure, we further design new metric, p-recall, for capturing if retrievers perform consistently across perspectives.
We test five off-the-shelf retrievers using the dataset. We find that existing retrievers are biased towards choosing candidates from certain perspectives and struggle to distinguish between semantically similar queries that convey different perspectives (Section 2.4). In fact, PIR also exposes potential social bias within these systems. For instance, we observe tendency for retrievers to favor news sources from specific countries, regardless of explicit instructions in the queries to target other countries (Section 2.5).
Then, to improve perspective awareness, we propose Perspective-aware Projection (PAP), i.e., to emphasize the perspectives, by projecting the embeddings of the queries and corpus candidates onto the perspective through vector space computation. We show that PAP outperforms other baselines in various settings (Section 3.2), offering straightforward and efficient way to improve the perspective sensitivity of the retrieval process without needing additional fine-tuning.
Moreover, perspective-aware retrieval can significantly improve downstream task perfor- mance (Section 3.3). With PAP, state-of-the-art LLMss like GPT-3.5- (OpenAT, 2022) answer ambiguous questions with higher accuracy (4.2-point higher on AmbigQA (Min et al., 2020)); Similarly, in writing tasks like Figure i, models can generate essays that more closely align with the designated viewpoints (29.9% more correlation than baseline).
In summary, our main contributions are:
2The proposed pipeline can be generalized to all domains with similar-formatted sources.
2

---

Published as a conference paper at COLM 2024
We introduce a novel benchmark, PIR, to study how retrievers aware of the intrinsic perspective in the queries in various real-world tasks reflecting users' intent.
We design projection-based method, PAP, to improve the perspective awareness of off-the-shelf retrievers, which requires no fine-tuning and is with a minimum adaptation of the original pipeline.
Further analysis of PIR reveals how bias exists in retrieval results and how downstream RAG performance can be improved with perspective-awe retrievers.
2 Perspective-aware Information Retrieval
2.1 Motivation and Task formation
Recently, various retrieval methods have shown impressive performance in finding semantic relevance. However, one issue is that the relevant document extracted may be biased towards certain perspectives and do not match the actual user need in the query. For example, in a scenario where user retrieves evidence (by find an arficle about lopic X) to write an article against a certain topic, with on average 58.5 percent chance, the top selected document will be on the supporting side". Such observation shows that retrieval action solely based semantic relevance may limit the diversity and effectiveness of the retrieval.
To tackle this issue, we identify an intrinsic dimension in addition to the query: the per- spective. Perspectives further specifications of users' actual needs (e-g., find an article that supports/ / opposes) attached to the queries (e.g,, the African government must imple- ment tougher protections for animals). These specifications naturally exist explicitly or implicitly in queries seeking information. They also tend to be conveyed by relatively short phrases compared to the whole query: One- or two-word differences may trigger contrastive perspectives and hence different target documents in the corpus.
We formally define the perspective-aware information retrieval (PIR) task as follows. We denote the set of retrieval candidates (corpus) as C, and each document or passage in the corpus as c E C. Given a query 9 and target perspective p, the goal finding c E C such that and are similar from the perspective of p. A perspectiv--wware retriever, Φ, takes the q p, and c as the input and output a similarity score Φ q, p, c). is then ranked based on the similarity scores of each candidate. In the rest of the paper, we also denote each query with perspectives deprived as root query r. Typically, q p r. For each r, we sample at least two p, with corresponding gold attached to it in the dataset, for the purpose of conducting contrastive experiments and evaluation. Notice that in the realistic retrieval scenarios, only q assumed to be observed, in Section 3.4, we explore the automatic deprivation of p.
2.2 Datasets
Existing retrieval benchmarks usually do not explicitly consider perspectives. However, empirically, we find that a lot of datasets in domains that need retrieval can be easily repurposed for testing perspective awareness. For example, AllSides (Baly et al., 2020) is a dataset originally designed for ideology classification on news articles, where each news also has the metadata of its topic. We can easily create queries like "Find news articles that are about [topic]" which specifies a root query for retrieval; We can further add "...from [ideology]" to specify the ideology perspective. Through such transformation, pairs of queries that differ in single perspective (e.g., "Find news articles that's about [topic] from [Left- vs. Right-wing ideologyl").
can create
Data source. Inspired by this observation, we build the PIR benchmark by reformatting six existing datasets, summarized in Table 1. Similar to the popular retrieval benchmark BEIR (Thakur et al., 2021), we cover diverse retrieval domains, including Argument, News, Question-Answering (QA), and Fact-check. As shown in the table, the data also spread across diverse topics and formats (in terms of query and corpus length).
Detailed analysis is in Section 2.5 after we introduce the data creation.
3

---

Published as a conference paper at COLM 2024
Domain
Perspectives (underline) Source Dataset
Query # Query Len
Corpus # Corpus Len
Argument
Stance
Perspectrum (Chen 2019)
1,431
2,773
Exi Find claim supports / opposes the argument: should alloned to have milltary recruifment schools.
QA evidence Question Speciion-Specific
AmbigQA (Min et al., 2020)
864
12.8
864
31.5
Ex: What &s the legal age of murriage, without parencall consentt or other unfiorization: in Nebruska all but two states in the USA?
News
Political ideology
AllSides (Baly al, 2020)
645
12.4
645
1,089.2
Ex: From Left-wing
Right-wing media, find article topic: terrorism
Fact-check
Evidence Type
FEVER (Ma al,, 2023)
1,104
1,104
Ex: Find a claim that this sentence refutes / supportsc Mother Teresa wai born Macedonia, a country in Europe where mssidntss cullest Mazadonians. Macrionia is Stane language.
News
Topic, Location
AGNews et al., 2023b)
1,450
2,900
162.2
Ex:: Find news article that happened in Canada Brazil and has the same topic as: An insestigutive analysis fus found that India's lack of immestment in public health infrastructine...
Story
Similarity Type
StoryAnalooyy (Jaaangg et al., 2023)
2,000
2,000
Ex: Find story that has similar entities with / an analogy to: Fertilize soil. Mix seads into the fertillzed soii.
Table 1: Statistics and examples of the selected reformatted datasets. Query Corpus Len denotes the sizes and average number of words for the queries and corpora, respectively.
Data format. We unify all the datasets, such that each task (i.e., domain + perspective) contains the following components: (1) queries: a piece text containing a question or sentence describing the need; (2) corpus: list of candidates (documents passages) that contain potentially useful information for queries of the same task; (3) key reference: the mapping between queries and corpus candidates.
Most importantly, we define each query to explicitly contain a root query (italicized the examples in Table 1), and a perspective (underlined). To evaluate perspective sensitivity, we make sure that for each root query, there are at least two queries that have different perspectives on each root query. To enable unambiguous evaluation of perspective awareness, we use heuristics to ensure that all the queries with perspectives have mutually exclusive matches golden retrieval documents. For example, (Yu et al., 2023b) carefully curates an AGNews-style dataset with one utterance per feature combination (e.g,, length, style, and source, etc.) Moreover, we carefully constructed the data transformation function, so as to make the synthesized queries are natural sounding. More details on how we reformat and extend the source datasets are in Section A.2.
2.3 Evaluation Metric
To capture the retriever awareness of perspectives, we adjust the standard metric Recall into a new metric denoted as p-Recall: For each root query, we collect the retrieval performance of all perspectives and average them. The overall performance will then be measured by the micro-average across all root queries, with the consideration of consistent perspective awareness.
Adopting the notations in Section 2.1, with a retriever 4 corpus C, root queries r E Qroot per query perspectives p E Pq. and a threshold k, the metric computed as:
P-Recall@k
Epep, (success p. p. C,k)). p rEg.
Σ
(1)
[2
where success(.) denotes if the gold document of query q with perspective appears in the top k in corpus C, ranked by o similar to Recall@k.
4

---

Published as a conference paper at COLM 2024
With Pempective
0.05
Yes
BM25 DPR SimCSE-sup SimCSE-unsup Contriever TART
Soryhnslgy
0.57
AGNews
9.1
8.0
11.0
10.4
Panquctum
0.30
0.62
Stury Analoyy
71.3
45.3
$6.1
76.4
9.1
12.3
0.50
Perspectrum
50.0
10.4
AnteGGA
0.37
AmbigQA
23.8
55.9
53.3
48.1
1.5
0.8
ABocs
1.09
AllSides
7.8
10.8
13.0
2.8
1.7
E-TEVER
Ex-FEVER
53.7
47.1
5244
53.0
1.9
21.8
Avg.
35.8
33.2
44.3
41.8
4.9
7.8
Recaligg
Figure 2: Retrieval performance (Recall@5) of queries with or with- out perspectives, macro-averaged over the retrievers.
Table 2:
Perspective-aware performance of different re- trievers on PIR, with p-Recall@5. as the metric. Avg. de- notes the macro-average performance across the tasks. Best- performing entries for each row are bolded.
2.4 Experiment: Measure Retrievers' Perspective Awareness
Baseline Retrievers Similar to Thakur et al. (2021), we present PTR as an evaluation bench- mark for zero-shot information retrieval. We test the following zero-shot baselines to reveal the perspective awareness of different kinds of retrieval systems. We consider retrievers with various design: token-based similarity (BM25, Trotman et al., 2014), dense embeddings fine- tuned on retrieval tasks (DPR, Karpukhin et al., 2020), sentence embeddings (SimCSE, Gao et al., 2021), embeddings learned through unsupervised contrastive learning (Contriever Izacard et al., 2022), and instruction-fine fine-ttnnnd embeddings (TART, Asai et al., 2022).
For SimCSE, consider both the unsupervised (-unsup) and supervised (-sup) versions. For TART, following (Oh et al., 2024), we use TART-dual that is based on Contriever. We introduce details of these baseline retrievers in Section A.3 in the appendix.
Results We first compare retriever performances on queries with and without perspec- tives, using the standard metric Recall@5. As shown in Figure 2, while retrievers can distinguish the semantically different queries (i.e., reasonable no perspective performance), they generally display insufficient perspective awareness In some extreme cases, they only achieve half the performance when required to include perspectives. This could result from retrievers erer-eiphasizing the overall semantic similarity while overlooking specific trigger words for perspectives. The observation highlights the usefulness of PIR for studying intrinsic perspective awareness.
One exception is AGNews, where retrievers perform slightly better when retrieving perspectives (though still very poor). As further analyzed below in Section 2.5, retrievers inherently biased towards News Articles from certain countries (e.g., Brazil); In such cases, more explicitly querying for less preferred countries might have counterbalanced the bias. More details are in Section A.5.
Moreover, we find that PIR can help effectively compare retrievers. In Table 2, we present per-retriever performance with our defined metric p-Recall@5. We observe that:
BM25 could perform better than other dense retrievers when it is appropriate for the task (echoing findings in Thakur al. (2021)).
Sentence embedding methods perform well. SimCSE, which is not specifically trained with retrieval tasks, performs the best among all retrievers. One reason behind this can be that sentence embeddings retain more semantic understanding than DPR, which helps identify the perspectives. Among all retrievers, SimCSE-sup, which is further fine-tuned on NLI tasks (not retrieval tasks), performs the best. This may reveal the innate connection between retrieval and natural language inference.
Domain-specific retrievers like DPR might have limited generalizability, echoing Thakur et al. (2021). Token-based BM25 performs better than DPR except for AmbigQA, a dataset that has questions on Wikipedia paragraphs which DPR is trained
5

---

Published as a conference paper at COLM 2024
Perspectrum Undermine
AllSides Right
Retriever
Support
Left
Center
BM25
55.6
44.4
27.9
44.3
27.9
DPR
55.5
44.5
29.4
37.6
32.9
SimCSE-sup
58.8
41.2
32.0
41.2
26.8
SimCSE-unsup
40.8
26.1
39.6
34.2
Contriever
66.1
33.9
28.6
42.9
28.6
TART
55.6
51.9
33.3
66.7
0.0
Table 3: Portion of different perspectives when the retrievers successfully retrieve relevant documents with root queries, We can observe that retrievers are biased towards supporting documents (for Perspectrum) or news articles from the right-wing media (for AllSides). In the corpus, the number of entries related to each perspective is designed to be equal.
Siifeeedd lesals
Expested Portie
1.0
News dusiied
0.5
0.0
Colombia
Brack
South Kores
Popus Gaine
Chins
Germany
Ethopia
japa-
Conads
India
Auerralo
New Zestord
Costa Rico
Magurie
Ital
South Affice
Egypt
Nothorluntss
Argendio
Samou
Sustaral
Vens
Desired countries the query
Figure 3: Expected portion of news articles from the desired or other countries in the top retrieval results with SimCSE-sup on AGNews. Queries are from the location perspective, e.g., Find a news article on X topic and happen in Y, where observe that retrievers show imbalanced performance across countries. For example, users seeking news from Guatemala will experience a lower chance satisfied retrieval than from Colombia. In the corpus, the numbers of articles per country are designed to be equal.
the desired country. We can
Current instruction tuning may not fully solve the sensitive perspective change issue. Though TART outperforms its base: Contriever, both perform worse than others.
2.5 Analysis: Biases in Retrievers Revealed by PIR
In Section 2.1, we briefly described the potential bias of retriever results. With PIR, we can now formalize the analysis with quantitative observations. We highlight two
Retrievers can present biased opinions. We conduct retrieval on PIR with perspective- neutral root queries, to answer the question: Are retrievers inherently biased towards doc- uments with certain perspectives? As an example, we run experiments on Perspectrum and AllSides as they have fixed set of perspective options (Stance: support vs. oppose; Political ideology: Left, Right, Center). From Table 3, we can observe that, even though the corpus candidates are balanced with respect to perspective by design, retrievers are still biased towards extracting Supporting documents right-wing news, which denotes that retrieval without perspectives may limit the result comprehensivenes,
Retrievers are not equally usable across countries. In our experiments on AGNews, we notice that news from some countries are more easily retrieved than others. For example, in Figure 3. Users who seek news articles from Colombia and Brazil have higher chance (around 80%) finding the desired articles in the top 5 retrieved results, compared to those who seek news articles from France Guatemala (around 20%). We present further analysis on (1) the bias when the desired countries are not in the queries; and (2) what the other countries are, in Figure 3 in Section A.10 in the appendix.

---

Published as a conference paper at COLM 2024
3 Improving Perspective Awareness
In this section, we explore improving the perspective awareness upon the best-performing retriever in Section 2.4: SimCSE. We hypothesize that one reason behind the lack of perspec- tive awareness is that the perspectives are over-shaded by the overall query semantics since different perspectives are commonly conveyed by only one or two words. Accordingly, we propose to enhance awareness by projecting the query representation to the hyperplane of perspectives, so that the similarity between the query and candidates is forced be conditioned on the perspectives.
Here, we assume that the perspectives in the queries are pre-extracted with heuristics. We explore the extended setting to extract perspectives from raw queries with generative model in Section 3.4.
3.1 Perspective-aware Projection (PAP)
Our method first encode each query, root query, perspective, and corpus candidate into embeddings q, r, P, and с respectively, with language models SimCSE). We treat all these elements as raw textual sentences and use the mean of token embeddings as sentence embedding. Theoretically, any transformer-based model could be used as the encoder, but by default, we choose to use SimCSE as we see which reflects its effective sentence embedding capability.
Then, we perform perspective emphasis, by projecting the query embedding 9 and (option- ally) corpus instance c onto the hyperplane that vertical to the perspective embedding (p), via:
9 P |p|pP
P I
qp = q
c
Such that we can retrieve relevant instances via the cosine distance between the projected qu and Cp. Corpus projection is arguably more expensive, which we make to be optional. We denote the vanilla and efficient version without corpus projection as PAP, and the one with corpus projection as PAP+. It is worth noting that there be tunable hyperparameter added before the projected component of & or c to control the level of perspective emphasis.
Computational Efficiency. practice, we would hope to implement both PAP variants through a combination of offline preparation and online inference. Following the RAG convention, we will always pre-compute all the corpus instance embeddings c É C. In real scenarios, for tasks with known desired perspectives (e.g., supporting and opposing for argument retrieval), we can also pre-compute the projection of corpus entries Cp E Cp to improve efficiency. 9 and 9p, on the other hand, have to be computed up the inference, we conduct the projection of all candidates at once with matrix operation and retrieve the most relevant one with maximum inner product search (MIPS) operation. Empirically, we could use existing MIPS algorithms such as FAISS (Johnson et al., 2019) to reduce the overall inference complexity to [[]().
the fly. To speed
3.2 Experiment: Retrieval Performance
We compare the projection- PAP with alternative interventions at the vector level for including perspectives, as summarized in Table 4. We consider baselines as operations both the embedding space level (e-g., add, concal., ett) and similarity score level (dual-sum and fri-sum) 4. From the table, we can observe that, although various methods designed all help with improving p-Recall05, our proposed PAP performs the best for both settings with or without corpus embedding modification. While we set to improve on SimCSE, we observe that PAP can robustly work with various backbones and scales. We include more details in Section A.7. Though the improvement might seem marginal, we show below that makes a significant impact on the downstream task performances.
"We use cosine similarity to keep consistent with the common training pipeline of various retrievers. Recently, Steck al. (2024) discuss the of other similarity functions
7

---

Published as a conference paper at COLM 2024
Scooring function per query
Description
Ava-
baseline
cos <)q,c)
original best performance in Table?
613
add
cos /it p.c)
encode the root and perspectives soparately use the vector sum for
encode the root and perspectives
45.8
concat.
cne (t.p)e)
using the vector concatenation for query
45.4
2 p.c)
compare query and corpus entries with the query perspective subtracted
43.7
cast+
2 P.C p)
compare query corpus
perspective subtracted
dual-sum
cos [tcc + cas 2(p.c)
consider separete score for root and perspectives
45.0
mri-sum
cos (1.4) cos ((pcc) 1 cos (g)cc
consider separate score for rool, perspectives, and queries
43.4
PAP
cos (
project the q query to the pessective plane
46.0
PAP+
(p
project query and corpus entries the perspective plane
46.4
Table 4: Performance and descriptions of different methods to improve the retriever perspec- tive awareness. denotes that the method contains modification over the corpus vectors. c,9 9 denote the embeddings of corresponding root queries, perspectives, corpus entries, and queries, respectively. cos (.)) denotes the cosine similarity. Details about the proposed PAP are in Section 3.1. Avg. denotes the average performance over the six tasks of PIR, with p-Recall05 as the metric.
3.3 Analysis: Impact on Downstream Tasks
To uncover the impact of PAP on downstream tasks, we study retrieval-uugmented genera- tion (RAG) performance with AmbigQA (120 examples, with top-1 retrieved document) and Perspectrum (30 examples, with top-5 retrieved documents). We use GPT-3.5-Turbo (Ope- nAI, 2022) as the reader model.
For AmbigQA, we define the downstream task to be the original question-answering task. For example, for the question: What the legal age of marriage, without parental consent or other authorization, in Mississippi? the answer is: 21. We use the exact match between answer and output as our metric and report the overall accuracy. For Perspectrum, design argumentative essay writing task. Specifically, we ask models to write short essay based on the topics and corresponding stances (derived from queries in our Perspectrum task), e.g., Topic: It should be allowed to have military recruitment in schools; Stance: Oppose. We measure if the essays written follow the sentiment of the topic-stance pairs by computing the Pearson correlation between their sentiment polarity. We the polarity scores extracted by TextBlob (Loria, 2020). Detailed experiment settings and prompts are presented in Section A.8. We also identify and call for human evaluation and other perspective-rrlated downstream tasks as important future work, for example, retrieving different lemma variants in TheoremQA (Chen et al., 2023b).
We set no retrieval (remove the knowl- edge part in the prompt) and gold retrieval (include gold knowledge attached to the question) as baselines, and compare our proposed PAP, PAP+ with their backbone vanilla retriever SimCSE-sup (pre-pend the retrieved results background knowledge before questions). From Table 5, we ob- serve that retrievers with perspective aware- ness lead to improved performance on the downstream task, besides the retrieval per- formance shown in Table 4. For Perspec- trum, without perspective-aware retrieval, involving controversial documents as the background may lead to output essays that do not follow the original instructions.
AmbigQA (acc.)
Perspectrum (corr.)
Retriever
62.5
16.1*
gold
77.5
38.7*
SimCSE
72.5
-2.4
PAP
74.2
18.5*
PAP+
76.7
20.9"
Table 5: Performance of different retrievers AmbigQA and Perspectrum. acc./ /corr. denote ac- curacy and Pearson correlation coefficient, respec- tively. Both metrics are the higher the better. We use to denote entries with p-value < 0.05.
Why is the downstream task so much better, when p-recall@s only shows marginal im- provement? Similar to Lee et al. (2021), we speculate that the current metric of retrieval performance may not necessarily correlate with the end task performance: Metrics like recall only measure the correct document is retrieved, but do not capture how incorrect other involved documents might be. If top-5 documents are included but four of them are relevant but incorrect, the model can easily get misled (similar to Figure 1). This aligns with the
8

---

Published as a conference paper at COLM 2024
findings from Yu et al. (2023a): even if the right answer is in the context, models can also potentially output the wrong answer that also in the context. Aligning the retrieval and downstream task metrics is important but out of the scope of this paper.
3.4 Feasibility Check: Generative Perspectives from Queries
In this paper, we discuss how perspectives naturally exist in user queries, and conduct ex- periments assuming known perspectives various tasks. However, besides the heuris- tics, can we extract perspec- tives automatically?
Dataset
QA-F1
A BM25
A DPR
A SimCSE
AGNews
78.0
12-12
44
8-8
Story Analogy
65.9
56-60
26-32
36-38
Perspectrum
68.6
26-28
38-38
50-50
AmbigQA
70.2
21-18
46-44
44-44
AllSides
66.4
6-8
8-10
8-8
EX-FEVER
68.6
80-82
60-60
72-72
We explore whether it would be practical to generalize the PIR setting to the real world, by trying to acquire meaning- ful generative perspectives with LLMs. Specifically, we extract the perspectives from queries with GPT-3.5- Turbo in a zero-shot setting. We select 50 examples from each task and test if the perspectives extracted match the annotated perspectives. We prompt the model extracting part of the sentence specifying perspectives. Following Zhao et al. (2023), we measure the similarity by a common metric QA-F1 used in open-domain QA, calculating the max uni-gram overlap between two pieces of text. We present the details for the prompt and metric computation in Section A.8. The prompt is simple and not heavily engineered to test if models can distinguish user meta-description (e.g., Find and general information (e.g., the source news content) well. Besides the overlap, we also measure if the performance consistent between annotated and generated perspectives. We use Recall@5 as the metric to measure the retrieval performance changes on each task, with the corresponding full corpus.
Table Performance of automatic perspective extraction (QA-F1) from queries on different tasks with GPT-3.5-Turee with corre- sponding retrieval performance changes (Recall@5). A-B denotes the performance with annotated perspectives (A) and generated perspectives (B).
with instructions
article)
From Table 6, we can observe that GPT-3.5-Turbo achieves good performance in extracting the perspectives from queries. On the one hand, good performance denotes the naturalness of decomposing queries with perspectives and root queries. On the other hand, since per- spective extraction can be done automatically, our proposed PAP can potentially be extended in general scenarios where can not rely on heuristics to find perspectives. Regarding the retrieval performance, we can observe that the retrieval performances from annotated and generative perspective extraction are similar across tasks and retrievers, which validates our claim of the potential to automate perspective extraction real scenarios.
Related Work
Task-aware Retrieval Recent neural retrievers (Karpukhin et al., 2020; Santhanam et al., 2022) have shown their superiority over lexical retrievers (e-g., BM25, Robertson & Zaragoza, 2009; Trotman et al., 2014) in various domains. Due to the high cost of annotating retrieval datasets for new target tasks, there is an increasing focus on improving the generalizability of neural retrievers across diverse sets of domains and tasks (e.g,, BEIR, Thakur et al., 2021). Retrievers trained with unsupervised optimization targets (Izacard et al., 2022) or large-scale datasets (Ni et al., 2022; Wang et al., 2022) show surprisingly good generalization ability across diverse domains.
Given improved cross-domain generalization of neural retrievers, studies have started to investigate the cross-task generalization (Asai et al., 2022; Su et al., 2022). Inspired by the promise of instruction tuning on language models, training retrievers with task-specific instructions achieves improved in-domain performance with an instruction-prefit (Yang et al., 2021; Mysore et al., 2022; Ravfogel et al., 2023; Zhang et al., 2023; Jeong et al., 2024) or

---

Published as a conference paper at COLM 2024
an instruction-mocuue in REMOP (Liang et al., 2023). Instruction-tuning also demonstrates zero-shot generalization on unseen domains and tasks (eg., TART, Asai et al., 2022). In our work, present TART still can not solve the perspective imbalance. Yet perspective-aware instruction fine-tuning can be one important future direction to reduce it.
Retrieval Evaluation The benchmarks designed for evaluating generalization ability across tasks include BEIR (Thakur et al., 2021), LoTTE (Santhanam et al., 2022), and X2 (Asai et al., 2022). In these datasets, search queries vary across different tasks. Therefore, we cannot compare the rank difference on the same query given different tasks (or perspectives). Additionally, M-BEIR (Wei et al., 2023) provides diverse retrieval tasks but constrained in our study by its multi-modal nature.
Some concurrent works also evaluate the retriever's instruction following ability given instructions in natural language form, while all these works define instruction and tasks from different views. InstructIR (Oh et al., 2024) focuses on user preferences given instructions that contain the user's characteristics such background, situation, job, hobbies, etc. FollowIR (Weller et al., 2024) constructs instructions to explicitly describe what documents are relevant and not relevant. In our work, we motivate intent perspectives from various end-tasks that users may need. We construct tasks by seeking different ways to measure the similarity between query and document, inspired by prior works on conditional semantic similarity (Deshpande et al., 2023). Despite the different definitions of instruction and tasks, these studies draw similar conclusions that the existing neural retrievers are struggling to follow the instructions.
Conditional Similarity and Text Embeddings Besides the conditional query-document relevance discussed above, recently, there has been a discussion regarding conditional text embeddings in the community. MTEB (Muennighoff et al., 2023) is proposed to evaluate text embedding performance on diverse tasks. C-STS (Deshpande et al., 2023) highlights how short textual description will influence similarity between sentence pairs. (Su et al., 2022) presents how involving instructions improves text embedding performance cor- responding tasks. In our work, we introduce perspectives as factor that influences the query-document relevance.
Conclusion and Future Work
Building information retrieval systems that efficiently and accurately respond to user queries is crucial. This paper introduces an additional intrinsic pepponnnt-perpeectiin information retrieval process, ensuring that documents are not only relevant but also accurate. We present new benchmark, PIR, which consists six realistic scenarios designed to evaluate retrievers' sensitivity to different perspectives. Our experiments reveal biases in current retrieval technologies. To address this, we introduce a novel, zero-shot, projection-based method called PAP, designed to enhance perspective awareness without the need for further fine-tuning. Our analysis highlights the critical role of perspective awareness, demonstrating its impact on various applications. We hope the proposed PIR and PAP can benefit the community by developing retrievers with sensitive perspective awareness and hence ensure consistent performance in various downstream tasks.
the
We also observe potential future work to extend the scope of this paper: (1) Alignment between the retrieval and downstream task performance: as discussed in Section 3.3; (2) Perspective-aware re-ranking: in our paper, we only adapt the retrievers. In Section 3.4, we show that LLMSs can identify perspectives in queries. However, using LLMSs to identify the gucry-document relations will significantly increase the cost. Further methods should be designed to improve the efficiency of perspective-aware re-ranking, e.g., by identifying the necessary cases; (3) P'rspective-awart instruction finetuning: TART achieves improved performance compared to the backbone Contriever. Our data creation pipeline and use of generative models can potentially help extract natural collection of contrastive perspective- aware instructions in TART style. Further cnarructionfinetuniii with such a collection is expected to improve retriever perspective awareness.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
The work was supported by the ONR Award N000142312840. The authors thank Xuanyu Zhou, Ruixin Hong, Vijay Viswanathan, Chenyang Yang, and Christina Ma for their valuable feedback, and anonymous reviewers for helpful discussions and comments.
References
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. arXiv preprint ar Xiv:2211 1.0260,,,, 2022.
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), EMNLP '20, pp. 4982-4991, 2020.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning, A large annotated corpus for learning natural language inference. In Lluís Mårquez, Chris Callison-Buree Burch, and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Pp. 632-642, Lisbon, Portugal, September 2015. Association URL
for Computational Linguistics. doi: 10.18653/v1/11--11 org/D15-1075.
Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. Seeing things from a different angle: Discovering diverse perspectives about claims. 2019.
Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense x retrieval: What retrieval granularity should we use? arXiv preprint ar Xiv: 2023a. URL httpp:://arxiv wwciiii.oorr//eeee orggpdf//311 /2312, 06648. 06648.pdf.
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: A theorem-driven question answering dataset. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 7889 7889-7901, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653/ /v15///vvvvv// htt///222..mmmmmmme 489. URL httts:://allanthollgyyy .rgge/oorr////0 2223. emnlp-main, 489.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models, 2023.
Valerie Cross. Fuzzy information retrieval. Journal of Intelligent Information Systems, 3(1): 29-56, 1994.
Ameet Deshpande, Carlos Jimenez, Howard Chen, Vishvak Murahari, Victoria Graf, Tanmay Rajpurohit, Ashwin Kalyan, Danqi Chen, and Karthik Narasimhan. C-STS: Conditional semantic textual similarity. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 5669-560 -669, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 wtt/2223.emnl-mmaae
URL https: http///aaaaaanttoaooooao org/ 2023. emnlp-main 345.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compuatational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. /1665//v//N991111 TTg/N19-1425
URL https: / h/wps//ttllaaoogg......oo
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of
sentence embeddings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural
11

---

Published as a conference paper at COLM 2024
Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 6894-6910. Association for Computational Linguistics, 2021. URL https://dbi.orr//1 18653/v1/2021, emm1p-main.552
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022. URL https:
//arxiv.org//df/21 09..0.9111.1....... p.f...
Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. Adaptive- rag: Learning to adapt retrieval-uugmented large language models through question complexity. In NAACL, 2024. URL https://arxii..ottt org./ wtw.:///gg/bb////gg////e abs/2403. 14403.
Cheng Jiayang, Lin Qiu, Tsz Chan, Tianqing Fang, Weiqi Wang, Chunkit Chan, Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song, Yue Zhang, and Zheng Zhang. StoryAnalogy: Deriving story-level analogies from large language models to unlock analogical understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11518-11537, Singapore, December 2023. Association for Computational Linguistics. doi: http:///111/222//lmm1mm
URL htttps///ccaani
org/ emnlp-main. 786.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. [EEE Transactions on Big Data, 7(3):535 547, 2019.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proczedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6769-6781. Association for Computational Linguistics, 2020. URL httpps///of org/10. 18653/v1/2028 wwt.::00000mmmmmmmmmmmmmmmmmmlmm main.mai..555 550.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (eds.), Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 39-48. ACM, 2020. URL https: //doi.org/1e /1111/39///7111... 3.012277...4.7....
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162 tt2t2//a2claa0076666 URL https: https////o/claaooooogoo wwwpeggy.orr////1 9 9-1011991-6660
Jinhyuk Lee, Alexander Wettig, and Danqi Chen. Phrase retrieval learns passage retrieval, too. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3661-3672, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v /201 12t0122em.llmmmaiii... emnlp-main.297
Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih
URL https:
//aclanthology. org/2021 www.p-mainnnn.m..eeemmmmmmmmmaem 297.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intenst nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Juhao Liang, Chen Zhang, Zhengyang Tang, Jie Fu, Dawei Song, and Benyou Wang. Modular retrieval for generalization and interpretation, 2023.
Steven Loria. textblob documentation. Release 0.16, 2, 2020.
12

---

Published as a conference paper at COLM 2024
Huanhuan Ma, Weizhi Xu, Yifan Wei, Liuji Chen, Liang Wang, Qiang Liu, Shu Wu, and Liang Wang. EX-FEVER: dataset for multi-hop explainable fact verification. CoRR, abs/231 htss/230..9954, 2023.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Bonnie Webber, Trevor Cohn, Yu- lan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5783-5797, Online, November 2020, Asso- ciation for Computational Linguistics. doi: https://aallantholooy org/2020. een.20...nlppmmmmm emnlp-mmin. 466.
URL
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014-2037, Dubrovnik, Croatia, May 2023, Association for Computational Linguistics. doi: (1186633v112000 waw.:///mm.....maam.mammmm 144.-mai...1. URL https://aclaatthlll/
eacl-main. or wwr//22..aallmminn 148.
Sheshera Mysore, Arman Cohan, and Tom Hope. Multi-vector models with textual guidance for fine-grained scientific document similarity. In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4453-4470, Seattle, United States, July 2022. Asso- ciation for Computational Linguistics. doi: 2t066///112222.aaa-mmoo/ https://aclanthologg httgg/222..aaaaeee aisacc-mmin.33
URL
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. 092668.
CoRR, abs/ 1611..69, 2016. URL
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vin- cent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. Large dual coders are generalizable retrievers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 9844-9855, Abu Dhabi, United Arab Emirates, December 2022. Asso- ciation for Computational Linguistics. doi: 10.1853/v1/20022emmppp mainm6.,....66. URL https://aclaanthology htwphhholloyyoog//e2/e2/e enn.e2maanpp-m.m6eeem
Hanseok Oh, Hyunji Lee, Seonghyeon Ye, Haebin Shin, Hansol Jang, Changwook Jun, and Minjoon Seo. Instructir: A benchmark for instruction following of information retrieval models. 2024.
OpenAI. Chatgpt, 2022.
OpenAl. Gpt-4 technical report, 2023.
Shauli Ravfogel, Valentina Pyatkin, Amir DN Cohen, Avshalom Manevich, and Yoav Gold- berg. Retrieving texts based on abstract descriptions, 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Langunge Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 29..980-000. 3980-3990. Association for Computational Linguistics, 2019. URL https://doi org/10 10. 18653/v1/D19-1411
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333 389, 2009, URL http:::/doooooo///// 1561/ 1500000019.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapler of the Association for Compulational
13

---

Published as a conference paper at COLM 2024
Linguistics: Human Language Technologies, Pp. 3715-37356 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 20.18653/1122222mmadd https://aclannthology hwcpllooyyoorg///// www.:22maa..mmminn.....ma 272.
URL
Harald Steck, Chaitanya Ekanadham, and Nathan Kallus. Is cosine-similarity of embeddingg really about similarity? In Companion Proceedings of the ACM on Web Conference 2024, WWW '24. ACM, May 2024. doi: 00.1145/36993533651222 URL http://dxx.di..rr///
1145/3589335.36511
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction- finetuned text embeddings. 2022. URL https:
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https:: //pperv//ppnrrvview.. wwwprrrweeiweeeeeeeeeeeeee www.neet//oonaee nett/foorummdddwtu555fffi
Andrew Trotman, Antti Puurula, and Blake Burgess. Improvements to bm25 and lan- guage models examined. In Proceedings of the 2014 Australasian Document Comput- ing Symposium, ADCS '14, pp. 58-65, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450330008. doi: 10.1145/2662862.266623 https://dbi.orr//ee 10. 1145/2682862.1. 2682863.
URL
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ar Xiu preprint Xrv: Arxio-2305.16991, 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. Xrv preprint arXtt:2212.03533, 2022.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. Xrv preprint arXio:2311.17136 2023.
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. Followir: Evaluating and teaching information retrieval models to follow instructions, 2024.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: benchmark for real-world planning with language agents. ar Xiv preprint arXi::2402.01622 2024.
Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Pp. 11542-11552, 2021. URL https://api. www.enniisshooa.oorooooorrreoree
237267174.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. Chain-of-note: Enhancing robustness in retrieval-augmennee language models, 2023a.
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. In Whirty-Seeentt Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b.
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. Wordcraft: Story writing with large language models. 27th International Conference on Intelligent User Interfaces, 2022.
URL https:/ //api. swwanticschol wwroollarrrrrr//rr/eeeeeoeoo www.:oorg///uuueee argg/CorpssDD: 24755555 247585187.
14

---

Published as a conference paper at COLM 2024
Wenzheng Zhang, Chenyan Xiong, Karl Stratos, and Amold Overwijk. Improving multitask retrieval by promoting task specialization. Transactions of the Association for Computational Linguistics, 11:1201-1212, 2023. doi: 10.1162 t/cllll//////// 00597. URL https://acllanthology org/ 2023. tac1-1.68.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
Xinran Zhao, Hongming Zhang, Xiaoman Pan, Wenlin Yao, Dong Yu, and Jianshu Chen. Thrust: Adaptively propels large language models with external knowledge, 2023.
Noah Ziems, Wenhao Yu, Zhihan Zhang, and Meng Jiang. Large language models are built-in autoregressive search engines. In Anna Rogers, Rogers,Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 2666-2678, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/ v1 v////11/22222 2013/222..immm/// aic. URL https://aclanntool.oo
findings-acl. accs 167.
15

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Dataset Examples
We present detailed examples of perspectives, root queries, and gold corpus candidates in Table 7. We can observe that different perspectives from the same task are commonly triggered by few-word differences, compared to the root queries. We can also observe the naturalness of root queries inherited from the original tasks and the diversity of involved components in style and semantics. Some details of the gold entries are omitted for better presentation.
Dataset
Perspectives
Root Queries
Gold Corpus Candidates
Find news article re- lated to Humanitarian crises in the same loca- tion of:
Investigative journalists have uncuv- ered shocking evidence of extreme hu- manitarian crises in parts India
An investigative analysis has found that India's 's lack of investment in public health infrastructure...
AGNews
Find news article that happened in Canada and has the same topic
the wake of the ongoing COVID-19 pandemic, researchers in Canada have raised concerns about...
Find sentence that is an analogy to: Find story that is with similar entities with:
Apply lotion the skin. Massage the lotion into the skin. Sprinkle hope into the nurtured soil, al- ish.
Fertilize the soil. Mix seeds into the fertilized suil.
StoryAnalogy
lowing dreams to take root and flour-
Find a claim that sup- ports the argument: Find claim that op- poses the argument:
Young people should hear of the oppor- tunities...
should allowed have military recruitment in schools
Perspectrum
School children are too young to target for military service...
in Nebraska
In Nebraska, the legal age of marriage is
What is the legal age of marriage, without parental consent or other authorization
AmbigQA
in Mississippi
In Mississippi, the legal age of marriage
21
in all but two states in the USA
Most states share the legal age of marriage ... 18.
From Left-wing media,
Story highlights Tsarnaev has said his brother drove the attuck... The twin bombings killed three peo- ple and wounded at least 176. Patrick told Fox...
find news article on the topic: terrorism
AllSides
From Right-wing me- dia,
Find claim that this sentence refutes:
Mother Tenea was born country in Europe where the residents called in Slavic language.
Mother Teresa was born in Macedonia, country in Europe where the residents called Macedonians. Macedonian is Slavic language.
EX-FEVER
Find claim that this sentence supports:
Mother Teresa is nun and missionary who was burn in country in Russia where the residents called in an Eastern South Slavic language.
Table 7: Examples of the selected reformatted datasets. Root queries are queries without perspectives. Real queries in the datasets are not always the simple concatenation of perspectives and root queries. The root queries and gold corpus candidates demonstrated can be part of the actual ones in our benchmark.
A.2 Details of Dataset Creation
The details of the involved dataset are as follows:
1. AGNews (Zhang et al., 2015; Yu et al., 2023b): we create a News Retrieval task from the synthesized AGNews generated by (Yu et al., 2023b). The queries for this task are finding a piece of similar news to the given news from particular aspect: e.g., Find an article that happened in France and is similar to News X. (Yu et al., 2023b) leveraged systematic approach to generating each piece of news with multiple control factors: length, location, subtopics, and styles. With this feature, we can reformat the dataset to a similar news retrieval task where each piece of query news only matches one target. We
16

---

Published as a conference paper at COLM 2024
choose topics (same style, same location, different topics) and locations (same style, same topic, different locations) as our perspectives.
StoryAnalogy (Jiayang et al., 2023): (Jiayang et al., 2023) create a large-scale dataset with human-annotated analogically similar story pairs, e.g,, the virus inoades cells is analogically similar to burglars break in the house. We sample the top 1,000 story pairs annotated with the highest analogical similarity and lowest semantic similarity in the original dataset. We create a story retrieval task with these pairs for the analogy perspective. To create similar stories from the semantic perspective, we leverage GPT-44 (OpenAI, 2023) to create similar sentences with respect to the query topic and entities.
3. Perspectrum (Chen et al., 2019): the authors of the original paper studied how evidence paragraphs from different sub-claims lead to different stances towards the original claims. We use the original claims as the queries of our retrieval task and leverage the human- annotated stances to form gold candidate evidence paragraphs from either supporting, opposing perspectives.
4. AmbigQA (Min et al., 2020): the authors of the original paper studied how the ambiguity of the questions leads to multiple plausible answers. Correspondingly, for each of the plausible answers, specific supporting evidence is attached. For example, for the question: Who sings the song what a beautiful name it is?, two perspectives are: which group and who the lead singer. Answering these unambiguous perspective questions requires different supporting documents. We create our QA retrieval task in line with the general open- domain QA setting (Kwiatkowski et al., 2019). The original questions are the queries and corresponding supporting paragraphs from Wikipedia are the retrieval candidates.
5. AllSides (Baly et al., 2020): the original authors study how bias and ideology shape argument writing with news articles from different media and ideology annotation from a popular online forum5. For each topic, e.g., 32-Hour Workweek Good Idea?, there are reports from multiple sources (e.g., Fortune, Mew York Post), with annotations from humans on whether the arguments are likely from the Left, Center, or Right-Wing ideologies. We choose the original topics the questions and these ideologies as the perspectives. For AGNews, we create similar news retrieval task. For AllSides, we create topic-news retrieval task. We anticipate both scenarios to appear in the real world when users seek news articles.
EX-FEVER (Ma et al., 2023): the original authors build a large-scale fact-checking dataset with multi-hop explanations attached to each of the claims. We treat the explanations as the queries for our Fact Retrieval task to study the relations among claims. Claims that explanations support, refute, or have no enough information as the retrieval tar- gets. These retrieval targets are typically on the same topic and with one or two-word differences that make them either piece of fact not.
A.3 Details of Baseline Retrievers
We present PIR as an evaluation benchmark for zero-shot information retrieval. We test the following zero-shot baselines to reveal the perspective awareness of different kinds of retrieval systems.
1. BM-25: We first compare with the traditional BM-25 retrieval method (Robertson & Zaragoza, 2009; Trotman et al., 2014), which is popular key-word matching algorithm with TF-IFF token weights.
2. Dense Passage Retrieval (DPR): DPR model (Karpukhin et al., 2020) was trained to retrieve the most relevant passage given a query with. In the experiment, we select a model trained with the natural question dataset as the representative. We treat all candidates (i.e., pre-trained encoder.
passsagsss as the passages and encode them with the
3. SimCSE: To better measure the sentence similarities, developed over sentence en- coder (Reimers & Gurevych, 2019), SimCSE (Gao et al., 2021) proposed to fine-tune
5 AllSides: https://wwwallssttt wttp:/wwwwwwll....ooo. com. /nbii ased- balanced-news
17

---

Published as a conference paper at COLM 2024
14
With Parrppacioon
With Pertpactio
0.55
0.84
0:67
0.58
0.93
0.43
14
0.38
0.40
0.30
0.30
0.07
0.99
Abeytakkyy
Task
Teat
Figure 4: Retrieval performance (Recall@k, k=5,10) between queries with without per- spectives attached. Per-task performance is the macro-average of all the retrievers.
pretrained language models, e.g,, BERT (Devlin et al., 2019), to capture the sentence-level semantics better. In the experiment, we use SimCSE as the representative sentence en- coder, which shows good performance on retrieval tasks (Chen et al., 2023a). Specifically, consider both the unsupervised (-unsup) and supervised (-sup) versions of SimCSE, where the former leverage dropout as signals and the latter fine-tunes the former with natural language inference data (Bowman et al., 2015).
4. Contriever (Izacard et al., 2022): Contriever is an unsupervised retriever, instantiated with BERT-base encoder. (Izacard et al., 2022) leverage contrastive learning fine-tune the encoder by segment pairs constructed from unlabeled documents from Wikipedia and web crawl data. Specifically, we use the Contriever model that is further trained on (MS-MARCO, Nguyen et al., 2016) for improved performance.
5. TART (Asai et al., 2022): TART is an instruction-tuned retriever that is fine-tuned on BERRI, an instruction-awarr information retrieval dataset on approximately 40 tasks. The instructions are typically meta-data and descriptions about the corpus, e.g., retrieving passage from Wikipedía. (Asai al., 2022) show that involving such meta-data helps improve the retrieval performance, yet TART is not sensitive to wrong instructions provided. Similar to (Oh et al., 2024), we use TART-dual that is based on Contriever.
A.4 Experimental Details
We collect datasets from the official GitHub repositories of the referred work. We conduct our experiments of retrieval on machine with 8 Nvidia A6000 (40G) GPUs with CUDA 12 installed. For experiments on UPT-3.5-Turbor we use the official API from OpenAl with temperature equal to 1.0 and a max length equal to 80.
A.5 Retrievers struggle to distinguish queries with perspectives
We explore how the retrievers perform with or without perspectives contained in each query for the PIR task. Ideally, the performance with perspectives shall be equal to or higher than the queries without perspectives (i.e., root queries) since additional information is provided.
We report the performance of queries with without perspectives in Figure The presented Recall@5 and Recall@10 are the averaged performance from all baseline retrievers. We can observe that, across different thresholds of recall, retrievers generally perform worse when the queries with perspectives, which demonstrates that current retrievers cannot identify and follow perspectives to extract the correct corpus entries. This finding validates the significance of studying the problem of PIR.
One exception is the Recall@10 score from AGNews. One reason behind this can be that retrievers are biased towards News Articles from certain countries (e.g., Brazil) even when the overall similarity is low. this case, providing perspective such as find a news article from Egypt can help.
18

---

Published as a conference paper at COLM 2024
Retrievers
AGNews
StoryAnaloy
Perspectrum
AmbigQA
AlISides
Ex-FEVER
Avg.
no-intervention
11.0
86.1
52.1
53.3
10.8
52.4
44.3
add
19.3
84.0
50.6
53.4
15.2
52.0
45.8
add+
21.1
68.8
32.7
53.2
2.3
46.7
37.5
concat.
6.5
84.2
53.6
45.9
29.6
52.7
45.4
concat.+
10.8
85.6
52.0
52.1
10.0
52.2
43.8
cast
4.6
83.3
51.7
43.5
26.4
52.6
43.7
cast+
86.6
55.3
52.9
22.8
46.0
dual-sum
20.1
81.1
48.5
53.1
15.8
51.5
45.0
trim-sum
17.4
85.0
50.5
53.6
14.0
51.9
45.4
PAP
5.3
86.2
55.4
52.9
23.4
52.6
46.0
PAP+
7.3
86.7
55.2
53.8
23.1
52.6
46.4
Table 8: Performance of different methods to improve the retriever perspective awareness on PIR, with p-Recall@5. as the metric. + denotes that the adaptation is also done on corpus entry embeddings.
Method SimCSE-base-sup
SinCSE-base-unsup
SimCSE-large-sup
SinCSE-large-unsup
PAP
46.0 (+1.7)
42.0 (+0.2)
46.9 (+1.7)
(+0.6)
PAP+
46.4(42.11
42.5 (+0.7)
47.6 (+2.4)
45.1 (+0.9)
Table 9: Performance of PAP and PAP+ with different backbone models and scales. The improvement compared to the retriever without adaptation is in bracket.
A.6 Full Results of Methods in Improving Perspective Awareness
In Table 4, we present the averaged p-Recall@5 for different zero-shot adaptation methods we perform. In Table 8, we present the task-specific performance. We can observe extra details from the results: PAP and PAP+ achieve consistent performance across tasks. How- ever, they degrade the performance of AGNews. Yet add hiddsmm-assed methods achieve good improvement on AGNews. These findings suggest that building further taxonomy on the perspectives may help develop task-specific zero-shot adaptation. This can be an interesting future work beyond the scope of this work.
A.7 Backbone and Scale Ablation
In this section, we test if PAP is robust with various backbones (supervised or unsupervised SimCSE) and backbone scales (SimCSE fine-tuned from BERT-base/-Large) From Table 9, we can observe that we get consistent performance improvement using PAP and PAP+ to adapt the query and corpus embeddings. The gain also higher with larger-scaled backbone SimCSE models.
A.8 Details about Analytical Experiments
In this section, we describe the experimental details for downstream task performance (Section 3.3) and automatic perspective extraction (Section 3.4).
Downstream Task Performance: For AmbigQA, we use the prompt: ("Knowledge:(k): Question:(q); Answer:"). For essay writing with Perspectrum, we use the prompt Y"You are writing short argumentative essay (100 words) on particular topic with a stance. We provide some background knowledge for your reference. Be concise and fluent; Back- ground:(k); Topic:(q); Stance:(s); Short Essay:"). For AmbigQA, we replace (k) with the top-1 retrieved entry. For Perspectrum, we replace (k) with the top-5 retrieved entries.
19

---

Published as a conference paper at COLM 2024
Perspectrum
AllSides
Retriever
Support
Undermine
Left
Right Center
INSTRUCTOR-ban -base
57.0
43.0
33.7
38.5
27.8
INSTRUCTOR-Lare 56.1
43.9
33.0
39.7
27.3
Table 10: Portion of different perspectives when the retrievers successfully retrieve relevant documents with root queries. We can observe that retrievers are biased towards supporting documents (for Perspectrum and Ex-FEVER) or news articles from the right-wing media (for AllSides). In the corpus, the number of entries related to each perspective is designed to be equal.
Generative Perspectives from Queries: Metric: Denoting the gold answer set as 9 and uni-gram tokens of each query and each corresponding gold answer as TT and Tg. the QA-FI score can then be written QA-F1 max Tg) (Tp UTg).
Prompt Template: the prompt template use is: "You are doing task to reformat queries into input for a retriever. here are two components in the query; short phrase describing the user intent perspective and the rest describing the general information; Can you decompose the query and find the intent perspective for me? Answer with part of the query only, no other information is needed; Query: {q); Intent:"
A.9 Other Instruction-tuned Sentence Embeddings
(Su et al., 2022) introduce method to involve instruction in text embeddings with fine- tuning on mixture of 330 multi-purpose tasks. The instructions are general descriptions of what the task is about, e.g,, Represent the caption for duplicate captions.
The extraction of INSTRUCTOR embeddings different from the general text em- bedding methods we introduced in Section 3.1, which makes them not fairly comparable. The model Pinstructor requires both an instruction and a piece of text as the input. We treat perspectives as instructions here and root queries the text input. The model design involves the decomposition of perspectives and root queries by default. We test both INSTRUCTOR- and INSTRÜCTÓR-large
We follow our setting in Section 2.5 to test if the INSTRUCTOR is also biased towards certain perspectives. From Table 10, can observe that the gap still remains and instruction fine-tuning on general tasks does not solve the problem of perspective awareness.
A.10
Details of retrievers can be socially biased towards certain perspectives
1500
1000
Bracd
China
Mextc
Nigento
Nuttor
Calombaaa
erera
Algeria
Unitart
Venennal
Smath 2dica
Gemman
Samoa
Prous Nov Goima
Vaeuatu
Austraia
Ethiogia
Palisann
India
South zone
Guatemaia
Arjontin
Zealand
troptoo
AGNews
Figure 5: Accumulated numbers of news articles the top 5 retrieval results with SimCSE-sup AGNews root queries. An example of the root query is: Find news article that is similar to: Article X. We can observe that retrievers prefer news articles from certain countries, e.g., Brazil. In the corpus and root queries, the numbers of articles per country are designed to be equal.
In Section 2.5, show how retrievers are biased from certain perspectives, which can degrade the downstream task performance. On the other hand, users seeking news articles from different countries may expect different rates of success.
20

---

Published as a conference paper at COLM 2024
5.54
0.06
0.03
0.02
2.03
0.24
D02
China
0.05
0.00
0.05
9.02
9.33
0.02
9:64
0:55
9.03
6:04
0.05
0.32
Nigefia
3.51
622
0.00
6.5
0.03
3.00
0.22
0:11
0.03
6.03
0.01
6.02
New Zealand
0:21
0.00
0.06
5.53
0.01
9.01
613
0.02
0.22
6/03
0.00
0.15 0.15
0.05
Colombia
Neu
0.22
0.0L
0.0
5.02
0.43
0.22
0:22
0.02
0.22
5.04
0.03
2.00
0.02
0.42
Countries the retrieved doc
United Kingdam
0.22
9
9.02
0.03
0.02
0.02
0.0
2.04
0.01
Venezuela
South Afiic
0.00
8.07
3.04
3.04
Germany
0.00
5.00
0.01
0:63
0.00
0.09
Samoa
0.22
0.00
0.23
0.02
2.04
3.04
Canada
0:22
6.01
0.02
2.00
0.23
0.43
0.24
0.05
0.02
Papua
0.04
0.02
Vanuatu
0.00
0:23
0.05
0.09
9.01
0.93
0.02
5.68
3.60
5.00
0.11
6.3
Ethopiia
5.45
0.53
6.23
8.07
0.65
0.05
5.00
Pakistan
India
0.00
0.02
0.53
0.00
0.63
0.01
8.03
0.17
South
Egypt-
8.02
0.02
0.04
0.04
2.04
2.00
2.02
0.24
0.03
2.42
0.08
Gustemala-
Argantina-
0.00
0.02 0.22
0.00
2.02
0.05
0.02
5.00
M.55
0.40
0.01
United States
Italy
0.00
0.01
0.00
0.02
0.02
0.02
2.03
5.07
0.00
Japan
5.00
0.02
0.00
0.22
8.04
0.22
Costa Rica -0.22
0.02
Bradi
China
Moxcc
Nightta
Netherlancs
New Zoaland
Paru-
Algerta
United kingtom
Venazueil
South Airic
Germany
Canada
Vanuatu
Ethiopla
Pakistan-
India
South Kores
Cuatomala-
Argentina-
United States
Italy
Frunce
Japan
Dosta Rice
Pepua
Countries in the query
Figure 6: Expected portion of news articles from the desired or other countries in the top retrieval results with SimCSE-sup on AGNews. Queries are from the location perspective, e.g., Find a news article on X topic and happen in Y, where the desired country.
In this section, we provide more details of the analysis of social bias by answering two questions:
1. if the queries do not contain desired country, will the bias still exist? Figure shows that the bias still exists, news articles from countries such as Brazil are over-repreeented.
2. What are the actual countries in News article from other countries in Figure 3? We unfold Figure and create a heatmap in Figure 6. We can observe that, similar to Figure 5, news articles over-eepeeeented even when the desired origin countries of the articles present in the queries.
21


Reviews:
[{"review_id": "HZvxcPkW5E", "reviewer": "Reviewer_8C7L", "summary": "This paper formally introduces the task of perspective-aware retrieval, that conditions retrieval upon a specific perspective (e.g. opposing or supporting a claim). This has important influence over downstream tasks, such as RAG. The paper introduced a benchmark for this new task, and designed a simple method that outperforms various baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and detailed feedback and for liking our paper. We would like to address your questions as follows.\n\n**Q1: Details about the PAP method**\n\nThank you for your valuable insights on the methodology side. By projection to the p space, we mean the hyperplane that is vertical to the perspective (p) vector. Reducing the query vector component on the perspective vector is equivalent to moving it toward the hyperplane (in the sense of cosine similarity). We will update the introduction of PAP in Section 3.1 with your suggestions.\n\nWe agree that a tunable coefficient is interesting and can help reveal the essence of the problem (relations between perspectives and the rest of the query). In the scope of the current draft, we focus on introducing the task and building zero-shot methods. However, we are indeed exploring the coefficient tuning and will update the resources in our code base to be publicized upon acceptance.\n\n **Q2: Notation and Baseline**\n\nThanks for the suggestion on the notation. We will improve the final version as suggested.\nWe design dual-sum and tri-sum for the comprehensiveness of possible operations on the similarity score level, while others, such as add and concatenation, are operations on the vector/embedding level. \n\n**Q3: Clarification on SimCSE** \n\nSimCSE is a sentence embedding model with two versions: the unsupervised one is fine-tuned with zero-shot contrastive signals from dropouts; the supervised one is further fine-tuned on NLI tasks. We mean that  SimCSE is not specifically fine-tuned on retrieval tasks (In contrast, the Contriever model used in our paper is further tuned on MS-MARCO, a retrieval task). We will add a detailed explanation in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very clearly written and easy to understand, and has important impact over downstream tasks such as RAG.", "reasons_to_reject": "The mathematics of the projection method seems dubious, see questions below.", "questions_to_authors": "- Section 3.1 the PAP method: the equation provided moves the query vector away from the perspective vector by some amount, it does not project it onto the p space. Additionally, in the term prefixed with the minus sign, there could be an coefficient parameter to tune. Please explain.\n - Table 4: $\\cos(q, c)$ is not a correct notation since $\\cos$ takes 1 parameter. Please write $\\cos\\angle(q, c)$. The `dual-sum` and `tri-sum` lacks intuition to me.\n - Section 2.5: you argue that SimCSE, not specifically trained with any specific task outperforms. However, you stated that an NLI-fine-tuned SimCSE performs better. Please clarify.", "extra_scores": {}}, {"review_id": "1TxjoZUs4F", "reviewer": "Reviewer_HLF6", "summary": "This paper explores a new retrieval setting in which different perspectives of the queries are to be recognized and attended to by the retrieval system. The key requirement is that the retrievers should attempt to address the subtle different perspectives expressed in the queries. A new benchmark for the task (Sec 2) is created, including a new variant of the recall metric (Eq 1) that takes perspectives into account. A new approach is proposed to improve the perspective awareness of a SimCSE baseline, called perspective-aware projection (Sec 3), by performing a perspective/corpus projection over the input embeddings. The main results in Table 5 show that the proposed methods can improve the accuracy and correlation of SimCSE on AmbigQA and Perspectrum datasets respectively by a wide margin.\n\nThis paper is written up carefully, has a clear storyline regarding a key limitation in mainstream retrieval paradigm about perspective fairness. It shows that while SimCSE as a retriever doesn't always do a good job in a perspective-awareness scenraio (Table 5), this side of effectiveness can be further enhanced by employing a geometric projection approach PAP and PAP+. The overall narratives and the experimental effort are interesting and insightful, the methodology is simple and easy to reproduce. \n\nI'm not generally across the fairness line of work for retrieval, but the main point raised in the paper (\"improving perspective awareness\") seems crucial and timely. Technical novelty aside, all three main claims are reasonably demonstrated and fulfilled in separate sections. I think the paper can meet the bar for acceptance and be included in the proceedings.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and detailed feedback and for liking our paper. We would like to address your questions as follows.\n\n**Q1: Beyond geometric approach**\n\nThank you for the valuable insights on the potential methods. In this paper, we focus on introducing the problem and designing the benchmark. We believe continuous effort shall be made to improve retrievers’ perspective awareness. As also mentioned in our response to Reviewer 8C7L, one direction is introducing a tunable parameter to decide how much perspective awareness should be involved per query. \n\nWith the motivation from retrieval diversity work [1][2], we also work on how to automatically mine and generate diverse and contrastive retrieval signals to help train retrievers with high perspective awareness. We will extend the future work discussion in current Section 5 in the final paper.\n\n**Q2: Writing details**\n\nFor the heuristics in creating the datasets, we introduce how we leverage and extend the design of source datasets to create the query-document pairs in Section A.2 in the appendix, with examples in Table 7. We will include a detailed introduction of the key reasons and unpack the mention in the final version.\n\nThe dependency issue is from the difference in the cases in the wild (general case) and in our dataset design (PIR). In the general case, retrievers only observe query (q), external tools can be applied to sample perspectives (p) and root queries (r) from q, such as the generative perspectives we describe in Sec 3.4 \n\nIn PIR,  to test the model perspective awareness, we design the dataset to have multiple p attached to each r. so that in the metric related to PIR (i.e., in Section 2.3), we write q = p+r, specifically for the PIR task. We will add an explanation of the setting differences in the final paper.\n\n𝑞=𝑝+𝑟 describes the construction of our PIR dataset: To ensure that the perspectives are explicit and have contrastive pairs, we first sample perspectives (p) from existing documents (such that we know the gold document to retrieve for each p), then add them to a root query (r), such that r+p forms the final query (q) submitted to the retriever. In real-world scenarios, p and r tend to be implicit and can only be decomposed from q (the only observable data point) – a case we discussed such decomposition in Sec 3.4. We will explain the equation in more detail. \n\n[1] https://plg.uwaterloo.ca/~gvcormac/novelty.pdf\n\n[2] https://ieeexplore.ieee.org/document/7987038", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a new formulation about perspective awareness and some practical dataset and methodology for evaluation. Given that LLMs are now used widely in all sorts of applications, this type of evaluation effort that strengthens model safety and reliability is particularly timely.\n- The paper shows that this new perspective-awareness evaluation criterion is not unreasonably difficult. A geometric projection method can be applied to SimCSE and improve the effectiveness.", "reasons_to_reject": "- On the methodology side, Sec 3 only reviews a simplistic geometric approach. I do not believe technical novelty would be a limiting factor for this type of work, but given that ranking/prediction fairness has been studied for many years in NLP & IR there should be some relevant prior art that we can refer to and compare the proposed method with.", "questions_to_authors": "- In Sec 2.2, you write \"we use heuristics to ensure that all the queries...\" Can you make unpack this mention of heuristics and call out explicitly what you do in the main text?\n- Eq 1 seems to use a cyclic dependency: $q = p + r$ but then $p$ is to be sampled from a set conditioned on $q$.", "extra_scores": {}}, {"review_id": "fVHC0Jif8P", "reviewer": "Reviewer_K88D", "summary": "The three main contributions of this paper are (1) the introduction of a perspective-aware information retrieval (PIR) benchmark by repurposing existing retrieval benchmarks, (2) a method for improving perspective-aware retrieval by projecting queries (and potential the datastore) into perspective space (PAP), and (3) an empirical analysis of off-the-shelf retrievers on PIR and how to build in PAP.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and detailed feedback and for liking our paper. We would like to address your questions as follows.\n\n**Q1:  Downstream Tasks**\n\nWe sincerely thank you for the insights into the choice of downstream tasks. In this paper, we focus on introducing the problem and designing the benchmark. We start with zero-shot PAP to improve current retrievers on the retrieval and downstream tasks that are currently within our included datasets (i.e., AmbigQA and Perspectrum).  \n\nWe believe continuous effort shall be made to improve retrievers’ perspective awareness (e.g., add tunable parameters and mine automatic signals). We will apply the retrievers with high perspective awareness (which PIR can reveal) to other tasks. We will add a section describing the potential application to other tasks as a roadmap for the authors and other researchers, for example, TheoremQA [1]. We will also work on including these tasks in the PIR benchmark.\n\n**Q2:  Writing details**\n\nThanks for pointing out the typos and writing comments. We will update them in the final paper. For the intuition on other questions:\nOne potential reason is that the LLMs can understand “two components” (where one is the “intent”) in the prompt in Appendix A.8. As shown in examples in Section A.1, the user meta-description (e.g., “Find a news article”) and general information (e.g., the source news content) can be easy to distinguish. We are experimenting with a zero-shot setting. We believe demonstrations can further stabilize the performance. We are also open to all prompting suggestions.\n\nThe large improvement over the no-PAP retrieval baseline is also interesting to the authors. We believe this is rooted in the decoupled relation between the retrieval and downstream task performance. The retrievers may find relevant but incorrect documents, which may hurt the instruction-following LLM performance compared to no-retrieval. We introduce some related readings in the last paragraph of Section 3.3. Together with these related papers, this improvement mismatch suggests our takeaway: beyond relevance that makes a retriever successful on retrieval tasks, we should be careful of the correctness of responding to the perspectives of user queries to make the whole RAG pipeline successful.\n\n[1] Wenhu Chen, Ming Yin, Max Ku, Elaine Wan, Xueguang Ma, Jianyu Xu, Tony Xia, Xinyi\nWang, and Pan Lu. 2023. Theoremqa: A theorem-driven question answering dataset. In EMNLP 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written and the experiments are thorough.\n* The proposed problem and benchmark are timely, as many works are concurrently identifying shortcomings in retrieval methods.\n* The perspective projection method is also simple and intuitive.", "reasons_to_reject": "The empirical improvement from incorporating the PAP project method is modest, both for recall and downstream tasks. My stance is that the authors have not found the best downstream task to really highlight their method and that this paper is one or two experiments away from being a strong paper. Perspectrum seems to be on the right track -- I suspect the downstream tasks that will benefit the most from perspective-aware retrieval will be reasoning-intensive (rather than knowledge-intensive) and require comparing multiple perspectives. One could imagine trying to prove a theorem by retrieving lemmas with different perspectives, e.g. geometric or algebraic.\n\nI am open to being convinced that the improvement in downstream tasks is significant.", "questions_to_authors": "1. Bottom of page 6: \"PAP variances\" => variants\n2. The intent perspective extraction prompt in appendix A.8 is simple, and I would expect \"intent perspective\" to be a rare bigram. I am therefore surprised the performance is high for generating perspectives from queries. Do you have any intuition for why this works?\n3. The improvement over the no-PAP retrieval baseline in Perspectrum is large, but the improvement over the no retrieval baseline is small. Can you explain this?", "extra_scores": {}}, {"review_id": "C8l4T1yv28", "reviewer": "Reviewer_CcDN", "summary": "This work introduces an additional intrinsic component—perspective—to the information retrieval process and presents a new benchmark, PIR. It also introduces a zero-shot, projection-based method called PAP. Experimental results show that current retrievers have limited awareness of subtly different perspectives in queries and can also be biased toward certain perspectives. Meanwhile, the method PAP outperforms other baseline methods in multiple settings and can enhance the perspective sensitivity of the retrieval process in a zero-shot manner without additional fine-tuning. Furthermore, perspective-aware retrieval can significantly improve the performance of downstream tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your valuable and detailed feedback. We would like to address your questions as follows.\n\n**Q1: Notations**\n\nThank you for the detailed suggestions on the notations. We will update the final version accordingly.\n\n1. Neutral describes the root queries in the PIR task, which is contextually used to describe the non-perspective part of the user query.\n\n2. q = p+r  is q = (q,r). We will unify the notation and add an explanation.\n\n3. S is a typo of the corpus C. \n\n4. The knowledge part refers to the prompt in Appendix Section A.8 used for the experiments in Section 3.3, as mentioned in the last sentence of the second paragraph in Section 3.3. \n\n**Q2: Generative Perspectives for Retrieval**\n\nThanks for the suggestions. We extend the experiment with the generative perspectives (50 queries per task, full corpus) on retrieval tasks to test the performance consistency, with Recall@5 as the metric. For each retriever (e.g., BM25), A - B denotes the retrieval performance with annotated perspectives (A) and generative perspectives (B)\n\n| Dataset | QA-F1| BM25 | DPR | SimCSE-sup |\n| :---: | :---: | :---:| :---:| :---:|\nAGNews | 78 |12 - 12 | 4 - 4 | 8 - 8 |\nStoryAnalogy | 66 | 56 - 60 | 26 - 32 | 36 - 38 |\nPerspectrum |\t69 | 26 - 28 | 38 - 38 | 50 - 50 |\nAmbigQA | 70 | 21 - 18 | 46 - 44 |44 - 44 |\nAllsides | 66 | 6 - 8 | 8 - 10 | 8 - 8 |\nEx-FEVER| 69 | 80 - 82 | 60 - 60 | 74 - 72 |\n\nFrom the table above we can observe that the retrieval performances from annotated and generative perspective extraction are similar across tasks and retrievers, which validates our claims in Section 3.4.\n\n**Q3: Existing research on cosine similarity**\n\nThe authors indeed read the referred paper and found it interesting. It was put on ArXiv on March 8 and would be considered concurrent work under COLM policy, but we still intend to add a discussion in the final paper: We use cosine similarity for fair comparison since current retrievers such as DPR and Contriever are tuned on similarity from the cosine similarity function.\n\n**Q4: Few-shot IR**\n\nWe speculate that the proposed PAP method can benefit from the few-shot IR setting. As in our discussion with Reviewer 8C7L, we can design a tunable parameter to decide how much perspective awareness to be involved per query, which can be tuned with the shots given to perform task-specific PAP. \n\nAnother potential benefit is to find the task-specific perspectives to improve computational efficiency, as described in Section 3.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The underlying motivation for this research is compelling and well-founded.\n2.The methodology introduced is both straightforward and efficacious.\n3.This study establishes a new benchmark that is thoroughly examined and analyzed in detail.", "reasons_to_reject": "1.Certain sections of the text lack clarity, and the logical progression appears flawed. We will address specific questions regarding these issues in the Questions section.\n2.While Section 3.4 employs GPT-3.5 to generate perspectives for QA downstream tasks, experiments do not extend to retrieval tasks. It would be advantageous to include retrieval experiments using perspectives generated by GPT-3.5 in Table 4 as well.\n3.The manuscript lacks a comparative analysis with existing research on cosine similarity. It would be beneficial for the authors to conduct comparative experiments with the methodologies outlined in https://arxiv.org/abs/2403.05440.", "questions_to_authors": "1. Regarding Weakness 1:\n-It is unclear whether the terms 'root queries' and 'neutral queries' used in the abstract refer to the same concept. Could you please clarify?\n-In the concluding sentence of Section 2.1, is the 'q' in \"For each q\" intended to represent the tuple (p, r)? Does this notation imply both an implicit 'p' within 'q' and an extracted 'p'?\n-Are the formulations 'q = (p, r)' and 'q = p + r' mentioned in Section 2.1 and Equation 1 consistent with their respective descriptions in Table 4?\n-What is denoted by 'S' in the final sentence of Section 2.3?\n-Could you define what is meant by 'knowledge part' and 'gold knowledge' as referenced in the first sentence of the third paragraph of Section 3.3?\n2. The study primarily investigates zero-shot information retrieval (IR). Could you speculate on the potential outcomes or adjustments necessary for few-shot IR scenarios?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper defines a benchmark for perspective-aware retrieval by processing six existing IR datasets into a format of queries separated into a root and perspective portions, and corresponding relevant documents. The authors study the performance of 5 off-the-shelf retrievers and analyze accuracy for the generic and perspective-aware tasks, indicating interesting inaccuracies and biases of the models.

The authors also propose a method to derive query and document embeddings from off-the-shelf retrievers to make the representation more perspective-aware. Even though the transformations are not intuitive, they lead to improvements in average performance, as measured on retrieval metrics and end-tasks of retrieval-augmented generation for two tasks on a small number of examples. The improvements are larger in the end tasks although the sample size is small and the evaluation on one of the tasks is a bit unconvincing, as it relies on an automatic measure of sentiment.

Pros
 * the paper focuses on an important problem
 * the results based on the new datasets are interesting and provide insights into directions for future improvement of retrieval systems
 * the presentation is good, except for the motivation of the query/document projection method
 * the authors have promised to fix the notation in the mathematical development and added new useful results based on the reviewer comments

Cons
  * the corpora used for the retrievers are very small, with the largest one having 2900 documents 
   * the representation projection method and its performance are not analyzed very well
   * manual evaluation would be more convincing for the perspective-aware Perspectrum RAG downstream task
Decision: Accept