CAMVR: Context-Adaptive Multi-View Representation Learning for Dense Retrieval
Authors: zhilin liao
XIANGTING HOU
Dongfang Lou
Ningyu Zhang
Huajun Chen
Venue: Submitted to ICLR 2023 (2023)

Abstract: The recently proposed MVR (Multi-View Representation) model achieves remarkable performance in open-domain dense retrieval. In MVR, the document can match with multi-view queries by encoding the document into multiple representations. However, these representations tend to collapse into the same one when the percentage of documents answering multiple queries in training data is low. In this paper, we propose a CAMVR (Context-Adaptive Multi-View Representation) learning framework, which explicitly avoids the collapse problem by aligning each viewer token with different document snippets. In CAMVR, each viewer token is placed before each snippet to capture the local and global information with the consideration that answers of different view queries may scatter in one document. In addition, the view of the snippet containing the answer is used to explicitly supervise the learning process, from which the interpretability of view representation is provided. The extensive experiments show that CAMVR outperforms the existing models and achieves state-of-the-art results.

Under review as a conference paper at ICLR 2023
CAMVR: CONTEXT-ADAPTIVE MULTI-VIEW REPRE- SENTATION LEARNING FOR DENSE RETRIEVAL
Anonymous authors Paper under double-blind review
ABSTRACT
The recently proposed MVR (Multi- Viel Representation) model achieves re- markable performance in open-domain dense retrieval. In MVR, the document can match with multi-view queries by encoding the document into multiple rep- resentations. However, these representations tend to collapse into the same one when the percentage of documents answering multiple queries in training data is low. In this paper, we propose a CAMVR Crontext-Adaptive Multi- View Repre- sentation) learning framework, which explicitly avoids the collapse problem by aligning each viewer token with different document snippets. In CAMVR, each viewer token is placed before each snippet to capture the local and global infor- mation with the consideration that answers of different view queries may scatter in one document. In addition, the view of the snippet containing the answer is used to explicitly supervise the learning process, from which the interpretability of view representation is provided. The extensive experiments show that CAMVR outperforms the existing models and achieves state state-of-the art results.
INTRODUCTION
Dense retrieval approaches based on pre-trained language models (Devlin et al. 2019 Liu et al., 2019) achieve significant retrieval improvements compared with sparse bag-of-words representation approaches [Jones] 1972 Robertson et al. 2009). A typical dense retriever usually encodes the doc- ument and query into two separate vector- based representations through dual encoder architecture (Karpukhin et al.) 2020 Lee et al., 2019 Qu et al.) 2021 Xiong et al. 2020), then relevant docu- ments are selected according to the similarity scores between their representations. To this end, the major approaches improve the retrieval performance by learning a high-quality document represen-
tation, including hard negative mining (Zhan et al. [021 Xiong et al.) 2020 Qu et al. 202 and
task-specific pre-training (Gao & Callan 0 al. 2022).
Intuitively, the capacity of a single-vector representation limited (Luan et al., 2020) when the
document is long and corresponds to multi-view queries. Recently, Zhang et al. (2022) proposes a MVR (Multi Miuli Representation) model to improve the representation capacity by encoding a document into multiple representations, and the similarity score between a query and document is determined as the maximum score calculated with their dense representations. However, when the percentage of documents answering multiple queries in training data low, the multiple vector- based representations of each document tend to collapse into the same one. For brevity, we use the average number of queries (AVQ) per document to represent the percentage of documents answering multiple queries in training data. As show in Table when the AVQ value is small (1.0), multi-view representations collapse into the same one and the performance deteriorates sharply in MVR.
Considering that the answers of queries may scatter in different snippets of one document and the viewer tokens in MVR model are located in front of the document, this makes it difficult to perceive the information from the snippets that may contain the scattered answers. As shown in Figure the two answers are located in the 2nd and 4th snippets in example 1, while the two answers are located in the 5th and 6th snippets in example 2. Obviously, MVR will not be able to adaptively capture the different answer information because the positions of viewer tokens are fixed. Well-learned multi-view representations are expected to capture the information from different snippets. In this paper, we propose a CAMVR (Context-Aa Addattie Multi- View Representation) learning framework, which explicitly avoids the collapse problem by aligning viewer tokens with snippets (as illustrated

---

Under review as a conference paper at ICLR 2023
Title orThe Lord Rings: Fellowship Ring
Title Wodd Cup
Decument: hh Lard the Ringe: The Fellmwship of the
Docament: Gprooding thme years to determine which tooms quality for the toumament plase, which is often called the "World
Ring is 2001 tpe Fantaoy adventure film directed by Petur
Jacksor based
first volume
Tolkion's
teams, including the automatioally qualityngg host nationis). cumpete in the toumament plase for the tide at venues pertod about month. by cight national teams. ('immzil have won live times, and they are the only beam have plaved in every loumament. TThhe other World winnots are Germany Iodly with four tille nach.
Finals". "
Locd of the Rines" 1995-1955). oi the Lirst instalment
in "The Lord of Rings series". and was followed by (2002)
and (2000), based the and third volumes
within the bost nationts]
of Rings": (*Stt Middle earth, the story
(rThe World Cup loumnmentts have
the Dark Lord Sauron Haker), who is seeking the One
Ring.
Query:1 I: When was the the rings made!?
Query1: The only country in the wodld to have played in every toumament of world cup football?
Answer1: 2002.
Query2: Who is the bad lort of rings?
Auswer 1: Brazil,
Answer2: Saurca.
Query2: Italy won the world cup many times?
Answer 2: four
Model Input
Model Iaput
MV R. Input
MVR Input
[vE]
[
W,
[e
[ve.]]
W,
w.
CAM Exput
CAMVR R Input
[VIE.]]
[V]e_]]
2001
[VIE_]
Sauron
[ve_]]
W. [VE.
Brazil
[VIE.]]
Rour
Figure 1: Two examples from Natural Question Dataset and the input comparison of MVR and CAMVR. [V]E,] is a viewer token and the answers are marked red. Compared with MVR model where the viewer tokens are located before the document, our CAMVR model is able to adaptively capture the information of answer snippets by aligning each viewer token with each snippet.
in Figure D- Firstly, to allow views to adaptively capture the information of answer snippets in document, we locate each viewer token before each snippet. Therefore, each view representation able aggregate local and global information via the selffattention mechanism of transformer Vaswani et al. 2017). Furthermore, we use the view corresponding to the answer snippet of a given query to explicitly supervise the learning task, which allows the view to attend to the corresponding local snippet, thereby increasing the view interpretability.
The contributions of this paper are as follows:
Different from MVR model, where the representations would collapse into the same one when the percentage of documents answering multiple queries in training data is low, our proposed CAMVR learning framework is able adaptively align each viewer token with each document snippet to capture fine-grained information of local snippets and global context, which effectively avoid the collapse problem.
We specifyy the answer snippet for positive document to supervise the learning process and allows the representation to attend to local snippets, which provides the interpretability for each view representation.
Extensive experiments are conducted to evaluate our proposed CAMVR in terms of the analysis of overall retrieval performance, collapse problem, view interpretability and view number, etc. The experimental results on open-domain retrieval datasets show the effec- tiveness of our proposed model, which achieves the state-of-the the-hhe-tt art performance.
RELATED WORK
In this section, we review the existing three model architectures for dense retrieval: dual encoder, late interaction and multi-vector model respectively. As shown in Figure 2 late interaction and multi-vector model are the variants of dual encoder.
Dual encoder (Karpukhin et al. 2020) (Figure 2(a) is widely used in the first-stage document re- trieval to obtain the relevant documents from a large-scale corpus. In general, the pre-trained model BERT (Devlin et al.) 2019) is utilized as a base model for dual encoder considering its text repre- sentation ability learned from massive data. To improve retrieval performance, [ee et al. (2019) and Chang al. (020) further pre-train the BERT with inverse cloze task, body first selection and wiki link prediction. To make the language models be ready for dense retrieval tasks, Gao & Callan 2002a b) propose pre-training architectures coCondense and Condenser, where contrastive learn-
2

---

Under review as a conference paper at ICLR 2023
Sum maxis im
maxiim
maxsim
maxsim
1215
[CLS]
Query Encoder
Doc Encoder
Query Encoder
Doc Encoder
Query Encoder
Doc Encoder
MEE
[VI
VIE,
B..
(a) Dual encoder (DPR)
(b) Late interaction (CoIBERT)
(c)
Mulit-vector model (MVR)
Figure 2:
The comparison of different model architectures designed for dense retrieval.
ing task is additional considered coCondenser. Besides pre-training, negative sampling is adopted
to improve retrieval performance by mining high-quality negatives. DPR (Karpukhin et al., (020)
picks out hard negatives through the BM25 retriever. Yang et al. (2017) and Xiong et al. (2020) gen- erate hard negatives dynamically with the newest checkpoint in training process. Qu al. (0011)
and Ren et al. 220TT) use cross encoder to mine hard negatives.
The interaction between each query and document also used improve retrieval performance by exploiting their token-level relevance (Nogueira & Cho 2019: MacAvaney et al. 2019 Dai &
Callan 2019 Diang et al. 2020). However, such token-level interaction is computation intensive, which is impractical for the first-stage retrieval given large-scale document collection. Therefore, late interaction paradigm (Figure 2(b) is proposed to improve the computational efficiency. In general, each pair of query and document is encoded independently and the similarity between them is computed through fine-grained interaction mechanism. ColBERT (Khattab Zaharia 2020)
Santhanam et al. 2022),, relevance score is determined the sum of the maximum similarity between the tokens of the query and document. Poly-Enne Doly-Encoder (Humeau et al. 2020) provides an attention mechanism to learn the global features of document, which is more effciint than directly using all token-level features of the document. COIL (Gao et al. (021) is proposed to only calculates the scores of the overlapped tokens between query and document reduce the computational costs.
Since the representation capacity of single vector is limited for a long document and late interaction between all tokens still requires high computational costs, multi-vector models (Figure 2 therefore proposed. ME-BERT (Luan et al.) 2020) is proposed to improve retrieval performance by using the first m token embedings the document representation. However, this would lose useful information in the latter part of the document. DRPQ (Tang et al. (2021) encodes document with class centroids by clustering all token embedings. MVR (Zhang al. 2022) encodes a document into multi-view representations and each of them can be used to answer different view queries. However, the multi-view representations would collapse into the same one when the percentage of documents answering multiple queries in training data is low. Though DCSR (Hong et al., 2022) produces multiple vectors for the document, there a gap between training and inference tasks, which limits its retrieval performance.
METHOD
3.1 PRELIMINARY
Dual encoder widely used encode a query (by the query encoder) and document (by the docu- ment encoder) into single vector respectively in open-domain dense retrieval as shown in Figure 2(a) Given query q and document the similarity score function is determined as follows:
f(g. (am(Eq())EEpl
3

---

Under review as a conference paper at ICLR 2023
where (-)-) is the query encoder, (-) is the document encoder and sim(-) is the inner product function. Then, the loss function of dual encoder is defined as
(()((() C = log (pp[[qq,+))/)) + exp(I d//T)
(2)
where dt and dj represent the positive and negative documents for q, and the hyper-parameter scaling factor for model optimization (Sachan et al. 2021).
MVR model takes the dual encoder architecture to encode the query and document into vectors, as shown in Tigure[(c) For the query encoder, the [C'LS] token is replaced with viewer token [VIE) to produce the query representation. Formally, the query representation is given as
E(g) ( qqq [SEP]),
(3)
where E(q) is determined by the vector of viewer token [VIE], [SEP] is a special token in BERT and is the concatenation operation. For the document encoder, viewer tokens [V [VTE,](i
n are placed at the beginning of document to produce multi-view representations:
E (d). E2(d). ......... En(d) E[ [VIE,] d [SEP]),
(4)
where the i-th view representation E; (d) of document is determined by the vector of [V] The similarity score function between the query q and i-th view of the document is defined
fi(q. d) sim (g)). (d)),
(5)
where sim(-) is the inner product function. The overall similarity score of query q and document is aggregated by
f(g. = ff. d)}-
(6)
Different from the loss function of dual encoder, MVR further adds regularization for different views to alleviate the collapse problem. Formally, the final loss function is given as
- log exp(S(a, (o([((a,+)//) d+)/r)++++++++ E, (xp(s(q,d7)/4 A log exp(f d' 1/T) expp(ii(g,
exp(f(q. (ff(gdd)//1) )/7)
(7)
where À is hyper-parameter.
3.2 CONTEXT-ADAPTIEE VILW REPRESENTATION
In MVR model, the input of a viewer token is given
(8) where is the number of viewer tokens, 2, is the token embedding of [VIE,], Po and the position and segment embeddings, respectively. Since Po and o are not altered for all viewer tokens, Equation (8) suggests that all the differences in view representations embedding. From another perspective, Equation (8) can be equivalently rewritten as follows:
e; = U. + Pa + +0 i= 1=
caused from the token
e, = 00 + P: + i i=,,..,,
where Un = Po and p. for = 1,..,, n. That is, the viewer token embedding d0 is fixed, while the position embedding is varying. However, shown in Figure D answer snippets for dif- ferent queries are scattered in document. The views with n fixed position embeddings in Equation (9) are not able to capture the information of fine- ane-ggained scattered answer snippets for each doc- ument. When the percentage of documents answering multiple queries in training data is low, the information to align the document views to different queries is insufficient. Thus, the multi-view representations tend to collapse into the same one and the performance will deteriorate sharply.
In this paper, we propose a CAMVR (Context-Adaptric Multi- Representation) learning frame- work in which the position embeddings of viewer tokens are adaptively changed with the context of a document. As shown in Figure [ it is crucial for different views to perceive fine- ame-grained informa- tion in different answer snippets. Since each document snippet may answer some queries, we place a viewer token before each snippet, aiming to capture the information of local snippet and global context.
4

---

Under review as a conference paper at ICLR 2023
5 mm
maxsim
Answer view
Max
sim
sim,
sim,
sim,
sim,
MIE]
[VIE
[VIE]]
V
[VIE
[VIE
[VIE,]
Query Encoder
Document Encoder
Document E Encoder
[VIE]
[VE,
S:
[VIE
[E
5,
[VIE
5,
Query
Positive document
Negative document
The architecture of context-adaptive multi-view representation learning framework. sim" denotes the similarity score between the query representation and answer view representation of positive document. The maxsim" denotes the similarity score between the query and one negative document.
Figure 3:
Algorithm Document Splitting
Input: A document d.
Parameter: View number n and empty sentence b.
Output: A sequence S containing snippets.
1: Split d into l sentences 5 [81, -2;--.; 81] by the sent. tokenize function in NLTK toolkit.
2: if k: S n then
3: Add n-k empty sentences b to S and obtain the final snippets S [s1, 82, ........ 8k, b, ..... bj.
4: else
5: while (en(S) do
6: Identify the shortest sentence s; in S.
7: Merge 8, with its shorter adjacent sentence 8j into a new snippet s/c
8:
9: Remove 8y from S.
10: end while
11:
end if
To guarantee the sentence completeness of each snippet, we design heuristic method to partition document into snippets, and the details are described Algorithm [ For the document d, we can obtain the n snippets ( 8n] with the document splitting algorithm. Then the input context of query and document can be given as follows:
q: [VIE] o oo [SEP].
d: [VTE]] o 81 ... [VIFA) 8, o [SEP].
We treat viewer tokens and document tokens equally in position embedding, and let the viewer token embedding be the same with that of MVR model. Thus, the input of [V _E, is given as
e U Pa. o, = ny ag [0, L]
(10)
where a; is the i-th viewer position in the sequence and L is the maximum sequence length. The position embedding is same with that of vanilla pre-trained model, mainly considering its capability perceiving local and global information which is proven in MLM (Masked Language Model)
task. In addition, different viewer token embeddings help to distinguish adjacent views, making their representations complementary and avoiding the collapse problem.
We use the last hidden states of dual encoder as the representation of the query and document,
E(q) Eq q [SEP]).
(11)
En (d), E2(d). E, (d) ED ED(VVE) 81 [VIE,) s, [SEP)),
(12)

---

Under review as a conference paper at ICLR 2023
where E(q) is the query representation, and (i 1.2....,n. is i-th view representation of the document. The definition of similarity function for query and document refers to MVR model.
In our method, each view representation contains the information of local snippets and global con- text. To make it attend more to local snippets, we specify the view answer snippet to align with the query in training process, shown in Figure Let the i-th view be the answer view, then the loss function of CAMVR is defined as follows:
xxp(fiq,d")// 1/T)
C log
(13)
exp(fiqq, = expl
where fi(q, d is the similarity score between the query and answer view positive document, and used as the similarity score between the query and positive document in training process.
In inference, the similarity score is given by maximizing over the similarity scores between the query representation and document multi-view representations. We firstly use the document encoder of our model to encode all documents in the retrieval corpus into multi-view representations and build the index for each document view. Then we retrieve the relevant documents for the given query, and boost the retrieval process by the ANN (Approximate Nearest Neighbor) technique (Johnson et al.) 2019). In contrast, the time complexity of CAMVR is the same with MVR, and lower than other multi-vector models such as DRPQ (Tang et al. 202I) and DSCR (Hong et al. (022).
4 EXPERIMENTS
4.1 DATASETS
SQuAD (Rajpurkar et al. 2016) is a crowdsourced reading comprchension dataset. The version we used in this paper created by DPR (Karpukhin et al. (2200) and it contains 70k training data. Since the test data is not released, we perform the retrieval evaluations in terms top 5/20/100 accuracy on the development set.
Natural Question (NQ) (Kwiatkowski et al. 2019) is a large dataset for open open-domain QA. All the queries scraped from the Google search engine and anonymized. The documents collected from Wikipedia articles. According to DPR (Karpukhin et al. 2020), 60k training data of NQ is
used.
TriviaQA (Joshi et al.) 2017) is a reading comprehension dataset containing over 650k query- answer-document triples. All the documents are collected from Wikipedia and Web. In our ex- periments, we leverage the version released by DPR (Karpukhin et al. 2020) which contains 60k training data.
The retrieval corpus used in our experiments contains 21,015,324 documents. According to DPR Karpukhin et al. 2020), all the documents are non-overlapping chunks of 100 words.
4.2 IMPLEMENTATION DETAILS
We use 2 NVIDIA Tesla A100 GPUs (with 40G RAM) to train MVR and CAMVR models with a batch size of 56 on each GPU. To save the computational resources, the length of query and document are
128 and 256, respectively. In addition, automatic mixed precision and gradient checkpoints (Chen et al., 2016) are used. We use the Adam optimizer with a learning rate of le-5, and the dropout rate is
O.I. Other hyperparameter settings follow Zhang al. (2022). To make a fair comparison with baseline models, we follow the mined hard negatives strategy adopted in coCondenser (Gao & Callan 202 lab and MVR (Zhang et al., (2022) in Section 4.3 Note that we
use the rre-traine coCondenser as the base model without any further pre-training.
4.3 RETRIEVAL PERFORMANCE
We compare our CAMVR model with the previous ofate-of-the-e art models. These models can be divided into two categories based on the representations of documents: single-vector models and multi-vector models. The single-vector models we compare in our experiments include ANCE Xiong et al. 2020), RocketQA (Qu et al. 2021), Condenser (Gao & Callan 2021b), DPR-PAQ

---

Under review as a conference paper at ICLR 2023
Oguz al, 2022 and coCondenser (Gao & Callan 202 For multi-vector models, consider DRPQ (Tang et al. 2021 and MVR (Zhang et al.) 2022) models for comparison.
As the experimental results shown in Table I our CAMVR model outperforms all the existing mod- els. Specifically, our CAMVR achieves 0.7 (on SQuAD), 0.4 (on NQ), and 3.0 (on Trivia QA) points of top5 accuracy improvements compared with the performance of MVR model. The improvement more significant on Trivia QA. The reason is that the AVQ value of the Trivia QA (1.2) is too small, result, the performance of MVR dercreases since the multi-view representations tend to collapse into the same one. In contrast, the multi-view representations of our CAMVR model can capture the information of local snippet and global context. In conclusion, the improvements on SQuAD, NQ and Triva QA datasets suggest that our CAMVR model is able to perform well for different AVQ values, especially for small AVQ value.
SQuAD R@ 20
Natural Question
Trivia QA
Method
R@5
R@ 100
R@5
@@20
R@ 100
R@5
R@20
R@ 100
BM25
59.1
73.7
66.9
76.7
DPR
76.4
84.8
74.4
85.3
79.3
84.9
ANCE
81.9
87.5
80.3
85.3
RocketQA
74.0
82.7
88.5
Condenser
83.2
88.4
81.9
86.2
DPR-PAQ
74.5
83.7
88.6
DRPQ
80.5
88.6
82.3
88.2
80.5
85.8
coCondenser
75.8
84.3
89.0
76.8
83.2
87.3
coCondenser
73.2
81.8
88.7
75.4
84.1
88.8
76.4
82.7
86.8
MVR
76.4
84.2
89.8
76.2
84.8
89.3
77.1
83.4
87.4
CAMVR
77.1
84.5
88.6
76.6
85.1
89.9
80.1
85.5
88.8
Table I: Retrieval performance on SQuAD dev, Natural Question test and Trivia QA test. The best performing models are marked bold and the results unavailable are left blank. The underlined coCondenser is reproduced by Zhang et al. (2022).
4.4 COLLAPSE ANALYSIS
In this part, we study the collapse problem that the multi-vector models may suffer from. We con- struct new training set with an AVQ value of by extracting the samples from NQ dataset. Then, we compare the retrieval performance between MVR and CAMVR using each view representation alone. shown Table 2 the performances of MVR model using each view representation and its overall accuracy are nearly identical. This indicates that all the view representations collapse into the same one. contrast, the performances of each view representation in our CAMVR are distinguishable, and the overall performance is significantly better than that using each single view representation since the multiple view representations can complement to each other.
AVQ=I, MVR
AVQ=I, CAMVR
NQ
R@5
R@20
R@ 100
R@5
R@20
R@ 100
View
55.62
73.19
84.35
32.55
49.70
66.87
View2
55.51
73.21
84.38
59.28
75.37
84.82
View3
55.40
73.32
84.43
58.92
73.96
84.24
View4
55.40
73.16
84.35
59.17
73.66
83.74
View5
55.60
73.16
84.32
58.09
73.24
83.63
View6
55.54
73.21
84.32
57.73
72.33
81.44
View7
55.37
73.21
84.32
55.40
71.69
83,02
View8
55.60
73.19
84.32
72.58
83.35
overall
55.51
73.16
84.35
63.80
78.70
86.70
Table 2:
Retrieval performance of each view Nature Question test set for MVR and CAMVR under the case that AVQ value of training data is 1.
To learn multiple distinguishable view representations, attention probability vectors views are ex- pected to be different. This means different views should attend different contextual information.

---

Under review as a conference paper at ICLR 2023
We further study the collapse problem by assessing the distance of each pair of attention vectors in the last layer of document encoder. The Kullack-Leible divergence (KL divergence) is used as the distance metrie. Let a; (aq a.2,....; agz)) be the attention probability vector of [VIE;]. The KL divergence between o; and oj is defined as follows:
L E
DK
ag log Oj Cx
(14)
which satisfies that DKL (i.j) 2 O, and (i.( if and only if a; = aj. We randomly select 1000 documents from the retrieval corpus to calculate the average KL divergence between each pair of attention vectors. The results are shown in Table All averaged KL divergence values between viewer tokens are in MVR, which indicates that all viewer tokens attend each token with nearly the same weights. In contrast, all the averaged KI. divergence values of our CAMVR are greater than 0, which indicates that the viewer tokens are able to attend to different contextual information.
MVR
CAMVR
View
2
3
5
7
8
I
3
4
5
7
8
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.50
0.63
0.54
0,42
0.37
0.36
0.33
0.0
0.0
0.0
0.0
0.0
0.0
0.21
0.51
0.62
0.52
0.40
0.36
0.0
0.0
0.0
0.0
0.26
0.65
0.81
0.77
0.69
0.0
0.0
0.0
0.0
0.25
0.59
0.72
0.64
0.0
0.0
0.0
0.20
0.41
0.42
0.0
0.0
0.12
0.20
0.07
Table 3: The KL divergences between attention vectors of viewer tokens for MVR and CAMVR.
4.5 VIEW INTERPRETABILIT ANALYSIS
In this section, we conduct the view interpretability analysis of our CAMVR. Unlike the MVR model, the views are not interpretable because each answer view is selected through the max-pooler operation in training shown as Figure 2(c) For CAMVR, we directly specify the answer view of positive document for each (q, d pair supervise the learning process, which would force each view representation to capture more information of the corresponding snippet. To analyse the interpretability of each view, compare the both overall and the retrieval accuracy exactly achieved by the answer view. For given query 9:. we denote the corresponding top K retrieved documents as [di...., dx], and their corresponding matched snippets are Sx [ms].... msx]. then the answer view accuracy defined as:
N. Σ
ACCK ans
T(at least one snippet in Sg that contains the answer of q:),
(15)
Ng
where N2 is the number of the test queries, I(-) is the indicator function. As shown in Table the gaps between overall performance and answer view performance given the three datasets top 5/20/100 are small, which indicates that the answer view representation successfully captures the information of the corresponding snippet, in which the scattered answers are located at.
SQuAD
Natural Question
Trivia QA
Method
R@5
R @ 20
R@ 100
R@5
R@ 20
R@ 100
R@5
R @ 20
R 100
Overall
69.0
79.2
86.0
71.1
82.5
89.1
77.3
84.1
87.9
Answer View
59.4
70.5
78.7
60.1
73.6
82.2
70.9
80.0
85,4
Other Views
9.6
8.7
7.3
10.0
8.9
6.9
6.4
4.1
2.5
The comparison overall performance and answer view performance for CAMVR on SQUAD, Natural Question and Trivia QA datasets.
Table 4:
8

---

Under review as a conference paper at ICLR 2023
To further illustrate the interetability of view representations, take example from NQ dataset for retrieval evaluation using MVR and CAMVR. As shown in Figure answers of two queries are located at the 3rd and 5th snippets, respectively. For MVR model, the 3rd view is selected for the two queries in terms of time view and number view, which indicates MVR cannot distinguish the two view representations. In contrast, CAMVR is able answer the two queries, where the 3rd and 5th answer views are successfully selected.
Title [VIF A mentinn at the FIFA World Cup
Argentina's results FITA World [VIE Amentin world, having won wodd Caps 1979 1986, [VIF, Agenttnn has boon ronners up thme times in 1500, 1990 and 2014. [VIEg]] The trom was present in all but four of the Wordd Cups. being belind only Brazil, Daly and Germany number of appearazes, IVIE, Argettin has Uraguay. [VIE,] Moreover, Argentina has the
Document: [VIE,1 This is
the
most suocesschl national foothall teams
14 Confederatious Cup and the gold medal. [VIE,]
Copa
Queryl: W Asgentina won FIBA world
Answerl: 1978
Query2: Italy has won the word cup how many times
Answer2: Four
query
view1
vimm2
view3
view4
view3
view6
view?
view@
query!
56.76
56.73
$6.77
56.76
56.75
56.74
56.73
56,72
MVR
query2
54.45
54.56
54.58
54.48
54.46
54.45
54.44
54.43
query!
134.11
133.29
134.59
133.24
133.00
132.40
131.80
19.26
CA MV
query2
13212
132.44
132,38
132.24
132.96
131.61
131.17
21.59
Figure 4: Example of the document retrieved by MVR and CAMVR. Correct answers and selected views are written bold.
4.6 VIEW NUMBER ANALYSIS
We conduct experiments to test the impact of view number on retrieval performance given the three datasets. The results with different view number varying from to 12 are shown in Table [ When the view number increases from to 8, the performance is greatly improved. improvements of top5 accuracy the datasets of SQuAD, Natural Question and Trivia QA are up to 7.7, 2.7 and 1.8 points. The reason that the average number of sentences per document is 6.5 in retrieval corpus, and 1 view or 4 views insufficient to capture the fine-grae fine-grained information of all snippets. The retrieval performance improved slightly when the view number is increased from to 12 since the snippets information answering multi-view queries has almost been captured by the first 8 views.
particular, the
SQuAD
Natural Question
Trivia QA R@20
CAMVR
R@5
R@20 20
R 100
R@5
@@20
R@ 100
R@5
R@ 100
61.3
74.5
83.4
68.4
80.8
87.8
75.5
82.9
86.0
n=4
65.2
76.7
84.8
70.2
82.4
88.7
76.7
83.5
87.8
n=8
69.0
79.2
86.0
71.1
82.5
89.1
77.3
84.1
87.9
n=12
69.0
79.2
86.0
71.9
83.0
89.1
77.8
84.2
87.9
Table 5: Performance of different view number in CAMVR.
5
CONCLUSION
In this paper, we propose a context-adaptive multi-view representation learning framework for dense retrieval. Different from MVR model, where the representations would collapse into the same one with a low percentage of the documents answering multiple queries in training data, our proposed CAMVR learning framework
able to avoid the collapse problem by adaptively aligning each viewer token with each document snippet. In addition, we supervise the learning process by speci- fying the answer snippets for positive documents and allowing the representations to attend to local snippets, this additionally provides the interpretability for each view representation. The experimen- tal results show that our proposed CAMVR is able to avoid the collapse problem and achieves the state-of- of-the-rrt retrieval performance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Wei-Cheng Chang, Felix X Yu, Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for mahhedding-based large-scale retrieval. arXiv preprint arXiv:2002.03932. 2020.
Tianqi Chen, Bing Xu, Chiyuan Zhang. and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:100.00117, 2016.
Zhuyun Dai and Jamie Callan. Deeper text understanding for ir with contextual neural language modeling. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 985-988, 2019.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. (0.18653/11/N19-1422 URL https: 7 /acclannhollogy.orr///
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. ArXiv, 0bs/2108800540,, 2021a.
Luyu Gao and Jamie Callan. Condenser: a pre-training architecture for dense retrieval. In Proceed- ings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 981-993, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. doi: 1118633/11222..mmmm// 2021 www.pennnlmmma.
emmlp-main.755 URL https: httpss////aaaatthoaooooo org/
Luyu Gao, Zhuyun Dai, and Jamie Callan. Coil: Revisit exact lexical match in information retrieval with contextualized inverted list. In NAACL, 2021.
Wu Hong, Zhuosheng Zhang, Jinyuan Wang, and Hai Zhao. Sentence-aware contrastive learning for open-domain passage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1062 1074, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1..18633vv1/2222 long......6 URL https:
7/acl Mlaclanthologe www.og..ogg//220 acl. acl-222..llloooo.oo. 76
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Architec- tures and pre-training strategies for fast and accurate multi-sentence scoring. In ICLR, 2020.
Jyun-Yo Yu Jiang, Chenyan Xiong, Chia-Jung Lee, and Wei Wang. Long document ranking with query-directed sparse transformer. In Findings of the Association for Computational Linguis- fies: EMNLP 2020, pp. 4594-4605, Online, November 2020. Association for Computational http://////22...nnnnneeeeeee orq/2020. findings--minnnn 412
Linguistics. doi:
URL
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. JEEE Transactions Big Data, 7(3):535-547, 2019.
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 1972.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-111 URL https:://///... http://aaaaatthoooooooooo /tt./ttss////eaaaattoooooooo http://aatttooooooooor///1111111
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781 Online, November 2020. Association for Computational Linguistics. doi: 10.11653/11/2200.cmm
URL https: /www.w..ommmmoo acwweb. org arg/anthology/
2020 emn 1p-ma 550
10

---

Under review as a conference paper at ICLR 2023
O. Khattab and Matei A. Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wa Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe V. Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
Kenton Lee, Ming-Wer Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 6086-6096, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.1665//1/P11-1666 URL https: // www acl web. org/
anthology/P19--11
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: /907. 1692, 2019.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. arXiv preprint arXiv:2005.00181, 2020.
Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. Cedr: Contextualized embed- dings for document ranking. Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp. 1 101-1104, 2019.
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert, 2019. URL https://
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Pik- tus, Xilun Chen, Sehastian Riedel, Scott Yih, Sonal Gupta, and Yashar Mehdad. Nomain-matcher pre-training tasks for dense retrieval. In Findings of the Association for Computational Linguis- tics: NAACL 2022, pp. 1524-1534, Seattle, United States, July 2022. Association for Computa- tional Linguistics. URLhttps https::////.mm acl anthology, wtg/202/22222 findings mewddings-naal.....m..mm. naacl. 114
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. ArXiv, abs/2010.08191, 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lunguage Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/ 1264. URL
Bxg/D16-11 1264
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. RocketQAv2: A joint training method for dense passage retrieval and pas- sage re-ranking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2825-2835, Online and Punta Cana, Dominican Republic, Novem- ber 2021. Association for Computational Linguistics. doi: 1011865///2222..mmmpmmmmiii wrt/ wwwee-mainn.n.eeeeeeeee.a
URL
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Retrieval, 3())3333-88, 2009.
Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping. William L. Hamilton, and Bryan Catanzaro. End-to-end training of neural retrievers for open-domain ques- tion answering, ACL-IJCNLP 2021 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Pro- ceedings of the Conference, pp.
2021. doi: 10. 00.18533/v12022 acl-long.519.
11

---

Under review as a conference paper at ICLR 2023
Keshav Santhanam, O. Khattab, Jon Saad-Falcon, Christopher Potts, and Matei A. Zaharia. Col- bertv2: Effective and efficient retrieval via lightweight late interaction. In NAACL, 2022.
Hongyin Tang, Xingwu Sun, Beihong Jin, Jingang Wang, Fuzheng Zhang, and Wei Wu. Improving document representations by generating pscudo query embeddings for dense retrieval. ArXiv, abs/2105.03599, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 6000-6010, 2017. URL http:://p///peerr nips /ctppaaer//11111-ttaatt
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808 2020.
Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Enabling the use of lucene for information re- trieval research. In Proceedings the 40th international ACM SIGIR conference research and development in information retrieval. pp. 1253-12555 1256, 2017.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
Shunyu Zhang, Yaobo Liang. Ming Gong, Daxin Jiang, and Nan Duan. Multi-view document repre- sentation learning for open-domain dense retrieval. In Proceedings the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5090-5990000 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: (0.18655//1/2022.aaa. 414. URL htu.ltpppp//aammtttoooo.ooo////.
A PERFORMANCE SIGNIFICANCE
To demonstrate the significant difference of the retrieval performance between CAMVR and MVR, we implement the two models based on preitrained BERT and coCondenser given three datasets (SQuAD, Natural Question and Trivia QA) with 5 independent runs for each experimental setting. Due to the constraints of time and computational resources, all models are trained without utilizing mined hard negatives strategy. 'he experimental results are shown in Table 6 It is clear that the top5 accuracy of CAMVR is significantly better than MVR with a significance level of 0.05, and the top20 and t0p100 accuracy are also better than comparable to MVR.
SQuAD
Natural Question
Trivia QA
Method
R@5
R@20
R@ 100
R@5
R@
R@ 100
R@5
R@ 20
R@ 100
MVR (coCondenser)
59.1
73.0
82.8
70.0
82.5
89.3
76.4
83.5
87.7
CAMVR (coCondenser)
69.0*
79.2*
86.0*
71.1*
82.5
89.1
77.3*
84.1*
87.9
MVR (BERT)
61.4
73.2
82.5
66.2
79.0
86.6
72.8
81.6
86.5
CAMVR (BERT)
63.0*
74.4*
83.1*
67.6*
80.7*
87.9*
74.5*
82.1
86,9*
Table 6: The performance comparison of MVR and CAMVR based on pre-trained coCondenser and BERT. The asterisk indicates that the result significant with a significance level 0.05, according standard t-test.
B VIEW CAPACITY ANALYSIS
To evaluate the view capacity of capturing global information, we investigate the retrieval perfor- mance with each view as the document representation alone. The results shown in Table 7 demon- strate that each view of CAMVR can also capture the document-level information. Specifically, the top20 accuracy of view view2, view3 and view5 are comparable to the coCondenser (single-vector model), and surpass DPR by at least 9.8 points.
12

---

Under review as a conference paper at ICLR 2023
Natural Question
R@5
R@ 100
DPR
74.4
85.3
coCondenser
75.4
84.1
88.8
View
75.1
84.3
89.5
View2
74.3
84.2
89.6
View3
75.1
84.2
89,4
View4
75.2
84.0
89.5
CAMVR
View5
74.7
84.2
89.1
View6
73.4
83.5
88.9
View7
71.9
82.7
88.8
View8
68.0
79.8
87.6
Overall
76.6
85.1
89.9
Table 7: The view performance comparison between CAMVR and single-vector models.
C HARD NEGATIVES ANALYSIS
Since negative sampling is crucial to improve retrieval performance demonstrated by the existing literatures, we conduct the experiments understand how the hard negatives improve the retrieval performance. To reduce the computational costs in negative sampling, we adopt the two-round train- ing pipeline as coCondenser (Gao Callan 2021a). We firstly train model by using the negative samples obtained via BM25, then the trained model is used to sample hard negatives for each query. Next, we retrain model with the augmented training data containing the BM25 negatives and hard negatives. As shown in Table 8 all retrieval performances of the given three datasets are improved significantly with the hard negatives, especially for the top5 accuracy.
SQuAD
Natural Question
Trivia QA
Method
R 5@5
<@22 20
R@ 100
5
R@ 20
R 100
R@5
@@20
R@ 100
without HN
69.0
79.2
86.0
71.1
82.5
89.1
77.3
84.1
87.9
with HN
77.1
84.5
88.6
76.6
85.1
89.9
80.1
85.5
88.8
Table 8: HN (hard negatives) ablation on CAMVR.
13


Reviews:
[{"review_id": "xrEAKEQzeCg", "reviewer": "Reviewer_tgtb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors improve multi-view representation learning by splitting documents and aligning viewer tokens along with different parts. The experiments conducted on three datasets demonstrate that the proposed CAMVR performs better than MVR. The analysis also shows that CAMVR has better interpretability. ", "summary_of_the_review": "In sum, I would recommend “3: reject, not good enough” because the paper is not novel, and has some flaws in the experiments.\n", "strength_and_weaknesses": "Strengths\n* Important research topic.\n* Publicly available datasets.\n* Well-written and easy-to-follow.\n\n\nWeaknesses\n* The idea of aligning special tokens along with different parts in the document is not novel in the retrieval field. For example, [a] aligns special tokens along with different sentences. The authors should also consider comparing this line of existing studies.\n* The semantic meaning of viewer tokens can become different after using the proposed arrangement. There may not be different viewer tokens for a specific span or at fair positions attending all documents. \n* Following the previous point, many of the experiments and analysis are unfair to the original MVR because it has no such capability of doing that thing. The authors should compare valid methods like [a] for fair experiments.\n\n[a] Jiang, J. Y., Xiong, C., Lee, C. J., & Wang, W. (2020, November). Long Document Ranking with Query-Directed Sparse Transformer. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 4594-4605).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XknkP1bg2C-", "reviewer": "Reviewer_b6Pg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve information retrieval models by introducing a new way of multi-representation learning, CAMVR. CAMVR extends MVR approach that added multi-view tokens at the beginning of the text inputs; instead, CAMVR adds the multi-view tokens in between the segments of text inputs, encouraging each multi-view representation to capture local information. Combined with the new loss function to prevent the collapsing multi-view representation problem, CAMVR were able to provide diverse document representations and demonstrated good performance in the common IR benchmark datasets.\n", "summary_of_the_review": "As discussed above, the paper provides a simple practical approach in IR. However, the novelty is thin and paper does require further polishing.", "strength_and_weaknesses": "**Strength**\n* Simple and practical approach.\n* Provides multiple analysis to examine the multi-views indeed are not collapsing.\n\n**Weakness**\n* Novelty could be questioned for some reviewers because it does feel it’s quite a simple extension of MVR paper.\n* There is no principled way of splitting the text even though splitting can play a critical role in the proposal. Input text was arbitrarily split with a toolkit (NLTK) and it does not seem to be based on solid assumptions.\n* Table 5 shows using 4 representations does not outperform other single-representation models (e.g. RocketQA/DPR-PAQ), which is a bit surprising. Only by using more than 8, it was able to outperform. Any discussions?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WoD5-zOZ2e", "reviewer": "Reviewer_dw1C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aim at overcoming the drawback of multi-view representation models for dense-vector based open domain retrieval or question answering. A context-adaptive multi-view representation learning framework is proposed to avoid the collapse of representations in different views by adaptively aligning each viewer token with each document snippet. The answer snippets for positive documents are specified in a supervised-learning setting and the representations are able to attend to local snippets, this additionally enhances the interpretability for each view representation. The experimental results on multiple reading comprehension and question answering datasets show that the proposed approach outperforms state-of-the-art methods. \n\n\n\n", "summary_of_the_review": "This paper studies the collapse problem that multi-vector models have in learning identical representations in different views. A simple and effective method is proposed to learn multi-view representations that attend to the local snippets of a document. \n\nIn general, the proposed method is well-motivated and reasonable. However, the paper has not presented critical technical details, including how significant are the results statistically and how exactly a document is divided into multiple short snippets by NLTK toolkit. \n\nThe contribution of this paper is incremental given prior works have presented many insights such as how to apply hard negatives. \n", "strength_and_weaknesses": "Strength\n\n1. This paper focuses on an interesting and impactful problem, that is the multi-view representations tend to collapse into the same one when the percentage of documents answering multiple queries in training data is low. Resolving this problem is critical to reading comprehension and question answering tasks. \n\n2. The proposed context-adaptive multi-view representation model is reasonable and empirically effective. \n\nWeakness:\n\n1. Missing technical details:\n(1) Are the results, for example, in Table 1 statistically significant? It'd be great t-test results could be presented, in order to make the conclusion of this paper more convincing. \n(2) It seems the proposed method introduces a hyper-parameter, that is the number of snippets to divide a document. More discussions would be interesting. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject