Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aim at overcoming the drawback of multi-view representation models for dense-vector based open domain retrieval or question answering. A context-adaptive multi-view representation learning framework is proposed to avoid the collapse of representations in different views by adaptively aligning each viewer token with each document snippet. The answer snippets for positive documents are specified in a supervised-learning setting and the representations are able to attend to local snippets, this additionally enhances the interpretability for each view representation. The experimental results on multiple reading comprehension and question answering datasets show that the proposed approach outperforms state-of-the-art methods. 




strength_and_weaknesses: Strength

1. This paper focuses on an interesting and impactful problem, that is the multi-view representations tend to collapse into the same one when the percentage of documents answering multiple queries in training data is low. Resolving this problem is critical to reading comprehension and question answering tasks. 

2. The proposed context-adaptive multi-view representation model is reasonable and empirically effective. 

Weakness:

1. Missing technical details:
(1) Are the results, for example, in Table 1 statistically significant? It'd be great t-test results could be presented, in order to make the conclusion of this paper more convincing. 
(2) It seems the proposed method introduces a hyper-parameter, that is the number of snippets to divide a document. More discussions would be interesting. 

clarity,_quality,_novelty_and_reproducibility: This paper is well motivated to learn distinguished representations for multi-views. The diagrams explicitly illustrate the difference between the proposed method and prior methods in terms of the model architecture. 

The idea of aligning each viewer token with different document snippets is intuitive. However, given the prior work MVR (Zhang et al., 2022), the novelty of this paper is thin. The method proposed in this paper seems incremental to existing work. 

The paper discusses how the parameters and hyper-parameters are set, hence the results should be reproducible. 
summary_of_the_review: This paper studies the collapse problem that multi-vector models have in learning identical representations in different views. A simple and effective method is proposed to learn multi-view representations that attend to the local snippets of a document. 

In general, the proposed method is well-motivated and reasonable. However, the paper has not presented critical technical details, including how significant are the results statistically and how exactly a document is divided into multiple short snippets by NLTK toolkit. 

The contribution of this paper is incremental given prior works have presented many insights such as how to apply hard negatives.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to improve information retrieval models by introducing a new way of multi-representation learning, CAMVR. CAMVR extends MVR approach that added multi-view tokens at the beginning of the text inputs; instead, CAMVR adds the multi-view tokens in between the segments of text inputs, encouraging each multi-view representation to capture local information. Combined with the new loss function to prevent the collapsing multi-view representation problem, CAMVR were able to provide diverse document representations and demonstrated good performance in the common IR benchmark datasets.

strength_and_weaknesses: **Strength**
* Simple and practical approach.
* Provides multiple analysis to examine the multi-views indeed are not collapsing.

**Weakness**
* Novelty could be questioned for some reviewers because it does feel it’s quite a simple extension of MVR paper.
* There is no principled way of splitting the text even though splitting can play a critical role in the proposal. Input text was arbitrarily split with a toolkit (NLTK) and it does not seem to be based on solid assumptions.
* Table 5 shows using 4 representations does not outperform other single-representation models (e.g. RocketQA/DPR-PAQ), which is a bit surprising. Only by using more than 8, it was able to outperform. Any discussions?

clarity,_quality,_novelty_and_reproducibility: **Clarity/Quality**
* Paper’s clarity and quality are in doubt. Paper has multiple typos and sections that require substantial polishing. For example, (13) does have a floating “i” variable and the connection to the per-view loss to the full loss function is not described. (7) has a form of the main loss function, but it is in the “Preliminary” section. Figure 4 has a typo. RAMVR->CAMVR. I encourage authors to improve the paper's clarity further.
* Even though snippet splitting can play a critical role in CAMVR, authors did not disclose information regarding the split. Authors just meant the split was by NLTK toolkit, but I am not sure how they exactly splitted. Please provide details.
* Important experiment details are missing. Authors should disclose models that they used in Section 4. (e.g. whether the model was pretrained, size, etc). I feel that the paper was rushed for the submission.

**Novelty**
CAMVR is a direct extension of MVR paper [1]. It essentially adds two minor contributions 1) inserting view tokens in between text segments unlike MVR, 2) adding the diversity loss function to prevent all multi-view representations to collapse. Both of the techniques do feel a minor addition to the original MVR paper.

[1] Zhang, Shunyu, et al. "Multi-View Document Representation Learning for Open-Domain Dense Retrieval." arXiv preprint arXiv:2203.08372 (2022).


summary_of_the_review: As discussed above, the paper provides a simple practical approach in IR. However, the novelty is thin and paper does require further polishing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors improve multi-view representation learning by splitting documents and aligning viewer tokens along with different parts. The experiments conducted on three datasets demonstrate that the proposed CAMVR performs better than MVR. The analysis also shows that CAMVR has better interpretability. 
strength_and_weaknesses: Strengths
* Important research topic.
* Publicly available datasets.
* Well-written and easy-to-follow.


Weaknesses
* The idea of aligning special tokens along with different parts in the document is not novel in the retrieval field. For example, [a] aligns special tokens along with different sentences. The authors should also consider comparing this line of existing studies.
* The semantic meaning of viewer tokens can become different after using the proposed arrangement. There may not be different viewer tokens for a specific span or at fair positions attending all documents. 
* Following the previous point, many of the experiments and analysis are unfair to the original MVR because it has no such capability of doing that thing. The authors should compare valid methods like [a] for fair experiments.

[a] Jiang, J. Y., Xiong, C., Lee, C. J., & Wang, W. (2020, November). Long Document Ranking with Query-Directed Sparse Transformer. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 4594-4605).
clarity,_quality,_novelty_and_reproducibility: * For clarity, the paper is well-written and easy-to-follow.
* For novelty and technical quality, the proposed idea is not novel, and the experiments do not have enough technical quality to validate the proposed idea and have fair comparisons. 
* As mentioned in weakness, the paper does not provide the information about reproducibility.

summary_of_the_review: In sum, I would recommend “3: reject, not good enough” because the paper is not novel, and has some flaws in the experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The paper has a neat contribution that improves MVR but the reviewers were not excited about the paper, mostly due to weaknesses as mentioned below. While the authors were able to improve the paper further during the rebuttal, it is not clear if the paper will have enough interested audience in the conference.

Strengths:
- Most reviewers agree that the problem is interesting and important.
- Most reviewers agree that the proposed method is reasonable and effective in the given datasets.

Weaknesses:
- Most reviewers think that the proposed method is a relatively simple extension of previous work (e.g. MVR). 
- Some reviewers think that comparing with MVR is not fair and there should be more appropriate baselines.
- Many reviewers are concerned that the paper is not clear enough.
justification_for_why_not_higher_score: While the problem is an important problem in the community, the contributions of the paper was not very new given what the previous work has presented with regard to applying hard negatives.
Decision: Reject