Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: CSFCube


This paper builds a test collection CSFCube for the "Query by Example" task in IR. QBE tasks take a document as a query and long documents tend to have multiple facets (much more than ambiguous queries). The authors, restrict the QBE task to a faceted QBE task in which users can specify a fine-grained aspect in addition to the query document. The dataset is curated with a scientific literature search use case in mind where the facets are rhetorical content structure units like objective, results, approach, etc,.  
The test collection consists of a smaller set of 50 documents using best practices from TREC i.e. using pooling with high-quality graded relevance scores. The authors also present experiments using term-based methods, pre-trained model-based sentence representations, and abstract-representation models. The abstract representation-based methods exhibit the most superior and the other embedding-based baseline suffer showing off-the-shelf models and approaches are not enough to achieve reasonable performance showcasing the difficulty of the task.




strengths: 
S1: The test collection is carefully constructed using standard IR procedures. 

S2: The error analysis section of the paper is worthy of appreciation. It makes the complexity of the task much clearer. 

S3: The writing is clear and is understandable. 
weaknesses: W1: The baselines considered are somehow simplistic and are not always based on established IR QBD baselines. This somehow also shows the lack of work in this area. This is also a concern in general for the benchmark. It seems like this is a niche IR task and might not garner enough interest from the ML community or the IR community at large. The authors claim the potential importance of the task but do not come up with effective QBD baselines in the literature "modified" for aspect modeling. 

W2: The aim of the test collection/benchmark is to promote interest in ML-based techniques and more specifically representation learning approaches to make progress in QBD tasks. However, it is not sufficiently clear that the scale of the dataset or the query workload is reasonably large to train supervised models. For a first dataset, a smaller dataset means that one has to heavily rely on task-specific modelling rather than transfer learning. The authors should reflect on this limitation of their collection.


W3: Simple baselines seem to be already quite close to the abstract-based representation approaches. On one hand, this means that simple embedding-based or transfer-learning-based approaches are not successful. On the other hand, it is also not clear how existing QBD approaches (modified for aspect-based querying) would compare against simple term matching baselines.
correctness: The claims in the paper are correct to the best of my knowledge 
clarity: The paper is well written and clear for most parts. The evaluation metrics could have been better clarified. Specifically, how were the ranking metrics calculated ? I failed to understand how the relevance assessment would be carried out between a selected passages and the ground truth passage, and how it could be used in the NDCG computation.
relation to prior work: Most of the related work is well covered. 
additional feedback: no additional concerns
review_title: Carefully constructed QBD dataset with unclear impact

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: Authors construct a new test collection for Query By Example (QBE) search. They use abstracts of 50 papers as queries and judge them based on different facets including method, background, and result. In order to select the papers to be judged, they do pooling using methods they developed. In total, they judge 6244 query-document pairs. They also provide performance of 9 different methods.  They release their data and annotation guidelines
strengths: The test collection will be useful for information retrieval (IR) research.
They share their data and annotation guideline.
They claim that they will release the code of baseline methods they implemented after acceptance of the paper. 
Their annotation guideline is comprehensive.
They conduct a qualitative analysis for errors of baselines.
The paper is well-written.

weaknesses: •	The authors provide judgments for 6244 query-document pairs. Considering the time-consuming nature of judging process, the size of the collection is acceptable. However, there are many unjudged documents and many systems which have not participated the pool, might return unjudged documents. Normally, these documents are assumed to be non-relevant. Therefore, I would love to see reusability analysis of the test collection. However, I accept that it is not easy when there is no shared-task to construct the pool. Therefore, at least a discussion about reusability would be good.

•	The authors judge query-document pairs based on abstracts of papers. This sounds reasonable, considering the cost of judging based on all papers. However, this judging might also cause incorrect judgment because the information in abstracts might be limited. Therefore, I would love to see whether judgments based on abstracts are in line with judgments based on full papers. In order to do that they could just sample some query-document papers and do their analysis.

•	The authors report Spearman rank-correlation and agreement ratio to show the agreement on relevance judgments, instead of popular Cohen’s Kappa score. The authors justify this because Cohen’s Kappa does not consider ordinal values but Spearman rank-correlation does. I agree on this and it makes sense. However, Cohen’s Kappa scores are easily interpretable. It is not clear to me whether spearman rank-correlation scores they report are good enough. Instead of reporting just agreement ratio, they could report Cohen’s Kappa.

•	As baseline systems, they use TF-IDF as one of their approaches. I would prefer BM25 since it usually outperforms standard TF-IDF.


•	Their pooling method is not the standard pooling method such that they get 33 or 13 results from each of the retrieval results and at each query the number of papers drawed from a system can change (either 13 or 33). It is not clear to me why they did so. They could use a fixed pool depth such that the total number of judged documents are 100 (or close to 100). This way the evaluation of participating methods would be fair. In this way, it might not be fair.

correctness: There are no major mistakes in terms of correctness. However, there are some issues as mentioned above in Weaknesses section. These are fairness among participants, judging based on abstracts without analyzing its impact and not reporting Cohen’s Kappa score.
clarity: The paper is well-written. There are only a few typos.  

- In section 2: “….pre-defined facets [26, p14], we expect QBE” p14?
- “…. this happen a very rarely, …”
- In Section 7, there are several singular-plural mismatches.
- “from full-text to the citance.”  Citance?
- Colors in Figure 2 are not interpretable. 

relation to prior work: The authors provide a detailed discussion about related work. It seems that their work makes a solid contribution compared to prior work.
additional feedback: In the revision process, the authors addressed all points I raised. I appreciate their effort for these. I changed my rating from 6 to 8.
review_title: A useful test collection for query-by-example task with concerns about its reusability and judgment accuracy

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The dataset lists similar papers for a given query paper, where similarity is seen based on one of three facets (objective, method, result). ground-truth similarity for research papers. After getting candidate similar pairs based on automated means, the pairs have been filtered and annotated manually, with sufficient redundancy and supervision for robustness. The paper is well written.
strengths: Scientific research analysis has become a popular research topic, and this dataset will have an impact in that field. There is a gap in the community in obtaining ground-truth data. The dataset has been made very accessible on github.
weaknesses: The paper is quite strong overall. However, the overall result that baselines perform poorly on the produced dataset, and so, it provides opportunity for improvement to the community, is not fully established in my opinion. This is because there is sufficient disagreement among human annotators too, that needed adjudication stage to rectify. So, it is not clear whether there is a real ground truth at all, or there is sufficient scope for multiple views in the similarity measurement process. Despite this, I think the dataset is a well-needed attempt, and its utility will need to be put to the test of time.
correctness: The dataset is constructed in a sound way. After getting candidate similar pairs based on automated means, the pairs have been filtered and annotated manually, with sufficient redundancy and supervision for robustness.
clarity: Clear. I feel Appendix E or a part of it, is important enough to become part of the main paper, if space can be accommodated.
relation to prior work: Differences from previous contributions are discussed.
additional feedback: 1. Def can be put in its full-form: Definition.
2. In line 145: "The remaining 18..."; but 50-16 is not 18!
review_title: The dataset will help researchers working on scientific research analysis by providing ground-truth similarity for research papers. The paper is well written and their methodology has the required rigour.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes CSFCube, a novel dataset for evaluating query-by-example retrieval methods in the computer science literature domain. Although I appreciate the authors' efforts in producing the dataset and evaluating the methods, the present version of the paper lacks description clarity. Also, the presented repository does not offer convenient means for method benchmarking.
strengths: - A successful non-trivial annotation study has been reported
- Good license choice
weaknesses: - Clarity issues
- Dataset comparison issues
- Documentation issues
- Discussion on scalability is missing
correctness: The authors have reported Spearman correlation coefficients as agreements, which is rarely used for this. Instead, I recommend reporting Krippendorff's alpha to evaluate the inter-rater agreement.
clarity: It was unclear how did the authors compute the NDCG values. The higher the number of elements, the higher its value is. Have the authors used NDCG@k, and if so, the authors should discuss the choice of the `k` parameter. I recommend citing the discussion in the Wang et al. (2013) paper: https://proceedings.mlr.press/v30/Wang13.html.

As the authors have performed manual annotation of the presented dataset, I am interested in reading the discussion on the scalability of the proposed annotation approach. How easily can one create a similar dataset using machine translation or crowdsourcing?

It was also unclear which data items caused the most confusion by the annotators to avoid such issues in the future.

Was it possible to use nearest neighbor search as a baseline?
relation to prior work: In Section 7, the authors mention several related datasets but offer neither qualitative nor quantitative comparison to their contribution. I recommend providing such a comparison.
additional feedback: This paper is a resubmission from the CIKM 2021 Resources Track.
review_title: A useful dataset for a challenging query-by-example retrieval task

Meta-review: This paper presents a dataset of annotated abstracts for faceted query by example (QBE) task. The new task is very interesting, the dataset, albeit not too large, is well annotated and clearly described. This paper will be an important contribution to the field of IR and will serve as a starting point into finer-grain and thus more useful QBE. While some reviewers had some concerns, the author response, as well as the revised paper, addressed most of the concerns.
Decision: Accept