Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new benchmark (SciRepEval) for scientific representation learning consisting of 25 tasks in 4 formats (classification, regression, ranking, and search). It shows that learning a separate document representation for each task format would improve the task performance compared to learning a single representation for all tasks.
strength_and_weaknesses: Strength
- The authors provide a comprehensive analysis on 25 tasks in 4 formats. 
- The authors use strong baseline models to make the results more convincing.

Weakness
- There is not enough information on the tasks in the main content. Would be better to provide some high-level info there and left the majority in the Appendix.
clarity,_quality,_novelty_and_reproducibility: The paper is well written with high clarity and above-average quality.
Some concerns on the Novelty, since the paper doesn't go deep on the technique.
summary_of_the_review: This paper is written in above-average quality and provides a new benchmark with a comprehensive analysis
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a new benchmark for scientific document representations, including 25 tasks across classification, regression, ranking and search. The paper also includes investigations on performances of existing models on the introduced tasks. 
strength_and_weaknesses: Strengths: 
- The paper consolidates useful tasks to make one benchmark for scientific document representations

Weaknesses: 
- The models mentioned in the paper are mostly created by others in the community
- The benchmark, while useful, is a consolidation of tasks that were already existing 

clarity,_quality,_novelty_and_reproducibility: I did not find the paper to be novel, but the quality and clarity of the writing was sufficient
summary_of_the_review: While the paper provides a useful benchmark, there is a strong component of novelty missing from the paper. ICLR does not seem like the correct venue for this paper to be presented.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors of this paper introduce a benchmark for scientific representation learning consisting of 25 tasks in 4 formats (classificaiton, regression, ranking, search) and evaluate multiple general-purpose scientific representation learning methods (SciBERt, SPECTER, SciNL) alongside adaptation methods to learn task-specific and format-specific representations. The results suggest that fine-grained scientific document representations significantly out-perform general-purpose representations.
strength_and_weaknesses: Strengths:
- Introduces a more comprehensive benchmark for scientific document representation learning
- Clearly distinguishes SciRepEval from previous benchmarks (SciDocs) and justifies why a new benchmark is needed (low-power of the recommendation task, a need for more realistic task settings, and lack of training corpus)
- Comprehensive ablation setup targeting fine-grained representation learning, multi-task learning with a strong base model

Weaknesses:
- It would have been nice to see an explicit section detailing what additional challenges remain in benchmarking scientific representation learning and how these limitations apply to the results seen with multi-task approaches.
- Would have liked to see more discussion about task relatedness when discussing benchmarks, especially since SciRepEval seems to be designed in order to improve breadth and coverage of common real-world applications of the representations. For example, it would be interesting to see the relationship as applied to the task of choosing which sub-tasks to group for multi-task learning (see Efficiently Identifying Task Groupings for Multi-Task Learning; Fifty et al. NeurIPS 2021).
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is written clearly and the logic flows well.

Quality: I believe this is a high quality paper with interesting results and strong justification.

Novelty: The authors identify a challenge with existing benchmarks and release a larger and more diverse benchmark. This is an important and novel contribution.

Reproducibility: Data is promised to be released; no mention of code release.
summary_of_the_review: The authors identify an area of opportunity relating to current datasets and methods to evaluate scientific representation learning models and propose a well-grounded, larger benchmark in SciRepEval. The ablation study is comprehensive and there is some interesting discussion of cross-task analysis, although I would have liked to see that section expanded.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces SciRepEval, which is a novel benchmark for training and evaluating scientific document representations. Additionally, the struggles of multi-task learning with regards to generalization in these tasks is explored. An alternative that explicitly encodes the task type is offered and evaluated.

strength_and_weaknesses: Strengths

- Not only are we treated to a single task in this benchmark, but 25, including classification, regression, and recommendation, each of which have appropriate measures for evaluation (though other measures may also have been appropriate).
- Given the variety of tasks, a ‘multi-format’ method of representation learning is proposed which is not entirely novel on its own, but is additively useful within this paper more ostensibly about the benchmarking suite.
- Suitable baselines (SciBERT, SPECTER) and several extensions (e.g., CTRL-code and embedding adapters) provide a suitable evaluation suite.


Weaknesses

- Although much of the information is found in the Appendices, more detail on the tasks themselves are expected within the main body, especially with regards to the source of the data, which could likely be added to Table 1, in some form.
- Tables 2 and 3 and their discussion in the text require some additional context to evaluate these results, especially with regards to the units of measure and highlighting that the first two columns are not exactly comparable with the third (SciDocs). Given the relative similarities of the performance between SPECTER and SCiNCL (especially considering the variants of each), statistical significance tests should be performed.  


Minor

- The use of task-specific control codes is reminiscent (somewhat) of ELMo, and a comparison or acknowledgement should be made to some of the literature around that model.
- Some further explanation around the deficiencies of SciDocs in Sec 2 (especially with regards to the correlations and easiness of negative candidates) really should be provided, especially as SciRepEval’s superiority is meant to be evaluated in contrast to the former.
- There is a higher-than-usual amount of repetitiveness, especially around the 25 tasks in SciRepVal in the first few pages, but this is a very minor complaint
- Some deeper dive into how performance is affected by covariates such as the field of study (Table 8) after one filters out the effect of data set size would be interesting.
- Be sure to check the formatting of your references

clarity,_quality,_novelty_and_reproducibility: - The paper is well-written. It does not go into very much technical depth, though it doesn’t need to as a benchmarking paper, although the nature of the data could have been explored more fully. It is an extension of a previous benchmark in this space; while it is a substantial extension, this still limits its novelty. 

summary_of_the_review: This paper extends an existing benchmark substantially, and offers a new approach to multi-task learning meant to deal with the multitude of tasks in a way that is not wholly novel, and beats the baseline, although only slightly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new benchmark, SciRepEval, for evaluating scientific document representation models. SciRepEval contains 11 additional tasks compared to the benchmark SciDocs. The paper also argues the necessity of developing the new benchmark and supports its arguments by demonstrating the performance loss of the SOTA scientific document representation models on SciRepEval compared to the SciDocs benchmark. The authors also did a good job addressing the reviewers' comments in the rebuttal. However,  besides introducing additional tasks in the new dataset, there is not much novel contribution.


justification_for_why_not_higher_score: The paper presents a new benchmark and is well-written. Besides introducing additional tasks in the new dataset, there is not much novel contribution.

justification_for_why_not_lower_score: The new benchmark is important for improving scientific document representation and modelling on the most commonly used tasks.
Decision: Reject