To expand the SchNovel benchmark, researchers can consider several strategies. First, incorporating additional research fields beyond the six currently included (Computer Science, Mathematics, Physics, Quantitative Biology, Quantitative Finance, and Statistics) would enhance the benchmark's diversity. Fields such as Social Sciences, Humanities, Engineering, and Health Sciences could be added, as they often have distinct novelty assessment criteria and publication practices.
Second, expanding the range of publication years is crucial. Currently, the benchmark focuses on papers published 2 to 10 years apart. By including older papers (e.g., 10-20 years) and more recent publications (e.g., within the last year), researchers can better understand how LLMs assess novelty across different temporal contexts. This would also allow for the evaluation of LLMs' adaptability to rapidly evolving fields versus more stable ones.
Third, a systematic sampling method could be employed to ensure that the selected papers represent a balanced distribution of publication years and fields. This could involve stratified sampling techniques to ensure that each field is adequately represented across various publication years.
Finally, collaboration with domain experts in the newly included fields can help curate a more comprehensive dataset, ensuring that the nuances of novelty in those disciplines are accurately captured. This multi-faceted approach would not only enhance the SchNovel benchmark but also provide a more robust framework for evaluating LLMs' capabilities in novelty assessment across diverse academic landscapes.