DOI: 10.5281/zenodo.18431943
title: Webis-SR4ALL-26: A Large-Scale, Cross-Disciplinary Corpus of Systematic Reviews

description: Overview

Webis-SR4ALL-26 is a large-scale corpus of 301,871 systematic reviews spanning 27 scientific disciplines. The corpus is derived from OpenAlex and enriched with structured methodological information extracted from full texts where available.

The dataset is designed to support research on systematic review retrieval, screening, and meta-research across domains. It provides a unified, open infrastructure for large-scale experimentation under consistent indexing and citation conditions.

This release accompanies the paper:

TBD



Files Included

1. sr4all_full.jsonl

This file contains 301,871 systematic reviews in JSON Lines format (one JSON object per line).

Each record includes:

OpenAlex Metadata (available for all reviews)





id (OpenAlex ID)




title




doi




abstract (if available)




year




type




source




cited_by_count




referenced_works_count




referenced_works (OpenAlex IDs)




pdf_url (if available in OpenAlex)




language




field




subfield




topics




keywords




authors



All citation links are derived exclusively from OpenAlex indexed reference lists. No external reference parsing or enrichment was performed.

Structured Methodological Fields (subset only)

For reviews where full text was available and successfully parsed, additional fields may be present:





objective




research_questions




keywords_used




exact_boolean_queries




inclusion_criteria




exclusion_criteria




n_studies_initial




n_studies_final




year_range




year_range_normalized




databases_used




snowballing



These fields are only present where full text was available and the extraction pipeline produced verified outputs. Missing fields are represented as null or absent.

Full texts themselves are not included in this release due to copyright restrictions. Where available, pdf_url links provided by OpenAlex are included for research access.

2. sr4all_normalised_queries.jsonl

This file contains normalized Boolean queries derived from reported search strategies.

Each record includes:





id (OpenAlex ID; maps directly to sr4all_full.jsonl)




boolean_queries (list of normalized Boolean expressions)



Normalization removes database-specific syntax and retains:





Topical terms




Logical operators: AND, OR, NOT



Temporal constraints and metadata filters are not embedded in the Boolean string and must be applied externally.

Only reviews with extractable and normalizable search strategies are included in this file. Queries that could not be normalized are omitted.



Corpus Construction Summary





Initial candidate retrieval from OpenAlex using title-based heuristics.




Deduplication via DOI, OpenAlex ID, and normalized titles.




Metadata filtering (English language, DOI present, non-update reviews).




Integration of benchmark systematic review datasets.




Final corpus size: 301,871 systematic reviews.




Full-text PDFs successfully retrieved and parsed: 72,678.




Structured methodological extraction applied to parsed full texts.




Extraction follows a verify-then-repair pipeline prioritizing precision over recall.





Format

Both files use JSON Lines (.jsonl) format:





UTF-8 encoding




One JSON object per line




No trailing commas




Suitable for streaming and large-scale processing.





What This Corpus Enables

The corpus supports multiple lines of research:

1. Retrieval Evaluation





Execute normalized Boolean queries against OpenAlex.




Compare retrieved results against cited reference sets.




Compute precision, recall, and F-scores.




Analyze cross-domain retrieval behavior.



2. Screening and Ranking





Use reference sets as weak supervision for relevance modeling.




Benchmark citation screening methods at scale.




Study cross-domain generalization of screening models.



3. Query Generation and Reformulation





Learn query structure from extracted expert-written strategies.




Compare LLM-generated queries against author-reported strategies.




Analyze logical structure across disciplines.



4. Meta-Research and Science-of-Science





Study reporting practices (objectives, inclusion criteria, databases).




Analyze disciplinary differences in search strategy formulation.




Examine temporal trends in systematic review methodology.



5. Large-Scale Observational Analysis





Characterize differences in methodological rigor across fields.




Study variation in Boolean complexity and vocabulary.




Analyze inclusion criteria formulations across domains.





Important Limitations





Full texts are not distributed due to copyright restrictions.




Structured fields are only available for reviews with accessible and successfully parsed PDFs.




Extraction prioritizes precision; some valid information may remain unstructured.




Reference sets reflect cited works, not strictly included-study gold standards.




Normalized queries are simplified approximations of original database-specific queries.





Intended Audience





Information retrieval researchers




Systematic review automation researchers




Bibliometrics and science-of-science researchers




LLM researchers working on grounded extraction and query generation




Developers of evidence synthesis tools