OurShash/MedIR-RAG-Milestone1

MedIR-RAG - Milestone 1: BM25 lexical baseline for biomedical retrieval

A. Project overview

This repository is the completed Milestone 1 of a biomedical document retrieval system, written for the course Recuperação de Informação em Bases de Texto (RIBT) at the University of Évora.

The system answers a retrieval question, not a medical one: given a question in natural language, it returns the PubMed titles and abstracts of the corpus that are relevant to that question, ranked best first. It retrieves documents; it does not generate or summarise medical answers. The ranking model is BM25 over a Terrier inverted index, used through PyTerrier.

The work follows the four parts required by the assignment: preprocessing and tokenisation of a biomedical corpus, indexing with a sparse retrieval library, a parameterised BM25 baseline, and quantitative evaluation with pytrec_eval on NDCG@5, NDCG@10, MRR@10, MAP and Recall@100 using human relevance judgments.

The Terrier index is included in the repository, so the search command in section G works directly after cloning, without downloading the dataset or rebuilding anything.

Repository contents

README.md                          this file
requirements.txt                   pinned Python dependencies
src/prepare.py                     dataset download, validation and audit
src/experiment.py                  indexing, BM25 tuning, test evaluation, efficiency
src/search.py                      command-line search over the saved index
analysis/build_evidence.py         tables, figures, statistics and failure cases
index/terrier_bm25/                the complete saved Terrier index (10 files)
results/raw/data_audit.json        dataset validation and counts
results/raw/dev_grid.csv           the 20 development configurations
results/raw/selected_config.json   frozen BM25 parameters
results/raw/test_summary.json      test metrics, cross-checks, sensitivity
results/raw/test_per_query.csv     per-query metrics for all 1,145 test questions
results/raw/metric_validation.json metric self-tests on synthetic rankings
results/raw/experiment_manifest.json settings, versions, index statistics, timings

Not included, because they are generated locally or are large: the downloaded dataset (data/), the TREC run files (about 70 MB each, produced by src/experiment.py), and the evidence figures (produced by analysis/build_evidence.py). The Java runtime is not included either; section F explains how to provide one.

B. Dataset

Item	Value
Dataset	DinoStackAI/bioasq-rag-13b-resplit
Pinned revision	ade805f1df0b16dbf19385597861146bdb0d9904
Licence	CC-BY-2.5
Subsets used	corpus, queries, qrels

python -m src.prepare downloads this dataset automatically from the Hugging Face Hub at the pinned revision, validates its schema and identifier integrity, writes the Parquet splits to data/, and records an audit in results/raw/data_audit.json. The answers and retrieved_docs subsets are deliberately not used: this milestone ranks documents, it does not generate answers.

Verified counts (from results/raw/data_audit.json):

Item	Count
Published corpus records	44,183
Indexable records	44,128
Records without a usable title or abstract	55
Train questions	3,668
Development questions	916
Test questions	1,145
Test questions with relevance judgments used for the main evaluation	1,073
Positive relevance judgments in the test split	9,478

The 55 excluded records have neither a title nor an abstract, so there is no text to index; nine test judgments point at them, which puts a small unavoidable ceiling on Recall@100.

What this corpus is and is not. The source is a community-prepared, BioASQ-derived fixed-corpus retrieval benchmark: PubMed titles and abstracts, with BioASQ question/answer pairs whose questions the dataset publisher re-merged and reshuffled into train, development and test splits. It is not the full PubMed collection, and the queries are not the official BioASQ challenge splits, so the scores reported here are internally consistent but not comparable to published BioASQ leaderboard results.

Attribution: the dataset was prepared by DinoStackAI and is distributed under CC-BY-2.5; the underlying questions and judgments come from the BioASQ challenge (Tsatsaronis et al., 2015, see section J), and the documents are PubMed records.

C. Preprocessing and indexing

Text preparation follows the standard Terrier English pipeline:

1. Fields. The indexed text is title + abstract. Title and abstract are also stored as metadata fields (400 and 4,000 characters) together with the document identifier, so search results can show titles and previews without opening the dataset.
2. Tokenisation. Terrier's EnglishTokeniser (lowercasing, splitting on non-alphanumeric characters).
3. Stop-word removal. Terrier's English stop-word list.
4. Stemming. Porter stemmer, so reducing, reduced and reduces collapse to one term and thyroid matches thyroids.
5. Indexing. The terms are written to a Terrier inverted index with document lengths, term frequencies, a lexicon and a direct index, plus the metadata index.

Verified index statistics (from results/raw/experiment_manifest.json):

Item	Value
Documents indexed	44,128
Distinct terms after stemming	92,366
Postings	3,947,805
Tokens indexed	6,493,411
Mean document length	147 tokens
Index size on disk	59.65 MB
Index build time	12.05 s
Retrieval depth	1,000

The complete index is included under index/terrier_bm25/: data.properties, data.inverted.bf, data.direct.bf, data.lexicon.fsomapfile, data.lexicon.fsomaphash, data.lexicon.fsomapid, data.document.fsarrayfile, data.meta.idx, data.meta.zdata and data.meta-0.fsomapfile. All of them are required: the .bf files hold the compressed postings, the lexicon files map terms to postings, and the data.meta* files hold the document identifiers and the stored title/abstract fields.

D. BM25 retrieval and parameter tuning

BM25 scores a document by summing over the terms of the question:

• Term rarity. A term that appears in few documents is informative (metformin); a term that appears everywhere is not (patients).
• Term frequency, with saturation. More occurrences of a term in a document help, but with diminishing returns, so a document that repeats a word twenty times is not twenty times better. The parameter k1 controls how quickly this saturation sets in.
• Document length. Longer documents contain more words by accident, so their scores are divided down. The parameter b controls how strong this penalty is (b = 0: none, b = 1: full).

Two configurations are reported:

Configuration	k1	b
Standard BM25 default	1.2	0.75
Selected on the development split	0.6	0.5

How the parameters were chosen. A grid of 5 values of k1 (0.6, 0.9, 1.2, 1.5, 1.8) and 4 values of b (0.3, 0.5, 0.75, 0.9) was evaluated on the 916 development questions, 872 of which have judgments. The best development NDCG@10 was 0.8199 at k1 = 0.6, b = 0.5; the default configuration scores 0.8097 on the same split. The selection is frozen in results/raw/selected_config.json before the test split is touched, and the test split is evaluated once. All 20 configurations are in results/raw/dev_grid.csv.

The chosen point sits on the edge of the grid (0.6 is the smallest k1 tested), so the tuned gain is probably not exhausted; the grid was not extended after seeing test results, because that would amount to tuning on the test split.

E. Experimental results

All numbers below are read from results/raw/test_summary.json and were produced by pytrec_eval (the trec_eval bindings) on the 1,073 judged test questions, at retrieval depth 1,000.

Metric	Default BM25 (1.2, 0.75)	Tuned BM25 (0.6, 0.5)
NDCG@5	0.8117	0.8135
NDCG@10	0.8171	0.8248
MRR@10	0.8746	0.8766
MAP@1000	0.7742	0.7839
Recall@100	0.9664	0.9683

What the metrics measure:

• NDCG@5 / NDCG@10 - position-weighted accuracy of the first five and first ten results, normalised so that the ideal ordering of the judged relevant documents scores 1.0.
• MRR@10 - the reciprocal rank of the first relevant document, so 1.0 means it is always at rank 1 and 0.5 means it is usually at rank 2. It is computed on the run truncated at rank 10, not on the untruncated list.
• MAP@1000 - mean average precision with a 1,000-result cutoff; relevant documents ranked below 1,000 do not count.
• Recall@100 - the share of the judged relevant documents found in the first 100 results, ignoring their order.

The improvement is small. Tuning raises NDCG@10 by 0.0077 and MAP by 0.0097, and it is not a uniform gain: across the 1,073 questions, 245 improve, 201 get slightly worse and 627 are unchanged, which can be recomputed from results/raw/test_per_query.csv. A paired bootstrap and a Wilcoxon signed-rank test on these per-query differences are reported in the accompanying technical report. The honest reading is that the standard BM25 default is already a strong baseline on this collection and that tuning k1 and b produces a modest, reliable gain.

These are retrieval scores, not clinical evidence. A high NDCG@10 means the ranking agrees with human judgments about which abstracts are relevant; it says nothing about whether any abstract is medically correct, current, or applicable to a patient.

Sensitivity check: 72 of the 1,145 published test questions have no relevance judgments. They are excluded from the primary population above, and results/raw/test_summary.json also records the variant in which they are retained and score zero (tuned NDCG@10 0.7729).

F. Requirements and installation

Tested with:

• Python 3.12.10 (Windows 11)
• Java: Eclipse Temurin OpenJDK 21.0.12 LTS (any Java 11+ runtime is sufficient for Terrier)
• Packages: see requirements.txt (pyterrier 1.1.2, pytrec_eval-terrier 0.5.11, datasets 5.1.0, pandas 3.0.6, numpy 2.5.4, scipy 1.18.1, matplotlib 3.11.2, ir_measures 0.4.3, pyarrow 26.0.0, huggingface_hub 2.3.0)

Install from the repository root in Windows PowerShell:

py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip
.\.venv\Scripts\python.exe -m pip install -r requirements.txt

Terrier runs on the JVM, so a Java runtime must be visible to Python. If java -version already works, nothing else is needed. Otherwise install a JDK (for example Temurin 21) and point JAVA_HOME at it for the session:

$env:JAVA_HOME = "C:\Program Files\Eclipse Adoptium\jdk-21.0.12.1-hotspot"
$env:PATH = "$env:JAVA_HOME\bin;$env:PATH"
java -version

src/experiment.py and src/search.py use JAVA_HOME when it is set; if it is not, they look for a portable JDK in a .jdk/ directory in the project root (a convenience used during development, not included here). The first PyTerrier call starts the JVM, which takes a few seconds; retrieval itself takes milliseconds per query.

No package is installed into the global Python environment, and no JDK is bundled with this repository.

G. Search using the saved index

From the repository root, with the virtual environment active:

.\.venv\Scripts\python.exe -m src.search --query "Does metformin reduce cardiovascular risk?" --top-k 10

The command loads index/terrier_bm25/ and the frozen parameters in results/raw/selected_config.json (k1 = 0.6, b = 0.5) and prints, for each result, the rank, the BM25 score, the PubMed identifier, the paper title, a preview of the abstract and a PubMed link:

query: Does metformin reduce cardiovascular risk?
model: BM25 k1=0.6 b=0.5 (selected on dev)
results: 10

 1. [25.7273] PMID 25954799
    Type 2 diabetes and metformin. First choice for monotherapy: weak evidence of ...
    Many guidelines recommend metformin as first-line therapy for patients with type 2 ...
    https://pubmed.ncbi.nlm.nih.gov/25954799/

 2. [25.2981] PMID 15529521
    Avandamet: combined metformin-rosiglitazone treatment for insulin resistance ...
    Insulin resistance is a major endocrinopathy underlying the development of ...
    https://pubmed.ncbi.nlm.nih.gov/15529521/

(the output continues to result 10; two entries are shown here, with the titles and previews shortened for readability)

The other options are --top-k (default 10), and --k1 / --b to override the frozen parameters in order to compare configurations by hand. The output is a ranked list of documents, not a medical answer.

H. Reproducing the complete experiment

The full pipeline is three commands, run from the repository root in order:

.\.venv\Scripts\python.exe -m src.prepare          # download and prepare the pinned dataset
.\.venv\Scripts\python.exe -m src.experiment       # index, tune, evaluate, write raw results
.\.venv\Scripts\python.exe -m analysis.build_evidence   # evidence tables, figures and statistics

• src.prepare downloads the dataset at the pinned revision, validates it, writes the Parquet splits to data/ and the audit to results/raw/data_audit.json. This needs network access and roughly the disk space of the corpus parquet file.
• src.experiment validates its own metric implementation on synthetic rankings, then builds the index (or reuses index/terrier_bm25/ if it is present; use --rebuild-index to force a rebuild), runs the 20-configuration grid on the development split, freezes the winner, evaluates both configurations on the test split, times retrieval, and writes results/raw/dev_grid.csv, selected_config.json, test_default.trec, test_tuned.trec, test_per_query.csv, test_summary.json and experiment_manifest.json. The two TREC run files are about 70 MB each and are not included in this repository; this command regenerates them.
• analysis.build_evidence reads the finished experiment outputs, including the TREC run files and the locally generated qrels, re-derives the reported metrics independently, and produces the tables, figures, bootstrap/Wilcoxon statistics and failure-case analysis in results/report/. It cannot reproduce the analysis from the small JSON files alone: it needs the TREC runs and the qrels that src.experiment and src.prepare create locally.

In short: data/ and the TREC runs are download-and-run artefacts, while the index and the seven result files in results/raw/ are included so that the reported numbers and the search command can be inspected without repeating the full run. The evaluation results included here were produced once and are not regenerated on upload.

I. Limitations

• Incomplete relevance judgments. The judgments are binary and sparse: about 79% of the corpus documents are never judged and are treated as non-relevant by TREC-style metrics, so every score in section E is a lower bound on true effectiveness.
• Restricted corpus. 44,128 indexable abstracts from a community-prepared BioASQ-derived benchmark, not the full PubMed collection, and with publisher-reshuffled questions. The numbers are not comparable to BioASQ leaderboard results.
• Missing records. 55 corpus records have no title and no abstract and cannot be indexed; some judgments point at them, which slightly caps Recall@100.
• Lexical retrieval only. BM25 matches words after stemming. It cannot handle synonyms it has not seen, and it breaks on identifiers: for example, the question "What is the mechanism of the drug CRT0066101?" fails because the tokeniser splits the alphanumeric code name, and the relevant abstracts are then never retrieved. Dense or hybrid retrieval is the subject of the following milestone.
• Not clinical advice. This is a research retrieval system evaluated with offline metrics. It must not be used to make medical decisions.

J. References

1. DinoStackAI. bioasq-rag-13b-resplit [dataset], CC-BY-2.5, revision ade805f1. https://huggingface.co/datasets/DinoStackAI/bioasq-rag-13b-resplit
2. Robertson, S. and Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333-389.
3. Macdonald, C., Tonellotto, N., MacAvaney, S. and Ounis, I. (2021). PyTerrier: declarative experimentation in Python. CIKM 2021.
4. Terrier IR platform, version 5.11, University of Glasgow.
5. Van Gysel, C. and de Rijke, M. (2018). Pytrec_eval: an extremely fast Python interface to trec_eval. SIGIR 2018.
6. Tsatsaronis, G. et al. (2015). An overview of the BioASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16:138.