CrossGAT — 3-class cross-entropy baseline

Three-class model (irrelevant / excluded / eligible) trained with focal cross-entropy and class weights. The control arm every reported comparison is measured against.

Output classes: irrelevant / excluded / eligible. These are the five cross-validation checkpoints — one per fold, each trained on 80% of the topics and never shown its own fold's.

⚠️ Not self-contained

This model does not take raw text. It scores parsed eligibility graphs — entity/relation structures extracted from trial criteria and patient descriptions by an LLM. The graphs for all 13,229 TREC CT trials are published separately: clinical-trials-eligibility-graphs-rerank.

What it is for

It reranks, it does not retrieve. Trained on an eligibility target, it reorders within the candidate set a first-stage retriever already found. It cannot judge whether a trial is even on the right disease — per-topic AUC for relevant-vs-rest is 0.533, i.e. chance. Fuse it with the retrieval score, never replace it: the pure-model ranking (w=1.0 below) is consistently worse than the first stage.

Benchmark

Fold-consistent 5-fold CV over 162 TREC CT 2021-23 topics, hybrid BM25 + embeddinggemma-300m-medical first stage, top-100 candidates, scored by p_eligible and fused convexly with the retrieval score. p from a paired Wilcoxon signed-rank test against the first stage (zero_method='zsplit', so topics left unchanged count as evidence of no effect).

configuration	graded NDCG@10	Δ vs first stage	p	judged@10
first stage only	0.5973	—	—	0.922
fusion w=0.3	0.6253	+0.0280	1.17e-05	0.917
fusion w=0.5	0.6340	+0.0367	0.0013	0.895
fusion w=0.7	0.6233	+0.0259	0.13	0.850
fusion w=1	0.5940	-0.0033	0.473	0.798

Graded gain is 2^label - 1 (0/1/3). Binary NDCG is not the metric to read here: excluded and eligible both count as relevant under it, so the ordering this model is trained to produce is invisible to it by construction.

Usage

pip install "crossgat @ git+https://github.com/JDev2001/msc_v2"

from crossgat_hf import load, score, rerank_score
pipe = load("2001jdev/crossgat-trial-eligibility-3class-ce", fold=0, metric="ndcg")
out  = score(pipe, patient_graph, inc_graph, exc_graph,
             patient_text="65yo male, type 2 diabetes, HbA1c 8.1%",
             trial_text="Metformin in adults with type 2 diabetes ...")
rerank_score(out, "p_eligible")

patient_text and trial_text are required — this checkpoint fuses whole-document embeddings with the graph features.

metric="ndcg" selects on val_ndcg10 and is what every reported number uses; metric="f1" selects on val_macro_f1, classifies better, and has not been benchmarked for reranking.

Checkpoint provenance

checkpoint_manifest.json maps each released file to its original training filename with a sha256 prefix. Checkpoints were resolved by modification time, not by the epoch in the filename — several training runs can share a fold directory.

crossgat-trial-eligibility-cv — 3-class LambdaRank + hard negatives, the best reranker of this family
crossgat-trial-eligibility-full — post-CV refit for deployment

Downloads last month: -; Downloads are not tracked for this model. How to track