ClosureRAG cross-encoders

This repository holds the trained checkpoints of ClosureRAG, a retriever for questions decided by a set of interdependent norms. A retrieval-augmented system answers such a question correctly only if it retrieves the whole set, the evidence closure.

All three models are cross-encoders fine-tuned from Qwen3-Reranker-0.6B. Each scores a (case, passage) pair by the yes/no logit difference at the last position.

Folder Role in ClosureRAG Test (1,160 questions): FullCov@5/10/20 / R@5/10/20
l1_coverage_ranker/ Level 1: group-wise coverage ranker. Scores every candidate of the 400-passage pool on its own; its scores are fused with the first-stage rank (μ = 1.0). ClosureRAG-L1: 20.8 / 38.3 / 75.1 / 60.4 / 76.2 / 91.1
l2_context_rescorer/ Level 2: context-conditioned rescorer. Rescores the fused top-60 given the other top-ranked passages (m = 8) and is mixed with the fused score (λ = 0.5). ClosureRAG: 27.6 / 52.2 / 81.3 / 69.1 / 82.0 / 93.5
l1_pointwise_ablation/ Ablation: the level-1 ranker trained with a pointwise loss instead of the group-wise loss (fused with μ = 1.5). 15.7 / 31.4 / 66.2 / 61.3 / 72.2 / 87.6

Each folder holds a complete transformers checkpoint (model.safetensors, tokenizer, configs) and train_log.json with the training hyper-parameters.

Usage

The models expect the input format of the ClosureRAG code: the case text, a candidate passage with its document card and provenance, and, for level 2, the context cards and graph relations. Use them through the GitHub repository:

git clone https://github.com/vivilinx66-maker/ClosureRAG && cd ClosureRAG
pip install -r requirements.txt
hf download FDU-INS/ClosureRAG --local-dir models                      # the three folders, about 3.6 GB
bash scripts/run_inference.sh test models/l1_coverage_ranker models/l2_context_rescorer

run_inference.sh runs the whole online pipeline on the released candidate pools:

  1. Build the cross-encoder requests (make_requests.py).
  2. Level-1 scoring (score_l1.py).
  3. Rank fusion with the first stage (fuse.py).
  4. Level-2 rescoring (score_l2.py).
  5. Final ranking (final_rank.py) and evaluation.

It takes about one hour for the 1,160 test questions on an A100-40GB. Download a single model with --include "l1_coverage_ranker/*".

Scoring individual candidates in Python (run from the repository root, so that closurerag and data/ are found):

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from closurerag.fmt import Fmt, score

path = "models/l1_coverage_ranker"
tok = AutoTokenizer.from_pretrained(path, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16).cuda().eval()
F = Fmt(tok, 768)                                   # reads the passages and document cards from data/

case = F.case(question_text)                        # the question text of a benchmark item
seqs = [F.encode(case, F.cand(cid, why)) for cid, why in candidates]   # candidate passage ids + provenance text
with torch.no_grad():
    s = score(model, tok, F, seqs)                  # higher = more needed to decide the case

Level 2 uses the same Fmt with closurerag.fmt.encode_set and a context of passage ids. See scripts/score_l2.py.

Training

Training data, lists and scripts are in the GitHub repository (scripts/run_training.sh). All models were trained on one A100-40GB in bf16 with AdamW (learning rate 1e-5) and 8 lists per update.

  • Level 1: 2,058 lists, 2 epochs, 516 updates; group-wise loss with τ = 1, up to two members per gold group, 24 hard and 8 random negatives, pointwise term of weight 0.2.
  • Level 2: 6,174 lists (three contexts per question), 1 epoch, 772 updates; 12 hard and 6 random negatives, pointwise term of weight 0.05.

Licence

Apache-2.0, as the base model Qwen3-Reranker-0.6B.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for FDU-INS/ClosureRAG

Finetuned
(38)
this model