ElyKok/hebrew-qa-reranker

Hebrew QA Reranker (fine-tuned bge-reranker-v2-m3)

A cross-encoder that scores how well a Hebrew paragraph answers a Hebrew question. Fine-tuned from BAAI/bge-reranker-v2-m3 on the answerable questions of HeQ, with 3 hard negatives per question taken from a hybrid BM25 + fine-tuned e5-large first stage, and BinaryCrossEntropyLoss (1 epoch, lr 1e-05, 2 x T4).

Use it as the second stage of a search: retrieve the top 20 paragraphs, then rerank them.

from sentence_transformers.cross_encoder import CrossEncoder
model = CrossEncoder("ElyKok/hebrew-qa-reranker")
scores = model.predict([("מי ניסח את מגילת העצמאות?", "paragraph 1"), ("מי ניסח את מגילת העצמאות?", "paragraph 2")])

Results (HeQ test set, 1,072 questions, reranking the top 20 of 4,930 paragraphs)

Model	R@1	R@5	R@10	MRR@10	NDCG@10
Hybrid (BM25 + e5-large, no training)	80.4	92.1	95.1	85.5	87.8
Hybrid (BM25 + e5-large, no training) + bge-reranker-v2-m3 (no training)	91.5	96.5	96.8	93.8	94.6
Hybrid (BM25 + e5-large, no training) + fine-tuned reranker (ours)	93.8	96.6	96.8	95	95.5
Hybrid (BM25 + fine-tuned e5-large)	83.3	93.4	95.5	87.6	89.5
Hybrid (BM25 + fine-tuned e5-large) + bge-reranker-v2-m3 (no training)	91.7	97.2	97.4	94.1	95
Hybrid (BM25 + fine-tuned e5-large) + fine-tuned reranker (ours)	93.9	97.2	97.4	95.4	95.9

Test articles are disjoint from training articles. HeQ is licensed CC BY 4.0 (Webiks / MAFAT).