Intelligent Candidate Ranker (LightGBM LambdaRank)
The ranking model from the Redrob Intelligent Candidate Discovery and Ranking Challenge submission.
Given a 22-feature vector describing a candidate's fit against a job description, this model outputs a relevance score. It is trained on labels generated by 2,500 pairwise judgments from a local LLM (Gemma3) rather than hand-coded heuristics, specifically to avoid label circularity.
Full pipeline (retrieval, feature engineering, consistency scoring, reasoning generation) lives in the GitHub repo: https://github.com/Pranjal1342/Intelligent-Candidate-Discovery-Ranking-System
This model's role in the pipeline
This model is one stage inside a larger offline candidate-ranking pipeline. It does not do retrieval, does not compute the input features itself, and does not produce the final rank on its own.
raw_score = this_model.predict(feature_vector)
final_score = raw_score * consistency_score # applied by the host pipeline, not this model
consistency_score is a separate, multiplicative honeypot/data-integrity check computed by the host application — it is not part of this model's output.
How to load
from huggingface_hub import hf_hub_download
import lightgbm as lgb
model_path = hf_hub_download(
repo_id="<your-username>/intelligent-candidate-ranker",
filename="lgbm_model.txt"
)
model = lgb.Booster(model_file=model_path)
raw_score = model.predict(feature_vector) # feature_vector: 22-dim float32
Input / Output
- Input: a 22-feature float32 vector per candidate (exact feature order below)
- Output: a single raw relevance score (higher = more relevant). Not yet penalized for data-integrity issues — combine with a
consistency_scoredownstream before final ranking.
Feature vector (in order)
| # | Feature | Formula / Source |
|---|---|---|
| 1 | bm25_score |
Stage 1 BM25 retrieval score (normalised) |
| 2 | yoe |
profile.years_of_experience |
| 3 | Param_A_Systems_Depth |
Fraction of career months in roles whose descriptions contain retrieval, search, or ranking keywords |
| 4 | Param_B_Availability |
(recruiter_response_rate + exp(-days_inactive / 90)) / 2 |
| 5 | Param_C_Tenure |
min(avg_tenure_months, 48) / 48, rewards 3+ year tenures |
| 6 | Param_D_Notice_Exp |
exp(-max(0, days-30) / 30): 30d → 1.0, 60d → 0.37, 90d → 0.14, 150d → 0.006 |
| 7 | Param_E_Credibility |
advanced_claimed_count / max(1, assessed_count), higher means less credible |
| 8 | Param_F_Consulting |
Fraction of career at IT-services consulting firms (industry == "IT Services" AND size == "10001+") |
| 9 | Param_G_Location |
Noida/Pune = 1.0, other India = 0.7, outside and willing to relocate = 0.3, outside and unwilling = 0.0 |
| 10 | Param_H_GitHub |
github_activity_score / 100; 0.3 imputed when the field equals -1 (absent) |
| 11 | title_ai_fraction |
Career-weighted fraction in AI, ML, or data roles via a static title taxonomy |
| 12 | prod_signal_log |
Log-compressed production keyword count, -1.0 if academic-only |
| 13 | consistency_score |
Multiplicative honeypot penalty, c1 × c2 × c3 × c4 × c5 (included as a training feature; also reapplied post-inference — see below) |
| 14 | hard_req_coverage |
Fraction of JD hard requirements satisfied by the candidate's skill list |
| 15 | flag_consulting_only |
consulting_fraction > 0.95 |
| 16 | flag_title_chaser |
avg_tenure < 18 months across 3+ jobs |
| 17 | flag_langchain_dabbler |
LLM-era months > 12 and pre-LLM months == 0 |
| 18 | flag_cv_specialist |
CV/speech months > 24 and NLP/IR months == 0 |
| 19 | flag_title_desc_mismatch |
Domain-category mismatch fraction across career history |
| 20 | flag_template_desc |
Max SequenceMatcher ratio against the template registry |
| 21 | interaction_req_x_consistency |
hard_req_coverage * consistency_score |
| 22 | interaction_yoe_x_prod |
yoe * prod_signal_log |
Training
Model configuration:
objective: lambdarankeval_at: [5, 10, 50], explicitly optimising Precision@5- Early stopping monitors NDCG@5, patience 30
- 200 boosting rounds
Training labels — Gemma3 pairwise annotation (the key differentiator):
Rather than a pure heuristic label, training labels are generated via 2,500 pairwise LLM comparisons using Gemma3:4b-it-q4_K_M running locally on Ollama, with zero external API calls and full reproducibility. A stratified sample of 500 candidates is drawn across three strata (top-100, boundary 101–300, and a broader pool with guaranteed low-consistency coverage), and each candidate receives roughly five matchups against random opponents.
For each pair, Gemma3 reads both candidates' full structured profiles alongside the JD requirements and disqualifiers, then produces a single verdict: CANDIDATE_A, CANDIDATE_B, or TIE. Win and loss tallies convert to Elo ratings via Laplace-smoothed win rates:
win_rate = (wins + 0.5) / (total + 1)
elo = 400 * log10(win_rate / (1 - win_rate)) + 1500
Elo ratings are thresholded to 0–3 relevance labels by quartile, producing a balanced training set with roughly 125 candidates per label.
Why this breaks circularity: Gemma had no knowledge of the 22 engineered features, the BM25 scores, or the penalty weights. It learned independently that IR-specific skills (FAISS, BM25, Qdrant, Sentence Transformers) outrank generic ML skills, and that production-company backgrounds outrank consulting-only careers. LightGBM then learns how the 22 features correlate with these independent judgments, surfacing interactions that were never explicitly encoded.
Model Comparison: Heuristic vs. Gemma-Trained
The competition provides no ground-truth relevance labels, so a standard NDCG@10 ablation against a labeled holdout set isn't possible to compute honestly. What is available, and what is reported here, is a direct head-to-head comparison between a LightGBM model trained on the original heuristic weak label and this model (trained on Gemma3 pairwise labels), run on the same candidate pool with the same feature vectors.
Method: both trained models score the full ~8,500-candidate retrieval pool. The same post-inference consistency multiplier is applied to both before ranking, so the comparison isolates the effect of the training label, not the honeypot suppression layer.
| Metric | Result |
|---|---|
| Top-10 overlap between the two models | 0 of 10 candidates in common |
| Spearman rank correlation (top-100) | 0.001 — statistically independent rankings |
| Honeypot leakage, heuristic-trained model | Required a hand-coded post-processing suppression list to keep keyword-stuffed non-technical profiles out of the top 100 |
| Honeypot leakage, Gemma-trained model (this model) | 0 of 100 candidates with consistency_score < 0.25, achieved with no post-processing suppression list |
Qualitative before/after: prior to the Gemma retrain, the heuristic-trained model's unsuppressed top-10 surfaced profiles such as Content Writer, Project Manager, and Sales Executive — each with AI-sounding skills listed but no underlying technical career history, because the heuristic label rewarded keyword coverage directly. After the Gemma retrain, the same pool's top-10 surfaced candidates with FAISS, BM25, Qdrant, Sentence Transformers, and Hugging Face Transformers in their skill history — sourced from a model that never saw bm25_score or hard_req_coverage during label generation and discovered the IR-relevance ordering independently from reading full candidate profiles.
The two models disagreeing almost completely (Spearman 0.001) is itself evidence of non-circularity: a model trained on labels derived from the same 22 features it predicts on would be expected to correlate strongly with a heuristic built from those same features, not diverge from it entirely.
Intended Use & Limitations
- Built for a hackathon submission (Redrob Intelligent Candidate Discovery and Ranking Challenge); not validated for production hiring decisions.
- Expects the exact 22-feature schema above, computed by the host pipeline's
src/features.py. Feeding hand-built or differently-ordered features will produce meaningless scores. - Raw model output is not the final ranking score — it must be multiplied by a separately computed
consistency_scorebefore use. - Trained on a synthetic/competition candidate dataset; label distribution and feature semantics may not generalize to other candidate pools without retraining.
AI Tool Disclosure
Gemma3:4b-it-q4_K_M (Google DeepMind, running locally via Ollama) was used offline to generate 2,500 pairwise relevance judgments on a stratified sample of 500 candidates. These judgments served as independent, non-circular training labels for this model. No candidate data was transmitted to any external service at any point.