FDS-Iterations/third-pass-feed-ranker-v10

Third-Pass Feed Ranker — v10 (preview)

A lightweight cross-encoder that scores how relevant an enterprise social-feed post is to a viewer, given only the viewer's job title and the post text. It is a third-pass reranker: it re-scores a small candidate slate (~20 items) from earlier passes to surface a post that is genuinely relevant to the viewer's profession but was buried below the top slot.

This is a separate v10 preview version. It is published alongside — not in place of — the current stable release, so you can evaluate the new behavior without changing your live model. v10 adds two capabilities on top of the stable release: bad-incumbent replacement and an expanded learning/discourse relevance surface (see below).

• Input: job_title (query) + post_text (passage) → single relevance score (higher = more relevant)
• Base: microsoft/MiniLM-L12-H384-uncased (~33M params). English, uncased — robust to inconsistent real-world capitalization.
• Runtime: server-class CPU at scale — very cheap at ~33M params.
• Trained with a listwise ranking objective; pointwise inference unchanged.

What "relevant" means here

Relevance = professionally interesting or useful to someone with this job title — not merely operational work they personally own. The model surfaces, for the viewer's field, research findings, methods/primers, lessons, thoughtful questions, industry developments, resource shares, recorded talks, papers, playbooks, launch decisions, incident post-mortems, and strategic tradeoffs; and it down-ranks off-role operational incidents (a bug fix another function owns), off-field interest content (research for a different profession), keyword-bait, and dense technical jargon that doesn't fit the role.

New in v10

• Bad-incumbent replacement. The model now distinguishes three situations that the stable release treated almost identically: (a) a low-value post sits in the top slot and a genuinely relevant — if only adjacent — post is buried → replace; (b) nothing in the slate is worth promoting → abstain; (c) the top slot is already a strong role-relevant post → protect it. On held-out roles it fires to replace a weak incumbent far more often than it fires on abstain/protect slates (see table), rather than firing at a flat rate regardless.
• Expanded learning & higher-order discourse. More coverage of learning-value content (recorded talks, papers/abstracts, playbooks) and judgment content (launch decisions, incident analysis/ post-mortems, strategic tradeoffs), balanced so relevance tracks substance-for-the-role rather than the discourse form itself.

Role coverage

Trained over a broad occupation taxonomy spanning all major occupational groups, extended with high-headcount enterprise and modern knowledge-worker / content-creator titles (e.g. communications, community, people/HR, leadership, customer success, project management, and engineering specialties), each with seniority and alias title variants (e.g. "Staff", "Lead", "Principal", common synonyms) so the model maps title variants of the same role to the same relevance.

Usage — scoring + the gate

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
name = "FDS-Iterations/third-pass-feed-ranker-v10"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()

def scores(title, posts):
    enc = tok([title]*len(posts), posts, truncation=True, max_length=160, padding=True, return_tensors="pt")
    with torch.no_grad():
        return model(**enc).logits.squeeze(-1).tolist()

TAU = 0.4    # promotion margin; calibrate per deployment (see below)
def third_pass(title, slate):
    s = scores(title, slate)
    challenger = max(range(1, len(slate)), key=lambda i: s[i])
    return ("promote", challenger) if s[challenger] - s[0] > TAU else ("no_change", None)

The model only scores. The gate (which item to promote, and τ) is your logic. τ is distribution-sensitive — on realistically-worded feeds ~0.3–0.5 (plainly-worded relevant posts clear the incumbent by smaller margins than dense/formal ones); recalibrate on your own no-relevance feeds.

Input filtering: very short posts are expected to be filtered upstream before scoring (this model was trained to match a deployment that drops posts under ~20 tokens / ~80 characters of body text).

Evaluation

Measured on external hold-out titles and on adversarial held-out-role slates:

metric	value
relevant post surfaced to #1 (external hold-out feeds)	~0.73
replace a low-value incumbent with an adjacent post (fire rate, should-replace slates)	~0.83
abstain / protect (fire rate on no-promotable & strong-incumbent slates — lower is better)	~0.23–0.47
on-role professional-interest beats off-role operational incident	~1.0
primary-role post selected among strong competitors (adversarial)	~0.57
realistic false-promotion on no-relevance feeds	~0.1%

Release-gated: this version is checked by a standing regression suite (cross-role counterfactuals, unseen-role generalization, clinical relevance, off-field/discourse-shortcut, keyword neutrality, and the new replace/protect guards) and every gate passes versus the current stable release.

Intended use & limitations

• Re-ranking short enterprise-feed candidate slates by job-title relevance; abstains on role-less titles.
• English only.
• Trained entirely on SYNTHETIC data (LLM-generated). Validate on your own data before production.
• Fine role-discrimination (a role's exact post vs. a closely adjacent role's) is near the capacity ceiling of a small model, though improved by explicit adjacent-role negatives.
• Research-vs-incident framing for a few technical roles is a known residual: promoting adjacent operational content slightly raised how "something-broke" posts rate relative to research/finding posts (research still wins in nearly all roles). On an independent hand-authored check this version scores marginally below the stable release on a small number of borderline cases.
• Out-of-distribution phrasing (terse fragments, unusual wording) scores more noisily; titles with heavy req/location codes dilute the signal (casing is handled).
• Relevance is title-driven; recency/importance beyond relevance must live in your decision logic.

Training & method

LLM-generated posts where relevance is by substance and the role is usually not named; adversarial 20-item slates with held-out roles and posts, in five slate shapes: promote a buried role-core post, replace a low-value incumbent with an adjacent post, abstain when nothing is promotable, and protect a strong incumbent or a protected announcement. Objective = pointwise relevance MSE plus a listwise ranking loss that pushes the correct promote target (or the strong incumbent) to the top. Curriculum measures against label shortcuts: the role keyword is rebalanced to be label-uncorrelated; dense-technical hard negatives prevent technical vocabulary being a global relevance signal; professional-interest and learning/discourse positives across many forms teach interest-relevance; and off-role operational, off-field interest/discourse, and adjacent-role negatives ensure neither discourse form, field-mismatch, nor near-role similarity is rewarded. The occupation frame is extended with high-headcount/creator roles and seniority/alias title variants. Training content spans a range of writing registers — formal write-ups to terse, plainly-worded status updates, questions, lessons, talks, and decisions — balanced across occupational cohorts so relevance is judged by substance rather than by how densely or technically a post is written.

License & attribution

Apache-2.0. Inherits from microsoft/MiniLM-L12-H384-uncased — verify its license carries through. Training posts were generated with a Qwen model; review the applicable terms.