Zetik-Dev/magrec-0.6B-en-zh

MAGRec-0.6B (en-zh)

MAGRec is a small news click-through-rate (CTR) ranking model. Given a user (a short profile plus a list of recently browsed history items) and a set of candidate news items, it scores every candidate for how likely the user is to click it, in a single forward pass.

• Item tower — each item is represented by the embeddings of its title and abstract, produced offline by microsoft/harrier-oss-v1-270m (a multilingual sentence-transformer, 640-dim). A built-in ItemEncoder pools the two views into one item vector.
• User side — the profile is encoded by the same text encoder into one vector and injected as a "memory" token at the front of the sequence.
• Backbone — a Qwen3-style decoder stack attends over [user | history | candidate]; a linear head emits one logit per position. sigmoid(logit) at each candidate position is its predicted click probability.

Backbone	Qwen3-0.6B (hidden 1024, 28 layers, 16 heads)
Item text encoder	microsoft/harrier-oss-v1-270m (640-dim)
item_feature_dim / user_embedding_dim	640 / 640
Item encoder	attn
Languages	English + Chinese

Requirements

pip install "transformers>=5.0" torch sentence-transformers

Usage

The text encoder must be the same one used in training — harrier-oss-v1-270m, encoded without any instruction/prompt and L2-normalized (the sentence-transformers default). The embedding dim must equal config.item_feature_dim (640).

Sequence layout is [user | history | candidate]. The model overwrites position 0 with user_proj(user_embedding), so position 0 is a placeholder (zeros) and candidate_item_mask marks it 0; history is 0, candidates are 1.

import torch
from sentence_transformers import SentenceTransformer
from transformers import AutoModel

device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    "Zetik-Dev/magrec-0.6B-en-zh", trust_remote_code=True, dtype=torch.bfloat16
).to(device).eval()
encoder = SentenceTransformer("microsoft/harrier-oss-v1-270m", model_kwargs={"dtype": "auto"})
H = model.config.item_feature_dim

# user profile (bullet-list style) + recent history + candidate news (title, abstract)
profile = (
    "- Age: 29\n"
    "- Gender: Male\n"
    "- Occupation: Software engineer\n"
    "- Location: Kansas City, US\n"
    "- Interests: NFL football (die-hard Chiefs fan), fantasy football, sports betting\n"
    "- Reading style: Checks scores and NFL headlines every day"
)

history = [
    ("Patrick Mahomes returns from knee injury to lead Chiefs past Vikings",
     "Mahomes threw for three touchdowns in his first game back from a dislocated kneecap."),
    ("Fantasy football Week 11: must-start running backs and sleepers",
     "Our analysts break down the best waiver-wire pickups and start-sit calls."),
    ("Chiefs' defense steps up in gritty road win over the Chargers",
     "Kansas City forced two turnovers to hold on in a low-scoring divisional game."),
]

candidates = [
    ("NFL power rankings: where the Chiefs land coming out of their bye week",
     "Kansas City climbs after a statement win, while the Patriots slip a spot."),
    ("Fantasy football waiver wire: top Week 12 pickups at every position",
     "Streaming defenses and breakout running backs to grab before the deadline."),
    ("Lamar Jackson's MVP case gains steam after Ravens rout the Rams",
     "Jackson accounted for five touchdowns as Baltimore rolled on Monday night."),
    ("Ravens sign veteran safety as playoff push heats up in the AFC",
     "The move bolsters a secondary that has struggled with injuries down the stretch."),
    ("Impeachment hearing: key takeaways from the day's public testimony",
     "Diplomats testified before the House committee in a closely watched session."),
    ("Meghan and Harry announce they will step back from royal duties",
     "The couple said they plan to split their time between the UK and North America."),
    ("'The Mandalorian' premiere breaks streaming records on Disney+",
     "The Star Wars series drove a wave of sign-ups in Disney+'s opening week."),
    ("10 cozy soup recipes to get you through the winter",
     "Comfort-food ideas from readers, including a viral three-ingredient stew."),
]

# encode with harrier: no prompt, L2-normalized
enc = lambda texts: encoder.encode(
    texts, normalize_embeddings=True, convert_to_tensor=True).to(device, torch.bfloat16)

hist_title = enc([t for t, _ in history])
hist_abstract = enc([a for _, a in history])
cand_title = enc([t for t, _ in candidates])
cand_abstract = enc([a for _, a in candidates])
profile_vec = enc(profile)

# build [user | history | candidate]; pos0 is the user slot (placeholder zeros)
z = torch.zeros(1, H, dtype=torch.bfloat16, device=device)
title = torch.cat([z, hist_title, cand_title])
abstract = torch.cat([z, hist_abstract, cand_abstract])
cand_mask = torch.zeros(title.shape[0], dtype=torch.long, device=device)  # user=0, history=0, candidate=1
cand_mask[1 + len(history):] = 1

with torch.inference_mode():
    logits = model(
        title_embedding=title[None], abstract_embedding=abstract[None],
        user_embedding=profile_vec[None], candidate_item_mask=cand_mask[None],
    )["logits"][0].float()

probs = torch.sigmoid(logits[cand_mask == 1])
for rank, j in enumerate(torch.argsort(probs, descending=True).tolist(), 1):
    print(f"#{rank}  prob={probs[j]:.4f}  {candidates[j][0]}")

Example output

For the NFL-fan profile above, the four football candidates rank cleanly above the unrelated ones:

#1  prob=0.1836  NFL power rankings: where the Chiefs land coming out of their bye week
#2  prob=0.1097  Fantasy football waiver wire: top Week 12 pickups at every position
#3  prob=0.0737  Ravens sign veteran safety as playoff push heats up in the AFC
#4  prob=0.0656  Lamar Jackson's MVP case gains steam after Ravens rout the Rams
#5  prob=0.0454  'The Mandalorian' premiere breaks streaming records on Disney+
#6  prob=0.0307  Meghan and Harry announce they will step back from royal duties
#7  prob=0.0298  10 cozy soup recipes to get you through the winter
#8  prob=0.0164  Impeachment hearing: key takeaways from the day's public testimony

Probabilities are absolute CTR estimates (typically small); use them for ranking candidates for a given user rather than as literal click rates.

Inputs

forward(...) (see modeling_magrec.py) takes, per batch:

arg	shape	note
title_embedding	[B, SL, 640]	harrier embedding of each item's title
abstract_embedding	[B, SL, 640]	harrier embedding of each item's abstract
user_embedding	[B, 640]	profile embedding, injected at position 0
candidate_item_mask	[B, SL]	0 = user/history, 1 = candidate, -1 = padding

Returns {"logits": [B, SL]}; read the candidate positions (candidate_item_mask == 1). For an offline-precomputed item tower you can instead pass item_embedding ([B, SL, 640], the ItemEncoder output) and skip title/abstract.

Notes

• Encoder must match training. Embeddings from a different model (or the same model with a query instruction, or a different truncation dim) share the 640-dim shape but live in a different space and will produce meaningless scores.
• Keep candidate topics in-distribution. The model was trained on a fixed news corpus; candidates far outside that distribution (e.g. topics/entities absent from training) are scored unreliably. The example uses 2019-era English news to stay in distribution.