asharrm/juke-1.7b-dev

juke-1.7b: type anything, get a playlist

juke is a 1.7B language model that turns any prompt into a playlist of real songs, like "bossa nova brunch", "sad midwest emo for a long drive", "yttrium exoplanet" or "my cat's tax return". It can think out loud like a DJ first, then give a title, a one-line idea, and a numbered tracklist with a short reason for each song.

v1.1 (recommended): repeats inside a playlist cut from 5% to 1.8% by a weight edit, not retraining. Four "copy-suppression" attention heads were made 2× louder (about 0.03% of the weights). See What's new in v1.1.

It is a single model: there is no search index, database or extra code. The model has memorized a catalog of 229,042 songs, and each song has a short address it can write, for example <a_67><b_10><c_0> = Brian Eno – An Ending.

> yttrium exoplanet

<think>
I'm reading "yttrium exoplanet" as a cosmic object with a metallic, distant, slightly surreal feel... ambient,
post-rock, dream pop and spacious electronic pieces... opening with distance, moving through shimmer and gravity,
then landing in a hushed, suspended afterimage.
</think>

Yttrium Orbit
Logic: Spacious, reflective tracks that feel like orbiting a cold planet.
1. <a_67><b_10><c_0> Brian Eno – An Ending — Soft light fading into deep space
2. <a_12><b_1><c_0> Tangerine Dream – Phaedra — Classic cosmic drift and shimmer
3. <a_67><b_68><c_0> Vangelis – Alpha — Bright, floating, otherworldly calm
4. <a_9><b_27><c_0> Boards of Canada – Dayvan Cowboy — Warm distance with a metallic sheen
...

Quick start (🤗 transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import re

repo = "asharrm/juke-1.7b-dev"
tok = AutoTokenizer.from_pretrained(repo, subfolder="v1.1")
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="v1.1", torch_dtype="auto", device_map="auto")

prompt = "sad midwest emo for a long drive"
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,
                               add_generation_prompt=True, enable_thinking=True)   # False = no DJ monologue
ids = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=700, do_sample=False)
reply = tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=False).replace("<|im_end|>", "")

# hide the song addresses for display
print(re.sub(r"(<[abc]_\d+>){3}\s*", "", reply))

Thinking on or off: enable_thinking=True makes the model write a short plan (60–120 words) and a reason per track, which suits a "talking DJ". False gives just title, logic and tracks. In our tests, thinking on produced more varied playlists.

Removing repeats: about 2% of tracks (v1.1) are repeats of a song already in the same list (see Evaluation). Drop any line whose <a_><b_><c_> address already appeared, or ask for a few extra tracks and de-duplicate.

llama.cpp / GGUF

A ready Q8_0 file is in this repo: gguf/juke-1.7b-v1.1-q8_0.gguf (1.8 GB).

huggingface-cli download asharrm/juke-1.7b-dev gguf/juke-1.7b-v1.1-q8_0.gguf --local-dir .
llama-server -m gguf/juke-1.7b-v1.1-q8_0.gguf -ngl 99

Or convert yourself with llama.cpp's convert_hf_to_gguf.py (the 768 song-address tokens convert as normal added tokens):

python convert_hf_to_gguf.py final/ --outtype q8_0 --outfile juke-1.7b-q8_0.gguf
llama-server -m juke-1.7b-q8_0.gguf -ngl 99

Use raw completion with the Qwen3 chat format. For thinking off, add an empty think block: <|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n. A Q8_0 GGUF (1.8 GB) runs at ~146 tokens/s on a PS5 GPU (Vulkan), which is about 1.5 s per playlist.

Output format

<think>…plan…</think>            (thinking mode only)

<Playlist title>
Logic: <one sentence>
1. <a_x><b_y><c_z> Artist – Title — <reason>      (reasons in thinking mode only)
2. …

Other trained prompt styles:

• /tracks <playlist name> → just a tracklist, in the style of real user playlists
• /tag shoegaze → 10 songs carrying that genre/tag
• <song> <a_x><b_y><c_z> → Artist – Title (catalog lookup)
• <code> Artist – Title → <a_x><b_y><c_z> (reverse lookup)

How song addresses work

Every catalog song gets a 3-token address. Songs that sound alike share a prefix:

• a = one of 256 broad areas, e.g. shoegaze, bossa nova or modern country (~900 songs each)
• b = a small sub-style inside the area (~9 songs each)
• c = the song's position inside the sub-style (most popular first)

The addresses come from a balanced k-means tree over song vectors built from playlist co-occurrence (which songs people put together) plus genre/style tags and an artist signal. Writing the address first means the model "walks into the right neighbourhood" before naming the song, the same idea as Spotify's Text2Tracks and other semantic-ID generative retrieval work.

Training

• Base: Qwen3-1.7B, full fine-tune, 768 new address tokens plus <song> and <code>.

• Data: 579k examples, 39M tokens per pass, five tasks:

  Task	Examples	What it teaches
  Memorization	458k	address ↔ exact song name, every catalog song, both directions
  Teacher playlists	26.5k	prompt → title, logic, tracks; 10.9k with a thinking plan and per-track reasons
  Odd prompts	2k	absurd and abstract prompts with committed interpretations
  Real playlists	79k	real user playlist names → their tracks
  Tags	14.6k	genre/style tag → songs

  Teacher playlists, thinking traces and odd-prompt answers were written by GPT-5.4-mini. Every track was checked against the catalog, and any song not in it was dropped.

• Schedule: 8 rounds. Every round contains all memorization examples plus a sampled share of the other tasks. 2048-token packed sequences, AdamW, peak learning rate 1e-4 with cosine decay, bf16 autocast with fp32 master weights.

• Compute: about 4.5 h on one RTX PRO 6000 (96 GB). Total GPU cost for the project was about $10.

• Versions in this repo: v1.1/ (recommended: final/ plus the head edit below), final/ (end of round 8), and the pass7/ and pass6/ checkpoints.

What's new in v1.1: fixing repeats by editing the weights

v1.0 (final/) sometimes listed the same song twice in one playlist. About 5% of tracks were repeats, e.g. Leonard Cohen – Everybody Knows three times in "dark cabaret heartbreak". The usual fix is more fine-tuning. Instead we found the circuit responsible and edited it directly: no gradient steps, no new data, about 0.03% of the weights.

1. Collect the behaviour

Greedy-decode 400 playlists from training prompts that the evaluation does not use. Label every list item as repeat (its 3-token song address already appeared earlier in the list) or fresh: 239 repeats and 3,692 fresh.

2. Where is "about to repeat" visible? (correlational)

At the position that predicts each item's first address token, take the residual stream at every layer. The difference of class means gives a direction, scored on a held-out half by Cohen's d:

layers	1–9	10–15	16–19	20–21	22–28
Cohen's d	0.6–1.1	0.4–0.95	1.4–1.6	1.05–1.3	0.4–0.9

The decision is most legible in the middle-late layers. Correlation is not causation, so we went one step further.

3. Which units cause it? (attribution patching)

For each list item, target = log p(the item's 3 address tokens). For every MLP neuron (down_proj input) and every attention head (its slice of the o_proj input), attribution = activation × ∂target/∂activation, summed over the 3 predicting positions. Then:

repeat score = mean attribution on repeated items − mean attribution on fresh items

The largest scores were negative, meaning units that push repeats down. These are copy-suppression heads that look back at songs already listed and lower their probability:

head	repeat score
L22 h8	−0.068
L22 h7	−0.046
L26 h7	−0.042
L16 h14	−0.040
(most repeat-promoting: L15 h6)	+0.028

4. Test edits causally, against controls

Each candidate edit was applied to a fresh copy of the weights and scored by teacher forcing on held-out playlists (eval prompts). Δ is the change in mean log-probability of repeated vs. fresh items; we want Δrepeat ≪ Δfresh ≈ 0.

edit	Δ repeat	Δ fresh
8 / 32 / 128 random neurons → 0 (control)	0.00 / 0.00 / −0.03	0.00 / 0.00 / −0.01	control is flat ✅
top 8 / 32 / 128 repeat-promoting neurons → 0	−0.33 / −0.91 / −1.90	−0.27 / −0.65 / −1.49	hurts everything ❌
remove the probe direction (layers 15–19)	−0.41	−0.41	not selective ❌
boost top anti-repeat neurons ×1.5–2	−0.06 to −0.21	−0.03 to −0.11	too weak
top 2 / 4 repeat-promoting heads → 0	−0.49 / −0.82	−0.26 / −0.53	partly selective
4 copy-suppression heads ×1.5	−0.68	−0.25	selective ✅
4 copy-suppression heads ×2	−1.64	−0.60	chosen ✅
4 copy-suppression heads ×3	−3.81	−1.51	starts to damage

5. The edit

import torch
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("asharrm/juke-1.7b-dev", subfolder="final", torch_dtype=torch.bfloat16)
hd = m.config.head_dim                                   # 128
with torch.no_grad():
    for layer, head in [(22, 8), (22, 7), (26, 7), (16, 14)]:
        m.model.layers[layer].self_attn.o_proj.weight[:, head * hd:(head + 1) * hd] *= 2.0
m.save_pretrained("juke-1.7b-v1.1")

That is 4 × 128 columns of o_proj: 1.05M of 1.7B weights, about 0.03%.

6. Result (greedy decoding, 110 prompts, real generation)

metric	v1.0 final/	×1.5	v1.1 (×2)
repeated tracks inside a playlist	5.0%	2.9%	1.8%
song addresses that exist	99.9%	100%	99.9%
song name matches its address	96.6%	97.1%	98.0%
distinct songs across 40 unrelated prompts	90.9%	91.7%	93.0%
overlap with teacher playlists (trained prompts)	7.2%	5.9%	5.6%

Before and after:

prompt	v1.0	v1.1
dark cabaret heartbreak	Leonard Cohen – Everybody Knows ×3	The Dresden Dolls, Nellie McKay, Cold Cave, no repeats
wistful indie rock from the 2000s	The Postal Service – Sleeping In ×3	Jimmy Eat World, Bright Eyes, Death Cab for Cutie, Dashboard Confessional, The Shins…
weird punk songs	The Adverts ×2	Misfits, Bad Brains, Black Flag, Television, Dead Kennedys, The Undertones

Side effect: where the model knows few songs in a genre, it now drifts to a neighbouring style more often instead of repeating. For example, "banda" gave cumbia and mambo. The edit removes an obstacle; it cannot add songs the model never learned, which matches the weight-space-programming finding below.

Why this worked

The training data never contained repeats, so the model learned a copy-suppression mechanism. It was simply too quiet, and greedy decoding sometimes let a strongly-recalled song win twice. Turning up a mechanism that already exists is a targeted fix. Switching off neurons that merely correlate with repeats damaged everything.

Inspired by: Neural Networks Are Already Clean (Cammarata, Goodfire, 2026). The "Evil Qwen" single-neuron reproductions (HedwigAI, 2026), where the most active neurons were not the causal ones, so rank by causal attribution and check random controls. Copy-suppression heads catalogued in neuron-lens (stdlogits). Sparks of Weight-Space Programming (Takehi, Teoh, Zhang & Khattab, MIT CSAIL, 2026). Scripts: repeat_direction.py, repeat_attrib.py, repeat_edit_test.py, repeat_apply.py in the juke repo.

Evaluation (v1.0 final/, greedy, 110 prompts; v1.1 numbers above)

Metric	Result
Song addresses that exist in the catalog	99.9%
Song name matches its address	96.6%
Address → exact name (300 random catalog songs)	92.7%
Name → exact address	63.3%
Repeated tracks inside a playlist	5%
Songs shared across 40 unrelated prompts	91% distinct, none in 3+ playlists
Overlap with the teacher's playlist (trained / unseen prompts)	7% / 2.5%, so it is not copying

Limitations

• Catalog: 229k songs, mostly Western music popular in English-language playlist data (Spotify playlists from the mid-2010s and ListenBrainz playlists up to 2026). Newer and regional music is thin. The model can only name songs it memorized.
• Hallucinations: about 3% of track lines have a name that doesn't match its address, and these are sometimes half-remembered titles. Check addresses against your own catalog if exactness matters.
• Repeats: about 2% of tracks in v1.1 (5% in v1.0). De-duplicate (see above).
• Taste: the model inherits the biases of its playlist data and its teacher model, including over-represented artists and genres.
• Reasons are explanations, not ground truth. Per-track reasons can stretch to justify a pick.

Data sources and licence

The model weights are released under CC BY-SA 4.0: you may use, share and adapt them, including commercially, if you give credit and share adaptations under the same licence. The base model, Qwen3-1.7B, is Apache-2.0.

Data used to build the catalog, song addresses and training examples:

Source	Licence	Used for
Spotify playlist dataset (Pichl, Zangerle & Specht; Zenodo 2594557)	CC BY 4.0	real playlists, co-occurrence
ListenBrainz playlists, MusicBrainz	CC0	real playlists, catalog verification
Discogs data dumps, AcousticBrainz	CC0	genres and styles
Wikipedia / Wikidata	CC BY-SA 4.0 / CC0	genre vocabulary, artist genres
Last.fm dataset (Million Song Dataset)	research / non-commercial terms	song tags (genre tag task, address grouping)
Art of the Mix 2011	research	popularity counts
GPT-5.4-mini outputs	OpenAI terms	teacher playlists, thinking traces

⚠️ Caveat: Last.fm and Art of the Mix data come with research-use terms. They shaped the tag task and the song grouping, but none of their text is reproduced. If you need a strictly commercial-clean model, the planned fix is to rebuild tags from CC0/CC-BY sources only.

Citation

@misc{juke2026,
  title  = {juke: a standalone prompt-to-playlist model with semantic song addresses},
  author = {asharrm},
  year   = {2026},
  url    = {https://huggingface.co/asharrm/juke-1.7b-dev}
}

Related work: Text2Tracks (Palumbo et al., 2025), TIGER / semantic IDs for generative retrieval (Rajput et al., 2023).