orb-human-typeahead-1b-v2.2

A 1.6B typeahead model that predicts the human's next few words in a roleplay chat — inline "ghost text" for the person typing, not a reply generator for the character. Full fine-tune of ibm-granite/granite-4.0-1b-base.

Most small LMs asked to continue a user's half-typed roleplay message produce fluent but irrelevant text. This model is trained specifically on (conversation context + partial user message → the words the user actually typed next), so its suggestions stay on-scene and in-voice.

Variants

Path	Format	Use with
/	safetensors bf16, plain granite arch	transformers
GGUF/orb-human-typeahead-1b-v2.2-Q4_0.gguf	GGUF Q4_0 (default)	llama.cpp / llama-cpp-python
GGUF/orb-human-typeahead-1b-v2.2-Q8_0.gguf	GGUF Q8_0	llama.cpp / llama-cpp-python

The original fine-tune used the granitemoehybrid architecture; since this variant is attention-only and dense, the weights are republished here as the equivalent plain granite architecture (logit-identical, verified), which loads everywhere without extras.

Prompt format

Plain text, no chat template. Optional character summary, a marker line, name-prefixed turns, and finally the user's draft — the model continues the draft. Cut the suggestion at the first newline.

<character summary, optional>
***Roleplay chat below***
Sylvara: *She looks down from the watchtower and sees you.*
Traveler: *I approach the encampment.*
Sylvara: *She lowers her bow as you approach the gate.* "State your business, traveler."
Traveler: *I raise both hands slowly and

A completion looks like step into the torchlight, keeping my voice low.*

Notes:

• Trained to trigger at word boundaries only (draft ends on a whole word, or on a trailing space). Mid-word completion is out of scope.
• Trained context: up to ~4 recent turns, summaries ≤400 chars, turns ≤500 chars.

Serving recipe

Greedy, short budget, stop at newline — mirrors how it was trained and evaluated:

from llama_cpp import Llama

llm = Llama("GGUF/orb-human-typeahead-1b-v2.2-Q4_0.gguf", n_ctx=1024)
out = llm.create_completion(prompt, max_tokens=12, stop=["\n"], temperature=0.0)
print(out["choices"][0]["text"])

Or with transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("chartreuse-verte/orb-human-typeahead-1b-v2.2")
model = AutoModelForCausalLM.from_pretrained("chartreuse-verte/orb-human-typeahead-1b-v2.2")
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=12, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).split("\n")[0])

Evaluation

Scored on a held-out validation set of roleplay conversations (120 prompts, conversation-disjoint from training; greedy, 12-token budget, suggestions cut at newline). word-EM@k = fraction of prompts where the first k words of the suggestion exactly match what the user really typed next; prefix-chars = mean length of the exactly-matching leading characters.

Metric	orb-human-typeahead-1b-v2.1	this model
word-EM@1	0.483	0.500
word-EM@2	0.367	0.358
word-EM@3	0.275	0.267
accept-rate	0.267	0.133
prefix-chars	10.89	9.44
completion perplexity	3.56	3.56

Regression detected?? Not bug, is feature — That is the expected result of the change. Turns out autocompletion isn't immune to overused slop. v2.2 is a data-only refresh whose entire purpose was to flatten the completion distribution — to stop the model defaulting to a small set of high-frequency phrasings. The eval set itself is synthetic and has a diversity problem, it rewards both coherency AND overused slop (which are unfortunately akin in ML). A model that spreads its probability mass across more plausible continuations agrees with such targets less often, which is what the multi-word columns show: word-EM@1 and completion perplexity are flat, while longer exact-agreement metrics give back a few points.

Read the table as how much text is saved while typing, not as an absolute quality ranking. Exact agreement with one recorded continuation cannot score suggestion variety, and creative roleplay is high-entropy enough that many suggestions the metric counts as misses are perfectly good ones.

You will find v2.2 much more pleasant and much less janky despite the benchmark numbers.

Training

Two-stage full fine-tune of granite-4.0-1b-base: stage 1 on a large, mostly-synthetic roleplay corpus (self-chat generated to cover the on-scene, in-voice register real users type in), then stage 2 on a smaller private in-domain set in the serve-time prompt format (with stage-1 replay to limit forgetting). Over v2.1, stage 1 grew and its completion distribution was flattened further: a 3-gram repetition cap retires canned mid-phrase filler that the earlier first-word cap could not see, post-dialogue beats got real supervision instead of being capped into starvation, and generator punctuation tells (em-dashes, curly quotes) plus non-English leakage are normalized out rather than dropped. Examples are user turns split at word boundaries; loss is on the continuation. Suggestions are single-line by construction (completions end at newline).

Limitations

• English-centric, roleplay register (asterisk actions, quoted dialogue). Out of domain for assistant chat, code, or formal prose.
• Roleplay corpora include mature themes; suggestions can reflect that. Intended as a typing aid for consenting adult users of RP chat apps.
• Not an instruction follower — it only continues drafts in the format above.
• Suggests at word boundaries; won't complete a half-typed word.