VertexAGI/codeinluau-1-small

CodeInLuau 1 Small

A Luau code-completion specialist, distilled from real Roblox source into Qwen3-8B

Part of the CodeIn family: where Aquamarine is a generalist across languages, CodeIn models go the other direction -- each one distilled for exactly one programming language, and nothing else. CodeInLuau is the first: Luau, Roblox's dialect of Lua.

Overview

CodeInLuau 1 Small is fine-tuned via LoRA on Qwen3-8B for raw Luau code completion -- given a prefix (a partial script, a function signature, a comment), it continues the code, not a chat reply. It's trained on real, human-written Luau scraped from live Roblox experiences, not synthetic/generated code.

Training

• Base model: mlx-community/Qwen3-8B-4bit
• Dataset: Roblox/luau_corpus -- real, human-written Luau, no synthetic generation. 19,608 train / 4,352 validation examples, each a genuine prefix/continuation pair from real Roblox source, trained as plain text completion (no chat template -- this is a raw code-completion task, not instruction-following).
• Method: LoRA fine-tuning (rank 8, scale 20, 16 layers), 3,000 iterations, batch size 1, sequence length 2048.
• Val loss: 1.342 at the end of training, down from 1.466 at initialization.

CodeInLuau's originally-spec'd base was Qwen3.5-9B-4bit, but that architecture's hybrid linear-attention layers hit a confirmed, unresolved upstream bug in mlx-lm during LoRA's backward pass (Metal Insufficient Memory, reproduced independently across chip generations and model sizes -- see ml-explore/mlx-lm#1206). Switched to the previous-generation, pure-transformer Qwen3-8B-4bit, which trains cleanly.

Evaluation

A held-out base-vs-tuned comparison used 100 prompts sampled from luau_corpus's official test split (never touched during training), feeding each model only the prompt half and comparing the generated continuation.

An earlier 10-prompt hand-authored eval suggested the base model frequently broke out of code-completion mode into chat-style narration -- that result did not replicate at n=100 on real corpus data (0/100 for both models). What did show a large, reproducible difference was actual code correctness, checked with the real Luau parser (luau-analyze), separating genuine mid-body syntax errors from harmless truncation-at-the-token-limit artifacts:

	fully clean	truncation-only (forgivable)	real syntax error
Base Qwen3-8B	23	59	18
CodeInLuau 1 Small	76	21	2

Base's failures are qualitatively worse than the count alone suggests -- on several prompts it abandons Luau entirely (hallucinating C-style // comments, or generating SQL instead of Luau), while CodeInLuau's rare failures are small, plausible mistakes (a : where . belongs, a ... used outside a vararg-scoped closure) -- the kind of thing an actual Luau developer might slip on, not a model losing the plot.

Usage

from mlx_lm import load, generate

model, tokenizer = load("VertexAGI/codeinluau-1-small")

prompt = '''local Players = game:GetService("Players")

Players.PlayerAdded:Connect(function(player)
\tlocal leaderstats = Instance.new("Folder")
'''
response = generate(model, tokenizer, prompt=prompt, max_tokens=150)
print(prompt + response)

This is a completion model, not a chat model -- feed it a real code prefix, not a question.

Formats available

MLX only for this release (4-bit, via mlx-lm, for Apple Silicon). No GGUF yet -- the conversion path (dequantize to fp16, then quantize) needs disk headroom this machine didn't have available at release time; may follow later.

Limitations

An 8B-parameter model fine-tuned via LoRA on ~20K examples -- capable at common Roblox scripting patterns (services, events, leaderstats, tweens, debounce, module returns) but not infallible, and it has no awareness of a specific game's actual object hierarchy or custom APIs. Treat completions as a strong starting point to review, not code to ship unread.

License

Apache 2.0, inherited from the Qwen3 base model.