Rainbow-Pony-100M-Flutter-direct

A 100M-parameter transformer, trained fully from scratch, fine-tuned to generate a complete Flutter/Dart file in one shot from a natural-language goal. Pretrained on 2B tokens (70% Flutter/Dart code, 30% English text), then fine-tuned on 5M tokens of flat goal → complete-file examples (bbidpa/flutter-full-examples-v1).

This is one half of a paired comparison: this model is trained to emit the whole file directly, while its companion, Rainbow-Pony-100M-Flutter-steps, is trained to build the same kind of code up iteratively via a sequence of small diffs. Both were fine-tuned from the same pretrained checkpoint, Rainbow-Pony-100M-Flutter-base. The research question behind the pair: for a small model, is it more sample-efficient to learn "whole file in one pass," or "one small verifiable edit at a time"?

Load it

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

REPO = "bbidpa/Rainbow-Pony-100m-Flutter-direct"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(REPO)
tokenizer.bos_id = tokenizer.bos_token_id
tokenizer.eos_id = tokenizer.eos_token_id
tokenizer.pad_id = tokenizer.pad_token_id
tokenizer.unk_id = tokenizer.unk_token_id
tokenizer.sep_id = tokenizer.convert_tokens_to_ids("<sep>")

model = AutoModelForCausalLM.from_pretrained(REPO, trust_remote_code=True).to(DEVICE).eval()

Prompt format

Prompts use a small XML-style tag vocabulary. <CODE> holds the current file contents (empty for a from-scratch generation), <HISTORY> holds prior steps (unused for this model's typical single-step usage, but part of the shared format), and the model completes everything after <OUTPUT>:

<GOAL>
Write a widget that displays a select button with months in it
</GOAL>

<CODE>

</CODE>

<HISTORY>

</HISTORY>

<OUTPUT>

Generate

def render_step_prompt_from_data(
    goal: str,
    code: str = "",
    history: list[dict] | None = None,
    action_type: str = "",
    action_desc: str = "",
    changes: list[dict] | None = None,
    is_last_step: bool = False,
    include_output: bool = False,
) -> str:
    history = history or []
    changes = changes or []
    history_text = "\n".join(
        f"<ACTION><TYPE>{h['type']}</TYPE><DESC>{h['desc']}</DESC></ACTION>"
        for h in history
    )
    text = f"""<GOAL>
{goal}
</GOAL>

<CODE>
{code}
</CODE>

<HISTORY>
{history_text}
</HISTORY>

<OUTPUT>
"""
    if include_output:
        hunks = "\n".join(
            f"<HUNK>\n<SEARCH>\n{h['search']}\n</SEARCH>\n<REPLACE>\n{h['replace']}\n</REPLACE>\n</HUNK>"
            for h in changes
        )
        output = f"<ACTION><TYPE>{action_type}</TYPE><DESC>{action_desc}</DESC></ACTION>\n<CHANGES>\n{hunks}\n</CHANGES>"
        if is_last_step:
            output += "\n<DONE></DONE>"
        text += output + "\n</OUTPUT>"
    return text


def generate_output(model, tokenizer, device, prompt, max_new_tokens=300, **generate_kwargs):
    prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
    idx = torch.tensor([[tokenizer.bos_id] + prompt_ids], dtype=torch.long).to(device)
    generated = model.generate(idx, max_new_tokens=max_new_tokens, eos_id=tokenizer.eos_id, **generate_kwargs)
    text = tokenizer.decode(generated[0].tolist(), skip_special_tokens=False)
    output = text.split("<OUTPUT>")[-1].split("</OUTPUT>")[0].strip()
    return output


prompt = render_step_prompt_from_data(
    goal="Write a widget that displays a select button with months in it",
)
text = generate_output(model, tokenizer, DEVICE, prompt, max_new_tokens=1024)
print(text)

Example

Input goal: Write a widget that displays a select button with months in it

Output:

[paste your actual generated <ACTION>/<CHANGES> output here]

Training details

Architecture	100M-parameter decoder-only transformer, trained from scratch
Pretraining	2B tokens (70% Flutter/Dart code, 30% English text)
Fine-tuning	5M tokens, flat goal → complete-file examples
Tokenizer	Custom 16k-vocab BPE + structural special tokens

Related

• Base pretrained checkpoint: bbidpa/Rainbow-Pony-100m-Flutter-base
• Companion model (iterative diff-based training): bbidpa/Rainbow-Pony-100m-Flutter-steps
• Training dataset: bbidpa/flutter-full-examples-v1
• Training/eval code: [Upcoming]