Rainbow-Pony-100M-Flutter-steps

A 100M-parameter transformer, trained fully from scratch, fine-tuned to build up a Flutter/Dart file iteratively: given a goal, the current code, and a history of prior actions, it emits one small search/replace diff at a time, repeating until it signals <DONE>. Pretrained on 2B tokens (70% Flutter/Dart code, 30% English text), then fine-tuned on 50M tokens of step-sequence examples (bbidpa/flutter-diff-steps-v1).

This is one half of a paired comparison: this model decomposes each task into a sequence of small, verifiable edits, while its companion, Rainbow-Pony-100M-Flutter-direct, is trained to emit the complete file in a single pass on the same underlying task distribution. Both were fine-tuned from the same pretrained checkpoint, Rainbow-Pony-100M-Flutter-base. The research question: for a small model, does decomposing generation into short, self-correcting steps beat generating the whole thing at once?

Load it

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

REPO = "bbidpa/Rainbow-Pony-100m-Flutter-steps"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(REPO)
tokenizer.bos_id = tokenizer.bos_token_id
tokenizer.eos_id = tokenizer.eos_token_id
tokenizer.pad_id = tokenizer.pad_token_id
tokenizer.unk_id = tokenizer.unk_token_id
tokenizer.sep_id = tokenizer.convert_tokens_to_ids("<sep>")

model = AutoModelForCausalLM.from_pretrained(REPO, trust_remote_code=True).to(DEVICE).eval()

Prompt format

Same tag vocabulary as the -direct model, but here <HISTORY> is the whole point — it accumulates prior <ACTION> entries as the loop progresses, and <CODE> reflects the file's state after those prior steps:

<GOAL>
Add TextEditingControllers for the email and password fields, connect them to their respective TextFields, implement a _submit method that displays the entered email and password length in a SnackBar, and add spacing above the login button.
</GOAL>

<CODE>
import 'package:flutter/material.dart';

void main() => runApp(MaterialApp(home: Scaffold(body: Center(child: EmailInputForm()))));
</CODE>

<HISTORY>
<ACTION><TYPE>add_widget</TYPE><DESC>Add a SizedBox widget for vertical spacing.</DESC></ACTION>
</HISTORY>

<OUTPUT>

A single step's raw output looks like:

<ACTION><TYPE>...</TYPE><DESC>...</DESC></ACTION>
<CHANGES>
<HUNK>
<SEARCH>
...
</SEARCH>
<REPLACE>
...
</REPLACE>
</HUNK>
</CHANGES>

with a trailing <DONE></DONE> on the final step.

Generate — single step

def render_step_prompt_from_data(
    goal: str,
    code: str = "",
    history: list[dict] | None = None,
    action_type: str = "",
    action_desc: str = "",
    changes: list[dict] | None = None,
    is_last_step: bool = False,
    include_output: bool = False,
) -> str:
    history = history or []
    changes = changes or []
    history_text = "\n".join(
        f"<ACTION><TYPE>{h['type']}</TYPE><DESC>{h['desc']}</DESC></ACTION>"
        for h in history
    )
    text = f"""<GOAL>
{goal}
</GOAL>

<CODE>
{code}
</CODE>

<HISTORY>
{history_text}
</HISTORY>

<OUTPUT>
"""
    if include_output:
        hunks = "\n".join(
            f"<HUNK>\n<SEARCH>\n{h['search']}\n</SEARCH>\n<REPLACE>\n{h['replace']}\n</REPLACE>\n</HUNK>"
            for h in changes
        )
        output = f"<ACTION><TYPE>{action_type}</TYPE><DESC>{action_desc}</DESC></ACTION>\n<CHANGES>\n{hunks}\n</CHANGES>"
        if is_last_step:
            output += "\n<DONE></DONE>"
        text += output + "\n</OUTPUT>"
    return text


def generate_output(model, tokenizer, device, prompt, max_new_tokens=300, **generate_kwargs):
    prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
    idx = torch.tensor([[tokenizer.bos_id] + prompt_ids], dtype=torch.long).to(device)
    generated = model.generate(idx, max_new_tokens=max_new_tokens, eos_id=tokenizer.eos_id, **generate_kwargs)
    text = tokenizer.decode(generated[0].tolist(), skip_special_tokens=False)
    return text.split("<OUTPUT>")[-1].split("</OUTPUT>")[0].strip()

Generate — the full loop

Since this model's whole point is iterating until <DONE>, the meaningful way to run it is the multi-step loop (parses each step's <ACTION>/<CHANGES>, applies the diff, appends to history, and continues):

result = run_step_sequence(
    model, tokenizer, DEVICE,
    goal="Write a widget that displays a select button with months in it",
    code="",
    history=[],
    max_steps=20,
    max_new_tokens=300,
)
print(result["final_code"])
print(result["stop_reason"])

run_step_sequence depends on a few more helpers (generate_output_data, parse_generated_output, apply_edit/apply_edit_with_fallback) that parse the tag-structured output and apply each hunk — see [the training/eval code repo] for the full implementation.

Example

Input: goal + starting code + one prior history step (shown in "Prompt format" above)

Output trace:

[step 1] add_import: Add the import statement for flutter/material.dart (1 hunk)
[step 1] code:
import 'package:flutter/material.dart';

-------------------------------------------

[step 2] add_class: Add the DatePicker class declaration with its createState method. (1 hunk)
[step 2] code:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

-------------------------------------------

[step 3] add_class: Add the _DatePickerState class with an empty body. (1 hunk)
[step 3] code:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

class _DatePickerState extends State<DatePicker> {
}

-------------------------------------------

[step 4] add_method: Add the build method with a Column widget returning empty children list. (1 hunk)
[step 4] code:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

class _DatePickerState extends State<DatePicker> {
  @override
  Widget build(BuildContext context) {
    return Column(
      children: [],
    );
  }
}

-------------------------------------------

[step 5] add_widget: Add a Row widget with mainAxisAlignment and empty children list inside the Column's children list. (1 hunk)
[step 5] code:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

class _DatePickerState extends State<DatePicker> {
  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        Row(
          mainAxisAlignment: MainAxisAlignment.spaceBetween,
          children: [
          ],
        ),
      ],
    );
  }
}

-------------------------------------------

[step 6] hunk 1/1 SEARCH matched 2 times -- using last occurrence
FALLBACK
[step 6] add_widget: Add the ElevatedButton widget with onPressed and child Text('Select Date') inside the Row's children list. (1 hunk)
[step 6] code:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

class _DatePickerState extends State<DatePicker> {
  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        Row(
          mainAxisAlignment: MainAxisAlignment.spaceBetween,
          children: [
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
          ],
        ),
      ],
    );
  }
}

-------------------------------------------

[step 7] hunk 1/1 SEARCH matched 4 times -- using last occurrence
FALLBACK
[step 7] add_widget: Add the last ElevatedButton widget with onPressed and child Text('Select Date') to the Row's children list. (1 hunk)
[step 7] code:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

class _DatePickerState extends State<DatePicker> {
  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        Row(
          mainAxisAlignment: MainAxisAlignment.spaceBetween,
          children: [
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
          ],
        ),
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
      ],
    );
  }
}

-------------------------------------------

[step 8] hunk 1/1 SEARCH matched 2 times -- using last occurrence
FALLBACK
[step 8] add_widget: Add the SizedBox(height: 10) widget between the ElevatedButton and the Text widgets in the Row. (1 hunk)
[step 8] code:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

class _DatePickerState extends State<DatePicker> {
  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        Row(
          mainAxisAlignment: MainAxisAlignment.spaceBetween,
          children: [
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
          ],
        ),
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
            SizedBox(height: 10),
      ],
    );
  }
}

-------------------------------------------

[step 9] add_method: Add the _selectDate method with date picker logic. (1 hunk)
[step 9] code:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

class _DatePickerState extends State<DatePicker> {
  void _selectDate(BuildContext context) async {
    final DateTime? picked = await showDatePicker(
      context: context,
      initialDate: DateTime.now(),
      firstDate: DateTime(2000),
      lastDate: DateTime(2100),
    );
    if (picked != null) {
      setState(() {
        // _selectedDate = picked;
      });
    }
  }
  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        Row(
          mainAxisAlignment: MainAxisAlignment.spaceBetween,
          children: [
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
          ],
        ),
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
            SizedBox(height: 10),
      ],
    );
  }
}

-------------------------------------------

[step 10] add_field: Add the field '_selectedDate' to the state class. (1 hunk)
[step 10] code:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

class _DatePickerState extends State<DatePicker> {
  DateTime? _selectedDate;
  void _selectDate(BuildContext context) async {
    final DateTime? picked = await showDatePicker(
      context: context,
      initialDate: DateTime.now(),
      firstDate: DateTime(2000),
      lastDate: DateTime(2100),
    );
    if (picked != null) {
      setState(() {
        // _selectedDate = picked;
      });
    }
  }
  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        Row(
          mainAxisAlignment: MainAxisAlignment.spaceBetween,
          children: [
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
          ],
        ),
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
            SizedBox(height: 10),
      ],
    );
  }
}

-------------------------------------------

FALLBACK
[step 11] add_field: Add the field '_monthController' declaration line.
[step 11] APPLY FAILED on hunk 3/4 -- Search must match exactly once, got 0: '// _selectedDate = picked;'
[step 11] all generated hunks this step:
  hunk 1
    SEARCH:
class _DatePickerState extends State<DatePicker> {
    REPLACE:
class _DatePickerState extends State<DatePicker> {
  final _monthController = TextEditingController();
  hunk 2
    SEARCH:
// _selectedDate = picked;
    REPLACE:
_selectedDate = picked;
  hunk 3 <-- FAILED
    SEARCH:
// _selectedDate = picked;
    REPLACE:
_selectedDate = picked;
  hunk 4
    SEARCH:
onPressed: () => _selectDate(context),
              child: Text('Select Date'),
    REPLACE:
onPressed: () => _selectDate(context),
              child: Text(_monthController.text),
[step 11] code the failed SEARCH was matched against:
import 'package:flutter/material.dart';

class DatePicker extends StatefulWidget {
  @override
  _DatePickerState createState() => _DatePickerState();
}

class _DatePickerState extends State<DatePicker> {
  final _monthController = TextEditingController();
  DateTime? _selectedDate;
  void _selectDate(BuildContext context) async {
    final DateTime? picked = await showDatePicker(
      context: context,
      initialDate: DateTime.now(),
      firstDate: DateTime(2000),
      lastDate: DateTime(2100),
    );
    if (picked != null) {
      setState(() {
        _selectedDate = picked;
      });
    }
  }
  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        Row(
          mainAxisAlignment: MainAxisAlignment.spaceBetween,
          children: [
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
          ],
        ),
            ElevatedButton(
              onPressed: () => _selectDate(context),
              child: Text('Select Date'),
            ),
            SizedBox(height: 10),
      ],
    );
  }
}

Training details

Architecture	100M-parameter decoder-only transformer, trained from scratch
Pretraining	2B tokens (70% Flutter/Dart code, 30% English text)
Fine-tuning	50M tokens, multi-step goal → history → diff sequences
Tokenizer	Custom 16k-vocab BPE + structural special tokens

Related

• Base pretrained checkpoint: bbidpa/Rainbow-Pony-100m-Flutter-base
• Companion model (single-pass, whole-file training): bbidpa/Rainbow-Pony-100m-Flutter-direct
• Training dataset: bbidpa/flutter-diff-steps-v1
• Training/eval code: [Upcoming]