Qwen2.5-Coder-0.5B-Flutter-steps

Qwen2.5-Coder-0.5B, fine-tuned to build up a Flutter/Dart file iteratively: given a goal, the current code, and a history of prior actions, it emits one small search/replace diff at a time, repeating until it signals <DONE>. Fine-tuned on 50M tokens of step-sequence examples (bbidpa/flutter-diff-steps-v1).

This model is part of a paired comparison studying whether small models benefit more from learning to build code up iteratively, or from learning to emit a whole file at once. Its companion on the same base model is Qwen2.5-Coder-0.5B-Flutter-direct. The same comparison is also run on a 100M-parameter model trained fully from scratch: Rainbow-Pony-100M-Flutter-steps / -direct.

Load it

Since Qwen2.5-Coder is a standard, already-registered transformers architecture, plain AutoModel loading works with no custom code:

from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "bbidpa/Qwen2.5-Coder-0.5B-Flutter-steps"

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO)

For the exact calling convention used in the examples below (and shared with the from-scratch TinyGPT models in this collection), wrap it with these two small adapters instead:

import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"


class QwenTokenizerAdapter:
    def __init__(self, model_id):
        self.hf = AutoTokenizer.from_pretrained(model_id)
        self.eos_id = self.hf.eos_token_id
        self.bos_id = self.hf.bos_token_id if self.hf.bos_token_id is not None else self.eos_id
        self.pad_id = self.hf.pad_token_id if self.hf.pad_token_id is not None else self.eos_id
        self.vocab_size = len(self.hf)
        self.tokenizer = self

    def encode(self, text, add_special_tokens=False):
        return self.hf.encode(text, add_special_tokens=add_special_tokens)

    def decode(self, ids, skip_special_tokens=False):
        return self.hf.decode(ids, skip_special_tokens=skip_special_tokens)

    def id_to_token(self, idx):
        return self.hf.convert_ids_to_tokens([int(idx)])[0]

    def tokens(self, text):
        return self.hf.tokenize(text)


class HFModelWrapper(nn.Module):
    """Matches TinyGPT's call convention -- model(xb, yb) -> (logits, loss),
    model.generate(idx, max_new_tokens=, eos_id=, top_k=) -> full sequence."""

    def __init__(self, hf_model):
        super().__init__()
        self.hf_model = hf_model

    def forward(self, xb, yb=None):
        logits = self.hf_model(input_ids=xb).logits
        loss = None
        if yb is not None:
            loss = F.cross_entropy(
                logits.view(-1, logits.size(-1)),
                yb.view(-1),
                ignore_index=-100,
            )
        return logits, loss

    def generate(self, idx, max_new_tokens, eos_id=None, top_k=None, temperature=None, do_sample=None):
        return self.hf_model.generate(
            input_ids=idx,
            max_new_tokens=max_new_tokens,
            eos_token_id=eos_id,
            pad_token_id=eos_id,
            top_k=top_k,
            do_sample=do_sample if do_sample is not None else (top_k is not None or temperature is not None),
            temperature=temperature if temperature is not None else 1.0,
        )

    @property
    def vocab_size(self):
        return self.hf_model.config.vocab_size


def load_hf_checkpoint(path, device):
    hf_model = AutoModelForCausalLM.from_pretrained(
        path,
        torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
        attn_implementation="sdpa",
    ).to(device)
    return HFModelWrapper(hf_model).to(device)


tokenizer = QwenTokenizerAdapter(REPO)
model = load_hf_checkpoint(REPO, DEVICE).eval()

Prompt format

Same tag vocabulary as the -direct model, but here <HISTORY> is the whole point — it accumulates prior <ACTION> entries as the loop progresses, and <CODE> reflects the file's state after those prior steps:

<GOAL>
Add TextEditingControllers for the email and password fields, connect them to their respective TextFields, implement a _submit method that displays the entered email and password length in a SnackBar, and add spacing above the login button.
</GOAL>

<CODE>
import 'package:flutter/material.dart';

void main() => runApp(MaterialApp(home: Scaffold(body: Center(child: EmailInputForm()))));
</CODE>

<HISTORY>
<ACTION><TYPE>add_widget</TYPE><DESC>Add a SizedBox widget for vertical spacing.</DESC></ACTION>
</HISTORY>

<OUTPUT>

A single step's raw output looks like:

<ACTION><TYPE>...</TYPE><DESC>...</DESC></ACTION>
<CHANGES>
<HUNK>
<SEARCH>
...
</SEARCH>
<REPLACE>
...
</REPLACE>
</HUNK>
</CHANGES>

with a trailing <DONE></DONE> on the final step.

Generate — single step

def render_step_prompt_from_data(
    goal: str,
    code: str = "",
    history: list[dict] | None = None,
    action_type: str = "",
    action_desc: str = "",
    changes: list[dict] | None = None,
    is_last_step: bool = False,
    include_output: bool = False,
) -> str:
    history = history or []
    changes = changes or []
    history_text = "\n".join(
        f"<ACTION><TYPE>{h['type']}</TYPE><DESC>{h['desc']}</DESC></ACTION>"
        for h in history
    )
    text = f"""<GOAL>
{goal}
</GOAL>

<CODE>
{code}
</CODE>

<HISTORY>
{history_text}
</HISTORY>

<OUTPUT>
"""
    if include_output:
        hunks = "\n".join(
            f"<HUNK>\n<SEARCH>\n{h['search']}\n</SEARCH>\n<REPLACE>\n{h['replace']}\n</REPLACE>\n</HUNK>"
            for h in changes
        )
        output = f"<ACTION><TYPE>{action_type}</TYPE><DESC>{action_desc}</DESC></ACTION>\n<CHANGES>\n{hunks}\n</CHANGES>"
        if is_last_step:
            output += "\n<DONE></DONE>"
        text += output + "\n</OUTPUT>"
    return text


def generate_output(model, tokenizer, device, prompt, max_new_tokens=300, **generate_kwargs):
    prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
    idx = torch.tensor([[tokenizer.bos_id] + prompt_ids], dtype=torch.long).to(device)
    generated = model.generate(idx, max_new_tokens=max_new_tokens, eos_id=tokenizer.eos_id, **generate_kwargs)
    text = tokenizer.decode(generated[0].tolist(), skip_special_tokens=False)
    return text.split("<OUTPUT>")[-1].split("</OUTPUT>")[0].strip()

Generate — the full loop

Since this model's whole point is iterating until <DONE>, the meaningful way to run it is the multi-step loop (parses each step's <ACTION>/<CHANGES>, applies the diff, appends to history, and continues):

result = run_step_sequence(
    model, tokenizer, DEVICE,
    goal="Write a widget that displays a select button with months in it",
    code="",
    history=[],
    max_steps=20,
    max_new_tokens=300,
)
print(result["final_code"])
print(result["stop_reason"])

run_step_sequence depends on a few more helpers (generate_output_data, parse_generated_output, apply_edit/apply_edit_with_fallback) that parse the tag-structured output and apply each hunk — see [the training/eval code repo] for the full implementation. These are identical to the ones used by the from-scratch TinyGPT models in this collection, since generate_output/model.generate behave the same way once wrapped.

Example

Input: goal + starting code + one prior history step (shown in "Prompt format" above)

result = run_step_sequence(
    model, tokenizer, DEVICE,
    goal="Write a widget that displays a select button with months in it""",
    history=[
        # {
        #     "type": "add_widget",
        #     "desc": "Make a full running program that would include givven widget",
        # }
    ],
    code="""""",
    max_steps=20,
    max_new_tokens=300,
)

Output trace:

[step 1] add_import: Add the import statement 'import 'package:flutter/material.dart';' (1 hunk)
[step 1] code:
import 'package:flutter/material.dart';

-------------------------------------------

[step 2] add_class: Add the class SelectMonthButton with its createState method. (1 hunk)
[step 2] code:
import 'package:flutter/material.dart';

class SelectMonthButton extends StatefulWidget {
  @override
  _SelectMonthButtonState createState() => _SelectMonthButtonState();
}

-------------------------------------------

[step 3] add_class: Add the _SelectMonthButtonState class with an empty body. (1 hunk)
[step 3] code:
import 'package:flutter/material.dart';

class SelectMonthButton extends StatefulWidget {
  @override
  _SelectMonthButtonState createState() => _SelectMonthButtonState();
}

class _SelectMonthButtonState extends State<SelectMonthButton> {
}

-------------------------------------------

[step 4] add_method: Add the build method to _SelectMonthButtonState class. (1 hunk)
[step 4] code:
import 'package:flutter/material.dart';

class SelectMonthButton extends StatefulWidget {
  @override
  _SelectMonthButtonState createState() => _SelectMonthButtonState();
}

class _SelectMonthButtonState extends State<SelectMonthButton> {
  @override
  Widget build(BuildContext context) {
    return Column(
      mainAxisSize: MainAxisSize.min,
      children: [
        ElevatedButton(
          onPressed: _selectMonth,
          child: Text('Select Month'),
        ),
      ],
    );
  }
}

-------------------------------------------

[step 5] add_field: Add the field 'int selectedIndex = 0;' to the _SelectMonthButtonState class. (1 hunk)
[step 5] code:
import 'package:flutter/material.dart';

class SelectMonthButton extends StatefulWidget {
  @override
  _SelectMonthButtonState createState() => _SelectMonthButtonState();
}

class _SelectMonthButtonState extends State<SelectMonthButton> {
  int selectedIndex = 0;
  @override
  Widget build(BuildContext context) {
    return Column(
      mainAxisSize: MainAxisSize.min,
      children: [
        ElevatedButton(
          onPressed: _selectMonth,
          child: Text('Select Month'),
        ),
      ],
    );
  }
}

-------------------------------------------

[step 6] add_method: Add the _selectMonth method to handle selecting a month and updating the state. (1 hunk)
[step 6] code:
import 'package:flutter/material.dart';

class SelectMonthButton extends StatefulWidget {
  @override
  _SelectMonthButtonState createState() => _SelectMonthButtonState();
}

class _SelectMonthButtonState extends State<SelectMonthButton> {
  int selectedIndex = 0;

  void _selectMonth() {
    setState(() {
      selectedIndex = (selectedIndex + 1) % 12;
    });
  }
  @override
  Widget build(BuildContext context) {
    return Column(
      mainAxisSize: MainAxisSize.min,
      children: [
        ElevatedButton(
          onPressed: _selectMonth,
          child: Text('Select Month'),
        ),
      ],
    );
  }
}

-------------------------------------------

[step 7] hunk 1/1 SEARCH matched 2 times -- using last occurrence
FALLBACK
[step 7] add_widget: Add the Row widget with the list of month names as children to the Column in the build method. (1 hunk)
[step 7] code:
import 'package:flutter/material.dart';

class SelectMonthButton extends StatefulWidget {
  @override
  _SelectMonthButtonState createState() => _SelectMonthButtonState();
}

class _SelectMonthButtonState extends State<SelectMonthButton> {
  int selectedIndex = 0;

  void _selectMonth() {
    setState(() {
      selectedIndex = (selectedIndex + 1) % 12;
    });
  }
  @override
  Widget build(BuildContext context) {
    return Column(
      mainAxisSize: MainAxisSize.min,
      children: [
        ElevatedButton(
          onPressed: _selectMonth,
          child: Text('Select Month'),
        ),
        Row(
          mainAxisAlignment: MainAxisAlignment.center,
          children: List.generate(11, (index) {
            final monthName = monthNames[index];
            return TextButton(
              onPressed: () => _selectMonth(),
              child: Text(monthName),
            );
          }),
        ),
      ],
    );
  }
}

-------------------------------------------

[step 8] add_field: Add the field 'final List<String> monthNames' with the list of month names to the class _SelectMonthButtonState. (1 hunk) [DONE]
[step 8] code:
import 'package:flutter/material.dart';

class SelectMonthButton extends StatefulWidget {
  @override
  _SelectMonthButtonState createState() => _SelectMonthButtonState();
}

class _SelectMonthButtonState extends State<SelectMonthButton> {
  int selectedIndex = 0;

  final List<String> monthNames = [
    'January',
    'February',
    'March',
    'April',
    'May',
    'June',
    'July',
    'August',
    'September',
    'October',
    'November',
    'December',
  ];

  void _selectMonth() {
    setState(() {
      selectedIndex = (selectedIndex + 1) % 12;
    });
  }
  @override
  Widget build(BuildContext context) {
    return Column(
      mainAxisSize: MainAxisSize.min,
      children: [
        ElevatedButton(
          onPressed: _selectMonth,
          child: Text('Select Month'),
        ),
        Row(
          mainAxisAlignment: MainAxisAlignment.center,
          children: List.generate(11, (index) {
            final monthName = monthNames[index];
            return TextButton(
              onPressed: () => _selectMonth(),
              child: Text(monthName),
            );
          }),
        ),
      ],
    );
  }
}

-------------------------------------------

Training details

Base model	Qwen/Qwen2.5-Coder-0.5B (~0.5B params, pretrained by Alibaba/Qwen)
Fine-tuning	50M tokens, multi-step goal → history → diff sequences
Tokenizer	Qwen's native BPE tokenizer, extended with structural special tokens

Related

• Companion model (single-pass, whole-file training, same base): bbidpa/Qwen2.5-Coder-0.5B-Flutter-direct
• Same comparison, 100M model trained from scratch: bbidpa/Rainbow-Pony-100m-Flutter-steps
• Training dataset: bbidpa/flutter-diff-steps-v1
• Training/eval code: [Upcoming]