QLORA SFT Distillation Effects on Qwen3.6 27B Agentic Coding Harness Fluency

This writeup documents my findings while researching the effects of QLoRA SFT distillation on Qwen3.6 27B for agentic coding harness fluency.

The results are based primarily on Terminal-Bench 2.0 evaluations.

All benchmark results are reported as single-run pass@1.

---

TL;DR

I fine-tuned and evaluated Qwen3.6 27B across multiple agentic coding harnesses and inference setups.

The main finding is that harness-specific fine-tuning can materially change behavior, but the effect is highly sensitive to the training traces, reasoning format, and harness interface.

For the Pi harness:

The base model remained the strongest Pi run, but the v2 reasoning-distilled model recovered most of the base performance and showed a different behavioral profile: stronger decomposition and validation on some systems/data tasks, but more tendency to over-explore and time out when it failed to converge.

---

1. Background

The goal of this work was to investigate agentic coding harness fluency with Qwen3.6 27B, an open-weight model that is both capable and runnable on consumer hardware.

The original inspiration came from Terminal-Bench, a benchmark of realistic terminal-based tasks that evaluates agentic software and system development capabilities.

Benchmarks from the team behind Terminal-Bench show that performance varies materially by harness.

For example, reported GPT-5 Terminal-Bench 2.0 results include:

I chose Qwen3.6 27B because of its strong coding ability and its practicality for local or consumer-hardware inference.

According to the Qwen release, Qwen3.6 27B reaches 59.3% with Terminus 2 in BF16:

• Qwen3.6 27B on Hugging Face

The research question was:

Can frontier harness fluency be distilled into a strong open-weight model while keeping the model practical to run on consumer hardware?

---

2. Harness Experiments

I fine-tuned and tested Qwen3.6 27B across three agentic harnesses and two inference engines.

The harnesses were:

1. Codex CLI
2. OpenHands
3. Pi

---

2.1 Codex CLI

Codex CLI was the first target because GPT-5.5 with Codex CLI was the highest-ranking entry on the Terminal-Bench 2.0 leaderboard at the time I began experimenting.

However, I ran into several practical issues:

• Poor support for local models
• Malformed tool calls
• Parser and streaming mismatches
• Chat template issues
• High context usage
• Infrastructure errors with agent environments and cloud GPU providers

After retrials and some manual stitching, I obtained the following unofficial baseline:

Due to the challenges above, I moved from Codex CLI to OpenHands.

---

2.2 OpenHands

OpenHands was a better candidate for local model experimentation. It is designed with local models in mind and has stronger first-class integrations for this use case.

OpenHands baseline and fine-tuned results:

v1

The v1 model was fine-tuned on a dataset of valid, passed trajectories from various agentic coding tasks and benchmarks.

Some task providers included:

• BFCL
• SWE-bench Verified
• Additional software and systems development task sources

The dataset contained approximately 1,000 curated trajectories intended to improve general software and systems development behavior in the OpenHands terminal environment.

These trajectories were collected with reasoning disabled.

Result:

v1 improved by roughly 10 points over the base OpenHands run.

v2

The v2 model was fine-tuned on the existing approximately 1,000 frontier teacher trajectories, plus roughly 500 additional passed trajectories with reasoning enabled.

This made the dataset approximately:

v2 showed a major regression.

It often made less progress per turn and got stuck in longer loops. My current hypothesis is that v2’s dataset may have shifted the model toward a more reflective style without enough action-taking behavior.

Overall, OpenHands provided evidence that harness-specific fine-tuning can improve base model behavior. However, after using OpenHands directly, I decided that it was not the right agentic interface for my own workflow.

---

2.3 Pi

After further experimentation, I moved to Pi because of its lightweight and minimal design.

Pi was the closest candidate to a barebones harness like Terminus. It was also a strong fit for my own development workflow.

Pi’s core design revolves around being small, explicit, and customizable. Its tool surface was ideal for fine-tuning experiments because it only exposed a small set of primitives:

• read
• bash
• edit
• write

Pi was attractive because its architecture made fine-tuning facilitation simpler and more stable. Its customizability also leaves room for flexible combinations of model fine-tuning and Pi extensions.

Pi results:

Between the base model and v1, there was a roughly 13-point regression.

v2 recovered most of the base model’s performance, although it did not exceed the base run.

---

3. Pi Fine-Tuning Iterations

v1

The v1 teacher dataset was composed primarily of closed-weight frontier model trajectories without reasoning traces.

The revised training set contained approximately 1,600 curated trajectories.

Observed regressions included:

• Tool-call failures
• Behavioral issues related to refusal
• Reasoning loops
• Reduced progress on some tasks

One possible factor is that some trajectories may have been poorly exported. This is something I plan to investigate further.

---

v2

The v2 teacher dataset was composed of open-weight frontier model trajectories with full, unencrypted reasoning chains of thought.

The dataset contained approximately 1,200 passed trajectories.

Unlike v1, reasoning was treated as a first-class training target. The goal was to preserve or distill reasoning-mode behavior into the base model rather than training only on final actions.

Compared to v1, v2 recovered most of the base model’s Pi performance.

After analyzing base and v2 trajectories using Docent, v2 showed:

• Stronger task decomposition
• Stronger targeted debugging
• Stronger end-to-end validation on infrastructure and data tasks

However, this same exploratory behavior can also become a weakness. When v2 does not converge on a solution, it tends to over-explore and time out.

One important caveat:

The v2 run used a 1-hour timeout, while the base run used a 3-hour timeout.

Because of this, timeout-related comparisons should be interpreted carefully.

---

4. Experimental Setup

Hardware: GPU / Cloud
Runtime: llama.cpp / vLLM
Model: Qwen3.6 27B
Dataset: Private
Eval: Terminal-Bench 2.0

I explored vLLM extensively as an inference engine, including existing patches such as Club 3090 and my own modifications.

In practice, vLLM was too unstable for generating high-quality teacher traces in this setup.

llama.cpp was the most reliable option. It had longer runtimes, but the reliability tradeoff was acceptable.

---

5. Main Pi Results

The Pi harness results below are single pass@1 Terminal-Bench 2.0 runs over the 89-task set.

Base Qwen3.6 27B Q4 + Pi

Qwen3.6 27B Q4 v1 + Pi

Qwen3.6 27B Q4 v2 + Pi

---

6. Interpretation

The base Qwen3.6 27B Q4 model remained the strongest Pi run.

However, v2 was close to the base model and showed a noticeably different behavioral profile.

What improved in v2

Based on trajectory analysis, v2 appeared stronger at:

• Breaking tasks into smaller steps
• Running more targeted debugging loops
• Performing end-to-end validation
• Handling some infrastructure and data tasks more systematically

What regressed or remained risky

v2 also appeared more prone to:

• Over-exploration
• Longer loops
• Timeout failures when it failed to converge
• Spending too much effort validating or investigating instead of finalizing

The result suggests that reasoning distillation may improve some forms of harness fluency, but it can also increase the model’s tendency to keep exploring.

---

7. Next Steps

Planned follow-ups:

1. Run another evaluation benchmark.
2. Train an iteration without a stop condition.
3. Explore additional task providers.
4. Publish additional artifacts.
5. Improve prompts, serving settings, and training data.

---

8. Acknowledgements

Thanks to the Terminal-Bench team for building and maintaining Terminal-Bench 2.0, and to the Qwen team for releasing Qwen3.6 27B as a capable open-weight model for coding and agentic evaluation.

This work also depended on the open-source infrastructure around local model serving and agent harnesses, including:

• llama.cpp
• vLLM
• OpenHands
• Codex CLI
• Pi

Thanks to the maintainers and contributors of those projects.

Thanks also to the Hugging Face ecosystem for model and dataset hosting, and to Docent for trajectory analysis tooling used to compare runs, inspect failures, and identify behavioral patterns across the base, v1, and v2 models.

---

Appendix

Evaluation Setup

All reported Pi results are single pass@1 Terminal-Bench 2.0 runs over the 89-task set.

---

Run Labels

---

Dataset Notes

The v1 model was trained primarily from passed frontier teacher trajectories without explicit reasoning traces.

The v2 model added reasoning-enabled trajectories and was intended to better preserve or distill reasoning-mode behavior.

The datasets are private.

---

Important Caveats

These results should be read as an applied harness-fluency study, not as a complete measurement of Qwen3.6 27B coding ability.

Important caveats:

• All reported results are single-run pass@1.
• The v2 run used a shorter timeout than the base run.
• Some differences may reflect runtime budget, stochasticity, or harness-level behavior rather than pure model capability.
• The datasets are private, so the exact training mixture cannot be independently reproduced from this post alone.
• Some Codex CLI results are unofficial and involved retries or stitching due to infrastructure issues.

---

llama.cpp Configuration

The Pi evaluations used an OpenAI-compatible llama.cpp server.

The llama.cpp server was run with the following settings:

---

Model Files

Notes:

• top_k=0 effectively disabled top-k sampling.
• min_p=0.0 effectively disabled min-p sampling.
• Pi and the bridge primarily set temperature, top_p, and max_tokens per request.

---

Closing Notes

This is my first blog post, so feedback, corrections, and questions are very welcome.

I am especially interested in whether the writeup is clear. If something is confusing, missing, or worth expanding, I would appreciate hearing about it.

I did not detail the entire scope of the journey here, but I am happy to discuss more.

The goal of this post is to share what I tried, what I observed, and what I learned along the way. I expect there are details I can explain better, and I plan to improve future writeups based on feedback.