GeMri98/hvac-twin-explainer-v4

HVAC twin agent v5: a fine-tuned LLM that calls a thermal-twin simulator as a tool

Research prototype on synthetic data (Melbourne-style weather, VIC1-style prices). The model writes <call> lines for a simulate tool backed by an RC thermal twin, reads TOOL_RESULTS, then answers. Gold answers are computed by the twin; the grader checks the tool calls, the verdict, the required numbers and invented numbers.

• adapter_v5/: v4 adapter + tool-use training. On the first test set (earlier, more lenient grader): 50.7% (v4, no tools) to 85.3% correct. Flaw found: level questions were refused, because the only training questions with that wording were out-of-range (invalid) ones.
• adapter_v5_1/: continued from v5 with level trained on valid numbers. This is the recommended adapter.
• A further targeted round on worth and override (called v5.2, not published) changed overall results by under 1 point, within noise.

Results for adapter_v5_1 (held-out days, full graded strictly)

Normal unseen days (n=300): 93.3% correct, tool calls right 95%, full 4 of 18, worth 78%, override 97%. Heatwave + price-spike days (n=300): 89.7% correct, tool calls right 93%, full 5 of 28, worth 83%, override 84%. With the more lenient first grader these were 98.0% and 97.0%; the strict grader is the one to trust.

What this does and does not show

• Seen question types are at or near 100%, tool calls are chosen correctly about 94% of the time, and no numbers are invented.
• full is the one question type never trained on. Graded strictly (the answer must state both the 100% result and the current-setting result) it scores about 20%. Fine-tuning here did not generalise to a new question type. A first, lenient check only required the 100% numbers and showed ~100%; it was wrong and is superseded.
• worth (yes/no from comparing two outcomes) is the weakest trained type: the verdict is written before the numbers are compared, and it fails most on ties.
• Not tested here: free-form questions, other buildings, real weather or tariff data.

Limitations

Synthetic data, one zone, template-style gold answers, about 18-30 test questions per type (roughly +/-10 points or more per type). The grader checks that numbers appear in the data, not that they are attributed to the right scenario.