GeMri98/hvac-twin-agent-v5

HVAC twin agent: one 3B LoRA that runs a thermal twin as a tool and answers from NCC Section J

Adapters: adapter_v5_1 (twin tool use only), adapter_v6 (twin tool use + grounded NCC answers; recommended). Also in this repo: rag_index/ (retrieval index), rag_train.jsonl, rag_test.jsonl, qa_*_v51.jsonl.

What it does

• Twin tool use: the model writes <call>{"tool":"simulate","cooling_pct":N}</call>; a 2R2C office-zone thermal twin runs; the model answers from the results.
• NCC answers: from retrieved NCC 2022 Vol 1 Section J passages (J1, J2, J6, J9), with [S#] citations, or "Not found in the provided sources."
• Checker (in the pipeline, not the weights): every answer sentence is checked against the retrieved passages and every number against the data or twin results; failures are redrafted once, then abstained.

Results (small, single-run; read the limits)

Test	Result
Twin tool use, seen question types, strict grading (v5.1)	93.3% normal day, 89.7% heatwave
Twin tool use, UNSEEN question type ("full cooling")	about 20% - no real generalisation
v6 keeps the <call> protocol (first turn matches gold call, 16 held-out rows)	14/16 (same as v5.1)
NCC held-out rows, grounded + cited (12): plain Qwen / v5.1 / v6	9 / 3 / 10
20 hand-posed NCC questions (16 answerable, 4 not answerable), with checker	12 answered, 4 abstained; 4/4 unanswerable refused; 0 clearly wrong on review
Twin numbers in the agent demo	matched a direct run of the twin exactly

Limits (important)

• v6 is not better than plain Qwen at reading the NCC; it is better at citing consistently and refusing, and it keeps the twin skill. Reference answers were written by plain Qwen, so it cannot exceed its teacher.
• 20 hand questions, drafted by an AI and spot-checked by the author against ncc.abcb.gov.au (3 of 12 answers checked). One run. Not a benchmark.
• The checker verifies faithfulness to the retrieved text, not relevance: two vague answers passed.
• It cannot decide whether a building scenario complies with a clause.
• Index mixes in some state variation text (e.g. NSW). Assume nothing about Victoria-specific rules.
• Synthetic twin and scenarios; not validated on a real building. Not for compliance or design decisions.

Data and attribution

NCC text: (c) Commonwealth of Australia and States and Territories, published by the Australian Building Codes Board, CC BY 4.0 (ncc.abcb.gov.au). The NABERS guide PDF is not redistributed in this repo.