bus-chat-35b

A LoRA adapter for Qwen/Qwen3.6-35B-A3B that teaches business concepts conversationally: it explains, checks understanding, and adjusts how technical it is based on how the learner is doing.

Trained in two stages:

1. Continued pretraining on 862 verified business passages (~1.4M tokens) spanning accounting, economics, finance, information systems, management and marketing, mixed with general-web replay data.
2. Supervised fine-tuning on 1,068 synthetic tutoring dialogues in which the tutor's technical level moves both up and down within a conversation — up when the learner is ready, down when they show confusion. Only the tutor's turns carry loss.

LoRA rank 32 throughout, one epoch of each stage — see Evaluation for why.

Evaluation

Measured against the unmodified base model on held-out data the adapter never saw. Business perplexity is exp(mean cross-entropy) over 43 held-out passages. Recall is a deterministic cloze test over the same held-out set (86 items, 4000-token budget). MMLU is 160 questions across marketing, management, econometrics and professional accounting, scored by answer likelihood.

	recall ↑	answered	business perplexity ↓	MMLU ↑
Qwen/Qwen3.6-35B-A3B (base)	0.4942	84/86	3.2932	0.8313
after stage 1 (CPT)	—	—	2.8021 (−14.9%)	0.8375
+ this adapter (SFT 1 epoch)	0.5116	85/86	2.8417 (−13.7%)	0.8313
SFT 2 epochs	0.5407	84/86	2.9192 (−11.4%)	0.8438
SFT 3 epochs	0.5349	—	3.0943 (−6.0%)	—
SFT 4 epochs	0.4593	—	3.3287 (+1.1%)	—

Continued pretraining produced the largest perplexity gain but left the model less willing to answer a question directly. The SFT stage restored that and improved on the base model's answering, at the cost of part of the perplexity gain. No stage reduced MMLU — general business ability is slightly above the base model at every checkpoint, so the domain training did not come at the expense of what the model already knew.

Training past one SFT epoch overfits. Across four epochs, training loss fell 64% while held-out perplexity degraded at every step, until the four-epoch model was worse than base on both perplexity and recall. The apparent recall advantage of the two-epoch checkpoint (0.5407 vs 0.5116) is not statistically distinguishable from noise (11 items better, 8 worse of 86; p = 0.648), and what signal it has sits in generic vocabulary rather than business terms. This adapter is the one-epoch checkpoint.

No learning-rate schedule was used — a flat 1e-4 throughout — so this result is specific to that configuration rather than a general claim about the data.

Not measured: safety behaviour, and performance on business material outside the six domains above. Teaching quality was assessed qualitatively, not by a benchmark. Treat the model as a research artefact, not a deployed tutor.

Intended use

Explaining business concepts to learners, at a technical level it adapts as the conversation goes. It is a research artefact for studying whether synthetic teaching data transfers pedagogy to a smaller open model.

It is not suitable for financial, legal, tax or accounting advice, for grading, or for any decision affecting a person. Its knowledge comes from synthetic passages written by a language model and filtered by another; factual errors survive that process.

Usage

The adapter expects a system turn describing the tutor role and the learner, which is the shape it was trained on:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3.6-35B-A3B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "ashcash15/bus-chat-35b")

messages = [
    {"role": "system", "content":
     "You are an expert business tutor at a university, teaching one student "
     "one-to-one. Adjust how technical you are to the student in front of you: "
     "go simpler when they show confusion, go deeper when they are ready or "
     "ask for it.\n\nTHE STUDENT\n- a junior business major, first accounting "
     "course, uneasy with anything that looks like calculus"},
    {"role": "user", "content": "I don't really get accruals. Can you help?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False,
                                 add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))

Serving with vLLM:

vllm serve Qwen/Qwen3.6-35B-A3B --enable-lora \
  --lora-modules bus=ashcash15/bus-chat-35b

Training data

The corpus and the tutoring dialogues are both synthetic. Passages were written by Kimi-K2.6 and screened by a deterministic code gate plus a GPT-5.4 judge. Dialogues were written by DeepSeek-V4-Pro against a sampled learner profile, screened by a structural gate (which enforced that difficulty moved in both directions) and a five-dimension GPT-5.4 rubric covering scaffolding, whether each difficulty shift was justified, misconception handling, level fit and learner-voice realism.

Limitations

• Synthetic training data throughout; no human-authored ground truth.
• Knowledge is limited to the six domains listed above.
• The base model reasons before answering; budget tokens accordingly.
• LoRA rank 32 over all linear layers; 431 non-zero lora_B tensors.
• Evaluated on cloze recall, perplexity and MMLU only — none measures teaching quality directly.

Citation

@misc{bus-chat-35b,
  title  = {bus-chat-35b: a two-stage business tutoring LoRA adapter for Qwen3.6-35B-A3B},
  author = {Ashley Castelino},
  year   = {2026},
  url    = {https://huggingface.co/ashcash15/bus-chat-35b}
}