bus-chat-model

A LoRA adapter over Qwen/Qwen3-32B trained on synthetic business-education passages. It is stage one of a business-tutor model.

Status: trained but not yet evaluated. Training loss fell cleanly, but the adapter has not been measured on held-out data, because no serving path has been available. Treat its quality as unknown — see Evaluation before relying on it for anything.

What it is

The adapter targets business-domain knowledge across six areas: finance, accounting, marketing, information systems, economics, and management. It was trained on expository textbook-style passages, so it is a knowledge adaptation, not an instruction- or chat-tuned model. It does not change the base model's conversational behavior.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = "Qwen/Qwen3-32B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "ashcash15/bus-chat-model")

With vLLM (serves base and adapter side by side on one endpoint):

vllm serve Qwen/Qwen3-32B --enable-lora \
  --lora-modules bus=ashcash15/bus-chat-model \
  --max-lora-rank 32 --max-model-len 8192

Hardware: the base model needs ~65 GB of GPU memory at bf16 (one A100/H100 80 GB), or ~18 GB quantized to 4-bit.

Note on <think>: Qwen3 emits reasoning blocks by default and will readily spend an entire token budget thinking without producing an answer. Strip <think>…</think> from outputs, or disable thinking, before showing results to a user.

Training

Base model	Qwen/Qwen3-32B
Method	LoRA, r=32, α=64, dropout 0.05
Target modules	q, k, v, o, gate, up, down projections
Epochs	4
Batch size	4
Learning rate	1e-4
Context length	8192, sequence packing on
Tokens trained	19,998,020
Platform	Nebius Token Factory
Duration / cost	37 minutes, ~$56

Training loss by epoch: 1.80 → 1.34 → 1.01 → 0.71 (no plateau; the model was still improving when training ended).

Four per-epoch checkpoints were produced; the published weights are the final one (step 700).

Training data

862 synthetic passages (~1.4M tokens, ~1.02M words) generated and verified by a two-stage pipeline:

1. Generation — Kimi-K2.6 wrote each passage from a topic drawn round-robin across the six domains (200 topics per domain).
2. Deterministic gate — free code checks for length, truncation, refusal and placeholder text, degenerate repetition, topic presence, and near-duplication against already-accepted passages.
3. LLM judge — gpt-5.4 graded factual soundness and instructional clarity; passages scoring below threshold were rejected with a critique, retried up to twice, and each rejection distilled a one-line rule appended to a rolling "lessons" file that every subsequent generation prompt inherited.

72% of attempts were accepted. The training mix was this corpus plus FineWeb-Edu replay text (~3.7M tokens) to limit catastrophic forgetting, run for 4 epochs. A seeded 95/5 split held out 43 passages, which have never been trained on.

Evaluation

None yet. This is the honest headline.

A deterministic cloze benchmark exists and was run on the base model to establish a baseline: salient business terms and figures are masked out of the 43 held-out passages and the model must fill them in.

Model	Cloze recall
Qwen/Qwen3-32B (base)	0.30
This adapter	not yet measured

The base model's errors were diagnostic: every correct answer was a generic English word, every miss was business-specific (interoperability, Double Marginalization, a dollar figure). So the metric has headroom, and a meaningful gain would show real domain absorption.

The adapter is unmeasured because no serving route has been available — the training platform does not serve fine-tuned weights without beta access, and GPU capacity elsewhere has been unobtainable. A falling training loss is not evidence of learning: with only ~1.33M unique corpus tokens seen four times, that 0.71 could equally reflect memorization. Until the held-out number exists, treat this model as unvalidated.

Limitations and bias

• Entirely synthetic training data. The corpus was written by a language model and checked by another language model. Both stages can share blind spots, and no human subject-matter expert reviewed the passages. Factual errors that both models agree on will have passed through.
• Unvalidated. No held-out score, no general-capability retention check (MMLU or otherwise). Whether it forgot general ability is unknown.
• Not instruction-tuned. Trained on expository prose; it has no tutoring, dialogue, or pedagogical behavior beyond what the base model already had.
• US-centric business framing. Topics follow a US university business curriculum; accounting and regulatory content reflects that context.
• Not suitable for advice. Do not use for financial, legal, accounting, or investment decisions.

Intended use

Research and development on domain-adapted educational models. It is a checkpoint in an ongoing pipeline, not a finished product.

Out of scope: production deployment, student-facing use without evaluation and human review, and any application where a factual error carries cost.

Citation

@misc{bus-chat-model,
  title  = {bus-chat-model: a business-knowledge LoRA adapter for Qwen3-32B},
  author = {Castelino, Ashleyn},
  year   = {2026},
  url    = {https://huggingface.co/ashcash15/bus-chat-model}
}