chrischarts8/pcare-dim4-phase2-comparator

PCare DIM4 Phase 2 — Clinical Outcome Comparator

Fine-tuned Qwen3-8B with LoRA (r=256, alpha=256, all-linear) for binary clinical outcome comparison (PASS/FAIL).

Task

Compare an expected clinical outcome (from document analysis) against an actual clinical outcome (from a compiled FHIR artifact) and judge semantic equivalence.

Input: Two clinical outcome strings
Output: JSON with verdict (PASS/FAIL), reasoning, and mismatches

Evaluation Results

Metric	Base Qwen3-8B	Fine-tuned	Target	Status
Overall accuracy	82.27%	100.00%	>95%	✅
Safety miss rate	23.24%	0.00%	<1%	✅
False positive rate	12.74%	0.00%	<10%	✅
JSON parse error rate	0.00%	0.00%	0%	✅
Latency (tok/s)	22.6	12.3	>50	⚠️

All 4 primary targets met. The model achieves perfect accuracy with zero safety misses on the 299-example eval set spanning 8 clinical domains and 15 scenario types.

Confusion Matrix (Fine-tuned)

	Predicted PASS	Predicted FAIL
True PASS	157 (TN)	0 (FP)
True FAIL	0 (FN)	142 (TP)

Training Loss Trajectory

Step	Epoch	Loss	Token Accuracy	Eval Loss
1	0.01	3.112	73.1%	—
20	0.27	0.259	92.9%	—
50	0.67	0.015	99.6%	0.0125
100	1.33	0.005	99.9%	0.0035
150	2.00	0.0003	100.0%	0.0009
224	3.00	~0	100.0%	~0

Training Details

• Base model: Qwen/Qwen3-8B (Apache 2.0)
• Method: SFT with LoRA (r=256, alpha=256, all-linear targets, dropout=0.05)
• Dataset: chrischarts8/pcare-dim4-phase2-synthetic — 1,196 train / 299 eval examples
• Epochs: 3
• Effective batch size: 32 (4 × 4 grad accum × 2 GPUs)
• Learning rate: 2e-4 (cosine schedule, 10% warmup)
• Loss: NLL with assistant_only_loss=True
• Hardware: 2×A10G (48GB total)
• Trainable parameters: 698M / 8.9B total (7.86%)
• Training time: ~1.5 hours

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "chrischarts8/pcare-dim4-phase2-comparator")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

system_prompt = """You are a clinical outcome comparator. Compare two sets of clinical outcomes for the same patient. Outcomes may be recommendations, diagnostic classifications, risk levels, or other clinical decisions.

## Rules
1. Different clinical outcomes = FAIL (even if both are 'reasonable').
2. Same clinical outcome with different wording = PASS.
3. One outcome is more specific than the other but clinically consistent = PASS.
4. Missing an outcome domain that the expected has = FAIL.
5. Extra outcome domains not in the expected are acceptable = PASS.
6. If the expected says 'not indicated' / 'excluded' and actual says the equivalent = PASS.

## Output Format
Return ONLY a JSON object:
{"verdict": "PASS" or "FAIL", "reasoning": "brief explanation", "mismatches": ["list of mismatches, empty if PASS"]}"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "## Expected Outcome (from document analysis)\nScreen for colorectal cancer with colonoscopy every 10 years, starting at age 45\n\n## Actual Outcome (from compiled artifact)\n[Colorectal cancer screening] Colonoscopy every 10 years — ages 45-75 (Grade A)"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# {"verdict": "PASS", "reasoning": "Both outcomes recommend colorectal cancer screening with colonoscopy every 10 years starting at age 45. The actual is more specific (adds age range and Grade A) but clinically consistent.", "mismatches": []}

Clinical Domains Covered

1. Cancer screening — colorectal, breast, cervical, lung, prostate
2. Diagnostic classification — transfusion reactions (TRALI/TACO), sepsis (qSOFA/SOFA), stroke (TOAST)
3. Treatment guidelines — antibiotics, anticoagulation, pain management
4. Risk stratification — cardiovascular (Framingham, ASCVD), fall risk, nutrition
5. Immunization schedules — childhood, adult, travel vaccines
6. Preventive care — wellness visit components, lifestyle counseling
7. Referral criteria — specialist referral thresholds
8. Monitoring protocols — lab monitoring frequency, imaging follow-up

Comparison Rules Learned

Rule	Example
Different outcomes = FAIL	"Screen every 3 years" vs "Screen every 5 years"
Same meaning, different words = PASS	"Do not screen" vs "Screening not indicated"
More specific but consistent = PASS	"Screen appropriately" vs "Colonoscopy every 10 years — ages 45-75"
Missing domain = FAIL	Expected has screening + follow-up, actual only has screening
Extra domains = PASS	Expected has 1 domain, actual has 2 (superset)
Negative equivalence = PASS	"Not indicated" ≈ "Excluded" ≈ "Do not perform"