Jinamshah/medical-intent-classifier

Medical Intent Classifier v2 (MiniLM-L6, ONNX)

Classifies a short chatbot message into one of 26 intents, including an explicit out_of_scope class: allergy, back_pain, body_aches, capabilities, cold, congestion, cough, diarrhea, dizziness, ear_pain, fatigue, fever, flu, goodbye, greeting, headache, how_are_you, insomnia, nausea_vomiting, out_of_scope, shortness_of_breath, skin_rash, sore_throat, stomach_ache, thanks, toothache.

Not medical advice. A portfolio and learning project. It doesn't diagnose anything and is not safe for triage.

Fine-tuned from sentence-transformers/all-MiniLM-L6-v2 (Apache 2.0). Code, data, and the full measurement log: https://github.com/Jinam-Shah/Medical-Intent-Classifier (METRICS.md).

Results (dataset v2 test split, n=673, evaluated once after choosing the model on validation)

Metric	Value
Accuracy, 26 intents	0.947 (95% CI 0.927 to 0.961)
Macro F1 / weighted F1	0.942 / 0.947
Accuracy on CLINC150 test sentences (independent authors, n=388)	0.941
Accuracy on project-written test sentences (n=285)	0.954
End-to-end with low-confidence fallback (T=1.082, threshold 0.49)	0.944
Out-of-scope messages rejected	0.960

Model comparison on the same split:

Model	Params	Val accuracy	Test accuracy	Test macro F1
BiLSTM + attention (from scratch)	16.6 M	0.757	0.741	0.551
DistilBERT base, fine-tuned	67.0 M	0.946	0.936	0.931
MiniLM-L6, fine-tuned (this model)	22.7 M	0.956	0.947	0.942

Files

• model.onnx: fp32 ONNX graph; inputs input_ids, attention_mask, token_type_ids (int64, [batch, seq]), output logits
• tokenizer.json: the base model's WordPiece tokenizer (use with the tokenizers library)
• model_meta.json: labels, input names, max length; serving.json: calibration temperature and threshold

Data

• Original project patterns and patterns written for this project (symptom intents; same author as the training data, so those test scores may be optimistic).
• CLINC150 (Larson et al., 2019, CC BY 3.0) for greetings, thanks, goodbyes, capability questions, and out-of-scope queries, keeping CLINC's own train/validation/test assignment.

Limitations

• One intent per message ("fever and a cough" gets one label). No injury intents ("I broke my leg" is misread).
• Symptom questions ("how long does the flu last?") get the symptom's general advice.
• Emergency phrases are handled by a rule-based check in the app, not by this model.