Doc Compass
Routes a plain-language health concern to a kind of doctor to book. It suggests a door to knock on; it does not diagnose and is not medical advice.
This repo holds the app code (doccompass/) and its routers, so the demo notebook can run
without access to the project's GitHub repo.
What is here
| File | What it is |
|---|---|
checkpoints/biomedbert_gold/ |
microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext fine-tuned (all weights) to route concerns to 12 labels |
checkpoints/tfidf_stage2.joblib |
TF-IDF + logistic regression router, trained from scratch; the fallback when PyTorch isn't available |
doccompass/ |
emergency rules, routers, explanation and the Gradio app |
Labels: Dermatology, Orthopedics, ENT, Gastroenterology, Neurology, Urology, Ob-Gyn, Mental health, Cardiology, Eye care, Dentistry, Start with a GP.
Training data
362 real r/AskDocs posts from stellalisy/MediQ_AskDocs (MIT on the dataset card; the posts
come from Reddit), each labelled with the kind of doctor to book. The labels were drafted by an AI
assistant from a written guideline and reviewed by the authors. No post text is in this repo.
The TF-IDF router was trained on the same posts plus Patient Comments and Specialist Types (Mendeley Data, DOI 10.17632/2twgjzpn82.2, CC BY 4.0), a public set of short comments that appear to be generated, with its symptom categories remapped to the labels above.
Results
On 50 dev posts the router scores 54.0% top-1 and 0.473 macro-F1; that is where it was chosen. On 115 test posts, scored once, it scores 60.9% top-1 (95% CI 51.3%–69.8%), 90.4% top-3 and 0.611 macro-F1, against 40.9% for always answering "Start with a GP" and 48.7% for the TF-IDF router. The test labels are the same labels, so this measures agreement with them, not with a clinician.
Limits
- English only, and trained on 362 posts, so some labels have few examples (Dentistry has 7).
- It often names a specialty where the labeller said "Start with a GP" (right on 15 of 47 such test posts).
- Short or vague messages get low confidence; the app then shows two options instead of one.
- The emergency check is a short list of written rules, plus a language-model check when one is loaded. It will miss emergencies phrased in ways it doesn't anticipate. In an emergency call 911.
- Not clinically validated.