Voice-Note-Intent-Classifier

v0.1.0 (2026-10-15): placeholder release. A first, deliberately simple model that works end to end and sets a baseline. It is not tuned, and its accuracy on real notes is modest (micro F1 0.56). Expect it to be replaced; the taxonomy_version in every prediction lets a router detect that.

A small multi-label classifier that reads a voice-note transcript and predicts its intent (shopping list, to-do, calendar event, email draft, AI prompt and so on), so that an agent system can route it to the right integration.

Use case

voice note ─► ASR ─► transcript ─► [ this classifier ] ─► router
                                                            │  one dispatch per predicted label
              ┌──────────────┬──────────────┬───────────────┼──────────────┐
              ▼              ▼              ▼               ▼              ▼
         shopping-list   to-do         calendar         email          AI prompt  …
           subagent     subagent       subagent        subagent        subagent

Each subagent holds an MCP connection to its service and extracts the details it needs. The classifier only decides where a note goes.

How to use

pip install sentence-transformers numpy huggingface_hub
hf download danielrosehill/Voice-Note-Intent-Classifier --local-dir vnic
python vnic/predict.py "um need to grab oat milk eggs and uh dish soap"

{"model_version": "0.1.0", "taxonomy_version": "0.1.0",
 "scores": {"shopping_list": 0.88, "todo": 0.54, "...": "..."},
 "labels": ["shopping_list", "todo"], "abstain": false}

abstain: true means no label cleared its threshold. The router should then fall back to an LLM or ask the user.

What it is

• Encoder: frozen BAAI/bge-base-en-v1.5 sentence embeddings (normalised, 512 tokens).
• Head: one logistic regression per label, stored as plain coefficients in head.json. Per-label thresholds were calibrated on real notes.
• Files: head.json (head, thresholds, config), predict.py (inference), labels.json (taxonomy v0.1.0 with definitions), eval_results.json.

Labels (taxonomy v0.1.0)

Label	Meaning
shopping_list	Items to buy
todo	Tasks to do, with no fixed time slot
note_to_self	Information to remember; nothing to do
calendar_event	Something at a specific date/time that occupies a slot
email_draft	Text to send as an email, or an instruction to write one
idea	A single discrete idea, stated briefly
brainstorm	Extended exploratory thinking across options
content_draft	Material for publication: blog post, article, script, outline
ai_prompt	Instructions addressed to an AI model or agent

Training data

• Synthetic (1492 transcripts). LLM-generated voice notes styled as raw ASR output: single intents, compound pairs, hard cases on confusable labels, and notes that fit no label. Generated by one model family (Gemini) and kept only where two other families (Claude, GPT), labelling blind, both agreed with the requested labels. This dataset is private for now.
• Real (226 notes). The author's own voice notes, screened for privacy with personal details replaced by fake names, labelled where two LLM families agreed. The notes the author has approved for release are public at danielrosehill/Voice-Notes-Screened.

Evaluation

5-fold cross-validation on the 226 real notes: each fold's head was trained on synthetic data plus the other four folds and scored on notes it never saw.

Metric	Value
Micro precision / recall / F1	0.50 / 0.64 / 0.56
Macro F1 (labels present)	0.48
Exact match	0.39
Notes with no intent, correctly abstained	10/26

Label	F1
shopping_list	0.78
todo	0.41
note_to_self	0.23
email_draft	0.50
idea	0.36
brainstorm	0.45
content_draft	0.35
ai_prompt	0.76

calendar_event had no examples among the real notes, so it is unscored on real data.

Caveat: the real labels are agreement between two LLMs, not hand labels. The two LLMs gave identical labels on only 63% of the real notes, so these numbers are a rough guide, not a benchmark.

Fit and limitations

• Built around one person's notes. The real data comes from a single speaker whose notes are dominated by long spoken prompts to AI coding agents. Shopping lists and AI prompts work best. Other intents, and other people's dictation habits, are weaker.
• Long agent-style prompts are sometimes labelled todo, because the instructions describe tasks.
• English only. Hebrew and English-Hebrew code-switched notes are out of scope.
• Adapting it: retrain the head on your own transcripts. That takes seconds on a CPU because the encoder is frozen. Or swap in the planned fine-tuned encoder when it ships.

Roadmap

#	Phase	Status
0	Taxonomy definition	draft v0.1.0
1	Baselines on real notes	v0.1.0 (this release)
2	Synthetic data v2 (length-matched to real notes)	generated, not yet trained on
3	Fine-tuned encoder (ModernBERT-base)	planned
4	Hand-labelled real evaluation set	planned
5	ONNX export	planned
6	Integration into a voice-to-action router	planned