Food Support Copilot — ticket classifiers
Two scikit-learn logistic-regression classifiers used by the Food Delivery Support Copilot Space.
Inputs
Not raw text. These models consume a 384-dimensional L2-normalized sentence
embedding of the customer message, produced by BAAI/bge-small-en-v1.5.
Encode with the BGE query prefix, exactly as the app does:
from sentence_transformers import SentenceTransformer
import joblib, numpy as np
encoder = SentenceTransformer("BAAI/bge-small-en-v1.5")
prefix = "Represent this sentence for searching relevant passages: "
vec = encoder.encode([prefix + message], normalize_embeddings=True)[0]
clf = joblib.load("clf_category.joblib")
print(clf.predict(vec.reshape(1, -1))[0])
Metrics
Trained on 10,153 synthetic tickets, 80/20 stratified split, random_state=42.
| target | accuracy | macro-F1 | majority baseline | beats baseline |
|---|---|---|---|---|
category |
0.9882 | 0.9882 | 0.1290 | yes |
urgency |
0.4471 | 0.3061 | 0.4584 | no |
Intended use and limitations
clf_category.joblib is reliable and is what the app leads with. Its accuracy is
high partly because the training data is synthetic and spec-conditioned — the
generator was told which category to write about — so expect materially lower
numbers on real support tickets.
clf_urgency.joblib does not work and is published for completeness. It scores
below the majority-class baseline, because urgency was sampled independently of
the text the generator wrote, so the message carries almost no urgency signal. Do
not use it to make decisions. See
notebook 04.
No sentiment classifier is published: ~84% of the dataset's positive labels
contradict their own text, so the app reads sentiment with an LLM instead.
Fitted with scikit-learn 1.9.0. Loading under a different version may fail.
- Downloads last month
- -