Agent UI lab
Collection
Lab artifacts on which UI is cheapest for agents. • 6 items • Updated
ui-mode-router
Tiny sklearn logistic-regression pipeline that maps a short user request to an agent interface label: cli, structured_api, dom_click, or form.
| Author | Akash Premkumar |
| Library | sklearn + joblib (model.joblib) |
| Train data | akashnaren/agent-ui-sft |
| Demo | akashnaren/agent-ui-router (static Space) |
| License | Apache-2.0 |
Honest scope / metrics
Values below are taken from the published metrics.json on this model repo (also linked from Hub files):
| key | value | meaning |
|---|---|---|
n_train |
80 | Rows from agent-ui-sft train split used for fit |
n_test |
20 | Held-out agent-ui-sft test split |
test_accuracy |
1.0 | Accuracy on those 20 held-out examples |
Lab toy, not production. Accuracy 1.0 on 20 synthetic held-out rows is expected to be brittle outside this tiny distribution. Do not cite this as a production or public-benchmark result.
Not Moonshot Kimi weights. The product idea shared with that line of work is “pick a cheap tool interface, then call tools.” This artifact is a free-tier sklearn classifier on 100 synthetic rows.
Files
| file | role |
|---|---|
model.joblib |
Fitted sklearn pipeline |
metrics.json |
{"test_accuracy": 1.0, "n_train": 80, "n_test": 20} |
infer.py |
CLI helper: load joblib, predict + predict_proba |
Use
import joblib
pipe = joblib.load("model.joblib")
print(pipe.predict(["list failed workflow runs"])[0])
# optional probabilities
print(dict(zip(pipe.classes_, pipe.predict_proba(["list failed workflow runs"])[0])))
From infer.py:
python infer.py list failed workflow runs
Hub download:
from huggingface_hub import hf_hub_download
import joblib
path = hf_hub_download("akashnaren/ui-mode-router", "model.joblib")
pipe = joblib.load(path)
Training data note
Labels come from ui_mode on synthetic SFT traces (agent-ui-sft), not from the human preference file agent-ui-human. The human set is a separate, smaller preference artifact.
Limitations
- Four-class toy on 100 synthetic English strings.
- Reported test accuracy is only on the 20-row SFT test split shipped with the training data.
- No calibration study, no out-of-domain eval, no GPU neural model.
- Static Space exports weights to JSON for in-browser demo; Gradio Spaces are not used on this free-tier account (per Space README).
Links
- Dataset: https://huggingface.co/datasets/akashnaren/agent-ui-sft
- Human prefs (related theme, different data): https://huggingface.co/datasets/akashnaren/agent-ui-human
- Efficiency scores (lab bakeoff table): https://huggingface.co/datasets/akashnaren/agent-ui-efficiency-scores
- Mode pairs (pairwise prefs): https://huggingface.co/datasets/akashnaren/agent-ui-mode-pairs
- Space: https://huggingface.co/spaces/akashnaren/agent-ui-router
- Question: https://github.com/akashnaren/agent-ui-metrics
- Collection: https://huggingface.co/collections/akashnaren/agent-ui-lab-6a9a8e06fec692165b0b3c07
- metrics.json: https://huggingface.co/akashnaren/ui-mode-router/raw/main/metrics.json
- Personal site: https://akashnaren.github.io/
- ORCID: https://orcid.org/0009-0001-8877-9527
- Cursor: https://cursor.com/@akashpn
- Fleet / bot page: https://akashnaren.github.io/bot/
- Downloads last month
- -