Az-1: the lexicon gate of Frumscupper's Cybernetic Roster

Az-1 reads one memory request and answers with a single call, route_to(agent), naming which of Frum.lexicon's six memory agents should handle it, or sending it back to Andrew-1 when it is not a memory job. It is a LoRA tune of Liquid AI's LFM2.5-230M, trained on CPU from data generated from the agents' own job descriptions and the app's schemas. No one's chats were used.

Job Lexicon gate: sorts each memory request
Brain region Thalamus (the lexicon's gate; Andrew-1 is the whole roster's)
Agents it routes to Cyril-1 (exact entries), Vedi-1 (recall by meaning), Slovo-1 (save facts), Dobro-1 (rank several), Myslete-1 (rewrite the question), Glagol-1 (answer with citations), Andrew-1 (not a memory job)
Base LiquidAI/LFM2.5-230M @ 35d5e77449d3a662bc0f7c7b79e1eded6b77006e
Method LoRA r=16, alpha 32, merged; 1 epoch, seed 7
Server file gguf/LFM2.5-230M-az1-r1-Q8_0.gguf, 246,597,824 bytes, SHA-256 c5922d2ba979b831faa2d183ba2d4705baebef67b4b3d84cc0071769392881ff

Scores

Greedy decoding, the same sets and grader for every row. Sealed test: 275 requests under two prompts (550 answers); red-team: 32 requests under two prompts (64).

Set Untuned base This tune
Its own sealed test 17 / 550 (3%) 525 / 550 (95.5%), macro-F1 0.956
Red-team (instructions hidden in quoted text, secrets, edit/delete requests) 2 / 64 50 / 64 (78.1%)
Dev set 6 / 280 280 / 280
General tool calls it never trained on (60) 41 / 60 57 / 60

Paired McNemar test against the base: p < 0.001 on every set. Per agent on the sealed test (F1): Cyril-1 0.99, Dobro-1 0.99, Vedi-1 0.97, Glagol-1 0.97, Slovo-1 0.96, Myslete-1 0.95, Andrew-1 send-back 0.86 (its weakest: some memory requests are sent back to Andrew-1).

The server file. Through llama.cpp's server the Q8_0 GGUF answers the same agent on 95.3% of the sealed test and 78.1% of the red-team set. llama.cpp's /completion text drops the special tokens <|tool_call_start|> and <|tool_call_end|> around the call, so the strict grader (which wants them) reads 0 and the lenient grader (the agent named) reads the numbers above; the markers are present in the token stream. The red-team set is the area to improve next.

What is in this repository

Path What it is
gguf/LFM2.5-230M-az1-r1-Q8_0.gguf The server file (Q8_0)
eval/ Full results for the base, the tune and the GGUF, with every miss
training/ The data builders, every set (train, dev, sealed test, red-team, replay), the trainer, the GGUF scorer, the recipe and the loss log
LICENSE The LFM Open License v1.0

Licence

Built on LFM2.5-230M by Liquid AI, licensed under the LFM Open License v1.0 (see LICENSE). This repository is a modified version: the base weights changed by a LoRA adapter trained by Frumscupper.

Downloads last month
20
GGUF
Model size
0.2B params
Architecture
lfm2
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for OGlaisin/frumscupper-az-1

Adapter
(12)
this model