PrevonFounder/prevon-pulse-mk5

Prevon Pulse MK5

Prevon Pulse is a 270M-parameter fine-tune of FunctionGemma that picks an app action and fills in its arguments for the Prevon notes app, entirely on device. It is trained to read the tool menu it is given (renamed, masked and shuffled tools), to answer NoMatchingAction when nothing fits, and to stop on its own after one call.

FunctionGemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms

Files

file	for
prevon-pulse-mk5.litertlm	LiteRT-LM: Android, iOS, Apple Silicon Mac, Windows
prevon-pulse-mk5-Q8_0.gguf	llama.cpp: Intel Mac
safetensors/	bf16 Hugging Face weights, tokenizer and config (source of the two files above)
training/	run settings, data hash and loss curve
SHA256SUMS.txt	checksums of every file

Training

Full fine-tune of google/functiongemma-270m-it on 183,302 rows covering 79 tools, 2 epochs, learning rate 3e-5, 32 rows per step, loss on the answer tokens only, trained on Tenstorrent Blackhole chips. The released weights are the mean of three independent runs (seeds 1, 2 and 3) of one training recipe that share the same starting model (a weight average: no extra training). See training/ for the settings, data hash and loss curves.

Evaluation

Scored on the exported .litertlm with the real LiteRT-LM engine (CPU). The test sets are model-written, were never used in training, and the raw result files are in eval/.

test	end to end	named right	wrongly refused	refusal ok	missed-tool declines	no answer
real3, one tool shown, native prompt (comparable with MK2 and MK3)	95.1%	99.5%	0.5%	93.0%	68.2%	2
real4: reminders, alarms, settings, suggestions, checklists (bucket menus)	91.4%	94.4%	3.0%	66.7%	0.0%	1
real7: the six newest tools and calendar questions (bucket menus)	89.3%	98.9%	0.0%	36.4%	0.0%	0
real8: the nine newest tools plus near-miss requests that must be declined (bucket menus)	90.2%	95.3%	2.4%	54.2%	0.0%	0
real8 scored on MK4 for comparison (MK4 never saw these tools)	46.9%	43.5%	19.4%	70.8%	0.0%	0
real8b: tools MK4 already knew, a regression check (bucket menus)	96.3%	96.3%	1.9%	0.0%	0.0%	0

Argument quality on real7 (the right values, not just the right tool):

mk5_soup3 real7 arguments
tool                           rows  named  exact  close  extra   (percent of NAMED-right rows)
AddEvent                         30  100.0   66.7   90.0    3.3
GetNoteStats                     29  100.0   96.6  100.0    0.0
GetWritingStats                  20  100.0   75.0   75.0    0.0
ListReminders                    12  100.0   91.7   91.7    0.0
ListTrash                        30   93.3  100.0  100.0    0.0
SetFolderIcon                    30  100.0   90.0   90.0    0.0
SetNoteIcon                      30  100.0   90.0   90.0    0.0
ALL                             181   98.9   87.2   91.6    0.6

per argument (close):
  AddEvent                     duration    75.0  of 4
  AddEvent                     time        96.7  of 30
  AddEvent                     what        90.0  of 30
  AddEvent                     where       60.0  of 5
  GetNoteStats                 id         100.0  of 29
  GetWritingStats              period      58.3  of 12
  ListReminders                time        87.5  of 8
  SetFolderIcon                icon        93.3  of 30
  SetFolderIcon                id          96.7  of 30
  SetNoteIcon                  icon        93.3  of 30
  SetNoteIcon                  id          96.7  of 30

first misses:
  AddEvent               | add an all day event for my birthday on march 3 | gold {"what": "birthday", "time": "march 3 all day"} | got {"duration": "all day", "time": "on march 3", "what": "birthday on march 3"}
  AddEvent   

Argument quality on real8:

mk5_soup3 real8 arguments
tool                           rows  named  exact  close  extra   (percent of NAMED-right rows)
ChangeEvent                      20  100.0   70.0   80.0    0.0
FileNotesToFolder                20   95.0   78.9   94.7    0.0
FindInFiles                      18  100.0   94.4  100.0    0.0
GetRecentChanges                 19  100.0   89.5   89.5    0.0
GetStatus                        17   76.5   92.3   92.3    0.0
ReadFileForAssistant             18  100.0   66.7   88.9    0.0
ReadLoginForAssistant            20  100.0   90.0   90.0    0.0
SetFolderColor                   21  100.0  100.0  100.0    0.0
ShowMe                           17   82.4   71.4   85.7    0.0
ALL                             170   95.3   84.0   91.4    0.0

per argument (close):
  ChangeEvent                  change      95.0  of 20
  ChangeEvent                  id          85.0  of 20
  FileNotesToFolder            folder     100.0  of 19
  FileNotesToFolder            which       94.7  of 19
  FindInFiles                  query      100.0  of 18
  GetRecentChanges             since       89.5  of 19
  GetStatus                    about       92.3  of 13
  ReadFileForAssistant         fileName    88.9  of 18
  ReadLoginForAssistant        title       90.0  of 20
  SetFolderColor               colorName  100.0  of 21
  SetFolderColor               id         100.0  of 21
  ShowMe                       what        85.7  of 14

first misses:
  GetStatus              | is my vault l

Known limits (read before relying on MK5)

• Declining near-miss requests is MK5's weak spot. On requests that sit next to a tool but that no tool can do, MK5 declines correctly 54.2% of the time on real8 (13 of 24; MK4: 70.8%), 66.7% on real4 and 36.4% on real7. When it does not decline, it picks a close tool. The app confirms before it acts, but this is the main thing to improve in the next model.
• real4 end to end is 91.4% (91.6% with the app's second try when the first answer has no tool call). That is under the 92.3% reached by MK3's average on the same set, even though the main real3 test (95.1%) meets the release rule.
• Seeds differ. The three runs that were averaged scored between 86.4% and 89.8% on real4 on their own; the average is steadier than any one run.
• New tools, measured fairly: on real8 (194 rows, the nine newest tools plus near-miss requests) MK5 gets 90.2% end to end and names the right tool 95.3% of the time; MK4, which never saw these tools, gets 46.9% and 43.5%. Weaker ones: GetStatus names the right tool 76.5%, ShowMe 82.4%.
• The raw result files in eval/ are the source of every number above.

The sets are written and judged by language models, so real-world accuracy is not yet measured.

Use and limits

Built for the Prevon app's own prompt format; it is not a general chat model. Use is subject to the Gemma Terms of Use and Prohibited Use Policy (https://ai.google.dev/gemma/terms, https://ai.google.dev/gemma/prohibited_use_policy).