PrevonFounder/prevon-pulse-mk4

Prevon Pulse MK4

Prevon Pulse is a 270M-parameter fine-tune of FunctionGemma that picks an app action and fills in its arguments for the Prevon notes app, entirely on device. It is trained to read the tool menu it is given (renamed, masked and shuffled tools), to answer NoMatchingAction when nothing fits, and to stop on its own after one call.

FunctionGemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms

Files

file	for
prevon-pulse-mk4.litertlm	LiteRT-LM: Android, iOS, Apple Silicon Mac, Windows
prevon-pulse-mk4-Q8_0.gguf	llama.cpp: Intel Mac
safetensors/	bf16 Hugging Face weights, tokenizer and config (source of the two files above)
training/	run settings, data hash and loss curve
SHA256SUMS.txt	checksums of every file

Training

Full fine-tune of google/functiongemma-270m-it on 170,899 rows covering 70 tools, 2 epochs, learning rate 3e-5, 32 rows per step, loss on the answer tokens only, trained on Tenstorrent Blackhole chips. The released weights are the mean of three independent runs (seeds 1, 2 and 3) that share the same starting model (a weight average: no extra training). See training/ for the settings, data hash and loss curves.

Evaluation

Scored on the exported .litertlm with the real LiteRT-LM engine (CPU). The test sets are model-written, were never used in training, and the raw result files are in eval/.

test	end to end	named right	wrongly refused	refusal ok	missed-tool declines	no answer
real3, one tool shown, native prompt (comparable with MK2 and MK3)	95.1%	99.3%	0.7%	94.6%	67.4%	0
real3, the app's menu policy (one tool when sure, three when not)	95.6%	98.6%	0.7%	94.0%	55.9%	0
real3, one tool shown, older app-format prompt	95.1%	99.3%	0.7%	94.6%	68.9%	0
real4: reminders, alarms, settings, suggestions, checklists (bucket menus)	91.9%	95.0%	3.0%	66.7%	0.0%	0
real7: the six newest tools and calendar questions (bucket menus)	91.6%	99.4%	0.0%	48.5%	0.0%	0

Argument quality on real7 (the right values, not just the right tool):

mk4_soupA real7 arguments
tool                           rows  named  exact  close  extra   (percent of NAMED-right rows)
AddEvent                         30  100.0   70.0   90.0    0.0
GetNoteStats                     29  100.0   96.6   96.6    0.0
GetWritingStats                  20  100.0   75.0   75.0    0.0
ListReminders                    12   91.7   90.9   90.9    0.0
ListTrash                        30  100.0  100.0  100.0    0.0
SetFolderIcon                    30  100.0   83.3   83.3    0.0
SetNoteIcon                      30  100.0   90.0   90.0    0.0
ALL                             181   99.4   86.7   90.0    0.0

per argument (close):
  AddEvent                     duration    75.0  of 4
  AddEvent                     time        96.7  of 30
  AddEvent                     what        90.0  of 30
  AddEvent                     where       60.0  of 5
  GetNoteStats                 id          96.6  of 29
  GetWritingStats              period      58.3  of 12
  ListReminders                time        85.7  of 7
  SetFolderIcon                icon        90.0  of 30
  SetFolderIcon                id          93.3  of 30
  SetNoteIcon                  icon        93.3  of 30
  SetNoteIcon                  id          96.7  of 30

first misses:
  AddEvent               | add an all day event for my birthday on march 3 | gold {"what": "birthday", "time": "march 3 all day"} | got {"time": "on march 3", "what": "birthday on march 3"}
  AddEvent               | add anniv

The sets are written and judged by language models, so real-world accuracy is not yet measured.

Use and limits

Built for the Prevon app's own prompt format; it is not a general chat model. Use is subject to the Gemma Terms of Use and Prohibited Use Policy (https://ai.google.dev/gemma/terms, https://ai.google.dev/gemma/prohibited_use_policy).