SearchingMan/basal-2.0-mini-LiteRT

basal-2.0-mini LiteRT

An unofficial LiteRT port of Remek/basal-2.0-mini (Remigiusz Kinas, 1.5B, fine-tuned from Bielik-1.5B-v3.0-Instruct), for running on Android phones, offline: on the CPU of any arm64 phone, or on the NPU of the Snapdragon 8 Gen 2 and 8 Gen 3.

basal answers a question about a text with one of the allowed answers and a calibrated probability for each. The Android app basal LiteRT downloads this repository's files and runs them.

Not affiliated with or endorsed by the basal or Bielik authors. The model is theirs; this repository only converts it.

On a Galaxy S23 Ultra, NPU, in the app's Polish UI: the model ready (the first load, 29.7 s, includes the NPU self-test; later loads take about 3 s), the download dialog, and the answers to three of the built-in examples (0.4 s for about 100 tokens).

Variants

variant	files	size	runs on
cpu	basal-2.0-mini-cpu_dynamic_int8.tflite	1.7 GB	any arm64 Android (LiteRT CPU, XNNPACK)
npu-SM8550	mini_*_Qualcomm_SM8550.tflite, *.f32	1.7 GB	Snapdragon 8 Gen 2 (SM8550), e.g. the Galaxy S23, S23+ and S23 Ultra
npu-SM8650	mini_*_Qualcomm_SM8650.tflite, *.f32	1.7 GB	Snapdragon 8 Gen 3 (SM8650), e.g. the Galaxy S24 Ultra

The other files are shared: tokenizer.json, CALIBRATION.json and the rest of the source model's metadata (unchanged), litert.json (what the app reads: the variants, their files and sizes, and the NPU self-test), port_check.json, port_check_npu-SM8550.json and port_check_npu-SM8650.json (the results below).

CPU: int8 weights, one file for prompts of up to 2048 tokens.

NPU: int8 weights and 16-bit activations, compiled ahead of time for each chip and split into several files that the app runs one after another, with a few small data files of its own. Prompts can be up to 2048 tokens.

Newer chips: the SM8550 files also run on the SM8650's NPU. On a Galaxy S24 Ultra they gave the same decisions as the SM8650 build (30 of 30 long prompts, identical logits), only 1.4% slower. So litert.json marks npu-SM8550 as the fallback: on a Snapdragon with no variant of its own (e.g. the 8 Elite, SM8750), the app tries the SM8550 files, marked experimental. Every NPU variant runs a self-test against reference logits on first load, and the app uses the CPU variant if it fails. Only the SM8550 and the SM8650 were tested.

How it works

The prompt (prompt format 2, see the basal engine's FORMAT2.md) ends with {"answer": ". The softmax over the logits of the option codes (A, B, …) is the decision, after the temperature from CALIBRATION.json. So there is no generation, no sampling and no KV cache between questions: one forward pass per question.

Port check

The bars are the ones the basal author publishes for ports of this model tier, applied to our item set (questions from the author's evaluation tasks; not the author's dev v2 results). The reference is the source model in bf16 (transformers).

run	items	agreement (2–10 options)	agreement (margin ≥ 0.2)	accuracy diff	ECE diff	result
bar		≥ 0.916	≥ 0.986	≥ −0.010	≤ +0.010
cpu, LiteRT on a PC	1540	0.971	0.997	−0.002	+0.002	pass
cpu, Galaxy S23 Ultra	300	0.961	0.995	+0.003	−0.006	pass
npu-SM8550, Galaxy S23 Ultra	300	0.992	1.000	−0.003	+0.004	pass
npu-SM8650, Galaxy S24 Ultra	300	0.992	1.000	−0.003	+0.008	pass

The phone runs used the app's engine (the instrumented test DeviceCheck) on 300 items each, from two item sets, so the CPU row and the NPU rows aren't the same questions; the NPU rows' set has more long prompts.

Speed

Per question, with the model loaded:

phone	variant	median	mean	median prompt
Galaxy S23 Ultra (SM8550)	cpu	1.06 s	1.64 s	185 tokens
Galaxy S23 Ultra (SM8550)	npu-SM8550	0.88 s	1.31 s	189 tokens
Galaxy S24 Ultra (SM8650)	npu-SM8650	1.16 s	1.64 s	189 tokens

The S24 Ultra was a remote test device (Samsung Remote Test Lab), so its timings may include its thermal state.

Licence

Apache-2.0, as the source model. basal-2.0 is by Remigiusz Kinas (Apache-2.0), fine-tuned from Bielik v3 (SpeakLeash, ACK Cyfronet AGH; Apache-2.0).