ThuGie/gemma-4-E2B-aerotools

Gemma 4 E2B — AeroSentinel drone-command model (.litertlm)

Fine-tune of Gemma 4 E2B-it that turns natural-language drone utterances into a single strict JSON command object, packaged as LiteRT-LM .litertlm containers, including an NPU AOT build for Snapdragon 8 Elite (SM8750 / Hexagon V79).

File	Size	Purpose
gemma-4-E2B-aerotools_gpu.litertlm	2.58 GB	CPU/GPU build (SP tokenizer, ExecutorMetadata)
gemma-4-E2B-aerotools_qualcomm_sm8750.litertlm	2.60 GB	NPU AOT build for SM8750 (QAIRT 2.47.0.260601 context binaries)
lora-adapter/	small	LoRA adapter (r=16)

Example (litert-lm run, Dutch prompt):

prompt: "klim 10 meter"
output: {"action":"MOVE_RELATIVE","reply":"Altitude is my friend!","target_altitude_m":10}

The model emits exactly one single-line JSON object per the AeroSentinel codec (action + mandatory reply + action params; relative altitude deltas capped at ±30 m by the app codec).

---

Full rebuild guide

Everything below was executed on Windows 11 + WSL2 Ubuntu 22.04 with a ~40 GB working set. Follow the pins exactly — nightly drift is the #1 source of breakage.

0. Toolchain pins

Package	Version
Python	3.12.x (uv-managed CPython 3.12.14)
litert-torch-nightly	0.10.0.dev20260824
ai-edge-quantizer-nightly	0.10.0.dev20260823 (do not use dev20260825+: recipe format change breaks gemma4_mixed48)
ai-edge-litert-sdk-qualcomm	2.2.0 (bundles Qualcomm compiler plugin)
litert-lm / litert-lm-builder	0.16.1
transformers	5.5.0
QAIRT SDK (host libs for NPU compile)	2.47.0.260601
Target app runtime	litertlm-android ≥ 0.16.x recommended (see compatibility matrix below)

uv venv --python 3.12.14 /root/venvs/litert
uv pip install --python /root/venvs/litert/bin/python \
  litert-torch-nightly==0.10.0.dev20260824 \
  ai-edge-quantizer-nightly==0.10.0.dev20260823 \
  ai-edge-litert-sdk-qualcomm==2.2.0 \
  litert-lm==0.16.1 litert-lm-builder==0.16.1
uv pip install --python /root/venvs/litert/bin/python \
  "transformers==5.5.0" sentencepiece huggingface_hub

Fresh distros also need: apt-get install -y libc++1 libc++abi1 libatomic1 (the Qualcomm plugin .so links against them).

WSL memory tuning (.wslconfig) — the per-layer-embedder MLIR phase OOMs at defaults: memory=28GB, swap=32GB. Keep ≥30 GB free on the Windows drive hosting the vhdx; failed exports orphan multi-GB temp trees.

1. LoRA fine-tune

Data: JSONL pairs (utterance, command_json) per the charter (data/tool_calls_train.jsonl / _eval.jsonl). Train with train_lora.py (unsloth-style LoRA r=16 on the language layers), then merge to bf16 with merge_lora.py → merged/ (a standard HF safetensors checkpoint of Gemma-4-E2B-it + merged deltas).

2. Tokenizer — use SentencePiece, not HF-JSON

This is critical for on-device compatibility. Newer litert-lm writers emit an HF_Tokenizer_Zlib section; several loaders reject it at stream time ("Streaming HF_Tokenizer_Zlib section is not supported yet"). Google's own release containers carry a legacy SP_Tokenizer section instead.

Get the original SentencePiece model (4,689,013 bytes):

# option A: extract from Google's official E2B artifact
python - <<'PY'
from litert_lm_builder import litertlm_peek
import sys
litertlm_peek.peek_litertlm_file(
    "gemma-4-E2B-it_qualcomm_sm8750.litertlm", "/tmp/ref_dump", sys.stdout)
# -> /tmp/ref_dump/Section*_SP_Tokenizer.spiece  == tokenizer.model
PY
# option B: take tokenizer.model straight from google/gemma-4-E2B-it

Place it at $WORK/tokenizer.model and pass --tokenizer_path_override=$WORK/tokenizer.model during export (or repack — §5).

3. Chat template

chat_template_litertlm.jinja implements the <start_of_turn> / Gemma conversational markers expected by the runtime prompt assembly; pass it via --jinja_chat_template_override.

4. Export CPU/GPU container

The stock exporter applies one quantization recipe to all tflite sections, which fails for Gemma-4's prefill/decode/embedder/per-layer-embedder split. scripts/export_fixed.py patches ex.quantize_model to pick the sub-recipe per section filename:

tflite contains	recipe function
per_layer	tf_lite_per_layer_embedder
embedder	tf_lite_embedder
default (prefill/decode)	tf_lite_prefill_decode

Run:

/root/venvs/litert/bin/python scripts/export_fixed.py \
  google/gemma-4-E2B-it-litert-lm "$WORK/merged" "$WORK/out_sp" \
  --externalize_embedder=True \
  --jinja_chat_template_override="$WORK/chat_template_litertlm.jinja" \
  --quantization_recipe=gemma4_mixed48 \
  --experimental_lightweight_conversion=True \
  --experimental_use_fp16=True \
  --keep_temporary_files=True \
  --tokenizer_path_override="$WORK/tokenizer.model"

Output: out_sp/model.litertlm (~2.58 GB), sections:

LlmMetadataProto → ExecutorMetadataProto → SP_Tokenizer →
TFLiteModel(prefill_decode) → TFLiteModel(embedder) → TFLiteModel(per_layer_embedder)

Smoke-test on desktop before anything else:

litert-lm run out_sp/model.litertlm --prompt="klim 10 meter"   # expect the JSON above

5. Alternative: retrofit SP tokenizer into an existing container

If you already have a working container with an HF_Tokenizer_Zlib section, repack it (all weights copied byte-for-byte, only the tokenizer swapped):

python scripts/repack_sp.py <in.litertlm> <out.litertlm> tokenizer.model

(repack_sp.py walks model.toml from the peek dump, swaps the tokenizer section via LitertLmFileBuilder.add_sentencepiece_tokenizer, rebuilds.)

6. NPU AOT compile for SM8750

Download QAIRT host libraries (x86_64 compile-time stack):

curl -L -o qairt.zip "https://softwarecenter.qualcomm.com/api/download/software/sdks/Qualcomm_AI_Runtime_Community/All/2.47.0.260601/v2.47.0.260601.zip"
python3 -m zipfile -e qairt.zip qnn-sdk/

Compile (in-place graph → QNN context binary conversion + repack):

export QNN_SDK_ROOT=/root/work/qnn-sdk/qairt/2.47.0.260601
export LD_LIBRARY_PATH="$QNN_SDK_ROOT/lib/x86_64-linux-clang:$LD_LIBRARY_PATH"

python $VENV/litert_torch/generative/export_hf/experimental/litert_lm_npu_compiler/litert_lm_npu_compiler_main.py \
  --input_litertlm out_sp/model.litertlm \
  --output_litertlm gemma-4-E2B-aerotools_qualcomm_sm8750.litertlm \
  --backend qualcomm --soc_model SM8750

Notes:

• The bundled ai_edge_litert_sdk_qualcomm/data/lib directory is empty; actual QNN libs are dlopened from LD_LIBRARY_PATH. Whatever generation you put there gets baked into the context binaries' version expectation.
• --disable_weight_sharing exists but does not change serialization completeness (verified: identical leftover-op profile).
• Compile takes ~2 min for prefill_decode on a desktop CPU.

7. Verification toolbox

# structure
python -c "from litert_lm_builder import litertlm_peek; \
  litertlm_peek.peek_litertlm_file('model.litertlm','/tmp/dump',sys.stdout)"

# which QNN generation the contexts expect (device log strings live here):
grep -aoc "260601114" <dumped prefill_decode.tflite>   # 2.47.0.260601 markers

# leftover-op analysis (what would need on-device finalize):
from ai_edge_litert.tools import flatbuffer_utils as fb
m = fb.read_model("prefill_decode.tflite"); len(m.subgraphs)
# fully-serialized release artifacts: 2 subgraphs, 1 CUSTOM 'DISPATCH_OP' each

---

On-device compatibility (learned the hard way)

Failure modes observed on SM8750 (litertlm-android 0.15.0, mixed QNN stack — Maven qnn-runtime:2.34.0 prepare lib + 2.47 backend in jniLibs):

Symptom	Root cause
Streaming HF_Tokenizer_Zlib section is not supported yet	Container written with new tokenizer format; use SP_Tokenizer (§2)
Prepare lib id mismatch: expected v2.47… detected v2.34… + per-op validate errors	Leftover tflite ops force HTP finalize at load; finalize requires prepare-lib matching the backend generation exactly
Official Google artifact loads anyway	It is 100% pre-serialized: 2 subgraphs × 1 DISPATCH_OP, zero live ops → finalize path never runs

Practical requirements for the NPU build:

• filename must contain the SoC tag (sm8750);
• libLiteRtDispatch_Qualcomm.so and libQnnHtpPrepare.so must be present in the app's nativeDir;
• keep one coherent QNN generation across backend/skel/prepare/context blobs;
• NPU decode path expects maxNumTokens = null.

---

Limitations

• Trained for the AeroSentinel JSON-text charter; not a general assistant.
• Relative altitude deltas capped at ±30 m by design (app codec limit).
• NPU context binaries are QAIRT-version-locked (2.47.x runtime expected).

Credits

• Base model: Google gemma-4-E2B-it; reference packaging: litert-community/gemma-4-E2B-it-litert-lm.
• Stack: LiteRT-LM / LiteRT-Torch / AI Edge Quantizer / QAIRT — all trademarks of their respective owners.