danieldraus/Phi-3.5-mini-ONNX-INT4

Phi-3.5 Mini Instruct (WebGPU INT4 ONNX)

This folder contains a browser-focused ONNX version of microsoft/Phi-3.5-mini-instruct, prepared for WebGPU inference with Transformers.js.

What This Model Is

• Base model family: Phi-3.5 Mini Instruct (instruction-tuned LLM).
• Runtime format: ONNX.
• Precision: INT4 quantized weights for lower memory usage and faster local inference.
• Target runtime: WebGPU in modern browsers.
• Primary use case: local chat-style text generation in web applications.

Why Use This Variant

• Runs locally in the browser (no server-side inference required).
• Lower VRAM/RAM footprint versus full precision models.
• Good latency/quality trade-off for client-side prototypes and assistants.

Folder Layout (Transformers.js-Compatible)

Transformers.js expects model weights under an onnx/ subfolder and tokenizer/config files at the model root.

phi3_5_webgpu_int4/
├── onnx/
│   ├── model.onnx
│   └── model.onnx_data
├── config.json
├── generation_config.json
├── special_tokens_map.json
├── tokenizer.json
└── tokenizer_config.json

How To Use In Transformers.js

1) Install package

npm install @xenova/transformers

2) Load model from local folder

Use the folder path (or hosted URL) as the model identifier.

import { pipeline } from '@xenova/transformers';

const generator = await pipeline('text-generation', './phi3_5_webgpu_int4', {
   device: 'webgpu',
});

const prompt = 'You are a helpful assistant. Explain what WebGPU is in two sentences.';

const output = await generator(prompt, {
   max_new_tokens: 128,
   temperature: 0.7,
   top_p: 0.9,
   do_sample: true,
});

console.log(output[0].generated_text);

3) Chat-style usage pattern

For instruction/chat behavior, pass a prompt that includes role context and user intent.

const chatPrompt = [
   'System: You are concise and technical.',
   'User: Give me 3 practical tips to reduce JavaScript bundle size.',
   'Assistant:'
].join('\n');

const chatOut = await generator(chatPrompt, {
   max_new_tokens: 160,
   temperature: 0.6,
   top_p: 0.9,
});

Notes

• Requires a browser/environment with WebGPU support.
• First run can be slower due to model initialization and shader compilation.
• If WebGPU is unavailable, you can switch device settings according to your runtime support.

License

This repository is licensed under the Apache License 2.0. See the root LICENSE file.

Portions of this repository (specifically the model weights and network architecture config) are derived from Microsoft's Phi-3.5-mini-instruct model, which is licensed under the MIT License.