Llama-3.1-8B-Instruct-ONNX-WebGPU

Built with Llama. meta-llama/Llama-3.1-8B-Instruct converted to ONNX for running in a web browser with ONNX Runtime Web's WebGPU backend, as used by Quick Inference, with an EAGLE-3 draft head for speculative decoding.

Changes from the original

• Converted to an ONNX decoder in ONNX Runtime GenAI's layout (input_ids, attention_mask, past_key_values.* → logits, present.*) by Quick Inference's own exporter (from unsloth/Llama-3.1-8B-Instruct, an ungated copy of the same weights).
• Linear layers are quantised to 4 bits (round to nearest, block size 32). Embeddings, activations and the KV cache are float16.
• The output layer is 4-bit too.
• The rotary embedding is applied inside GroupQueryAttention.
• An extra output, eagle_taps: the residual stream entering three layers (2, N/2 and N-3, concatenated), for the EAGLE-3 draft head in draft/.

These are modified files: their outputs are close to, but not identical to, the original model's.

Draft head (draft/)

yuhuili/EAGLE3-LLaMA3.1-Instruct-8B (EAGLE-3) converted to ONNX: it guesses the next tokens from eagle_taps, and the model checks every guess, so the output is exactly the model's own. Its matrices are quantised to 4 bits (round to nearest, block size 32); its own KV cache is static; it returns its guess as a token id. Licensed under its own terms: draft/LICENSE.

Licence

The model is licensed under the same terms as the original model: see LICENSE, NOTICE, USE_POLICY.md.