ariacompute/qwen3.5-2b_q8

Model Card for Qwen3.5-2B (Aria Quant Bundle, q8)

Model Details

Model Description

Qwen3.5-2B is a 2-billion-parameter, dense Transformer decoder-only language model developed by the Qwen team at Alibaba Cloud, featuring a 3:1 DeltaNet linear-attention to full-attention layer ratio for efficient long-context processing with 256K native context length. Pre-trained on diverse public corpora and aligned via SFT + DPO. This distribution is provided by Aria Compute as an aria-quant-bundle — a uniform 8-bit per-group quantized package using Hadamard pre-processing + uniform quantization with per-group codebooks (g=32). It delivers the near-lossless option in the Aria Compute lineup — at 1.9× smaller than BF16 (4 GB → ~2.1 GB) with negligible quality degradation (logprob delta +0.00685 on method reference). Optimized for CPU-only, on-device inference on mobile phones, edge devices, and single-board computers via the Aria Engine runtime. No GPU or cloud connection is required.

• Developed by: Qwen Team (Alibaba Cloud)
• Quantized and distributed by: Aria Compute
• Model type: Dense Transformer decoder-only (language, DeltaNet + full-attn hybrid)
• Language(s): English (primary), Chinese, and 20+ additional languages
• License: Apache 2.0
• Finetuned from model: Qwen/Qwen3.5-2B

Model Sources

• Original Repository: QwenLM/Qwen3.5
• Original Paper: Qwen3.5 Technical Report (pending)
• Aria Compute Dashboard: ariacompute.com/dashboard/models
• Aria Engine: ariacompute.com

Uses

Direct Use

This quantized bundle is intended for on-device, offline text-generation tasks on resource-constrained hardware, including:

• On-device chat and conversational assistants
• Real-time text completion and multi-line code generation
• Structured tool calling / function calling for mobile and IoT APIs
• Lightweight text embeddings for on-device retrieval and classification
• Short-to-medium form summarization of notifications, messages, and local content
• Local document analysis up to 256K context (chunked)

All inference runs locally on CPU. No data is sent to external servers.

Target Devices

Platform	Runtime Memory	Feasibility
High-end smartphone (8 GB)	~2.3 GB	✅ Near-lossless
Mid-range smartphone (4–6 GB)	~2.3 GB	✅
Budget phone (2–3 GB)	~2.3 GB	⚠️ Tight, functional for short contexts
Raspberry Pi 5 / SBC (4–8 GB)	~2.3 GB	✅
IoT gateway (1–2 GB)	~2.3 GB	❌ Insufficient
Wearable (1 GB)	~2.3 GB	❌ Insufficient

Memory breakdown (q8, at 4K context): ~2.0 GB quantized model weights (mmap) + ~96 MB KV cache + ~80 MB runtime overhead + ~140 MB codebook overhead ≈ ~2.3 GB.

Note: KV cache is moderate due to GQA depth (24 layers × 2 KV heads × head_dim=256), ~2.4× larger than Qwen3.5-0.8B's cache.

Out-of-Scope Use

• Long-form creative writing (>4K tokens per generation)
• Mathematical theorem proving or formal verification
• Full program/application synthesis
• Multimodal input (this model is text-only)
• Real-time audio/speech processing (use Aria speech models)
• Safety-critical decision systems without human oversight
• Deployment in production when batch inference or GPU acceleration is required (this bundle targets CPU-only, single-prompt inference)

How to Get Started with the Model

Download from Aria Compute

Authenticated dashboard users can download the bundle via: https://ariacompute.com/dashboard/models

Quantization Recipe

This bundle uses the uniform 8-bit per-group (g=32) quantization recipe — the near-lossless option in the Aria Compute lineup:

Component	Quantization Strategy	Details
Attention Q/K/V/O weights	8-bit	Per-group codebooks (g=32), Hadamard pre-processing
FFN up/gate/down weights	8-bit	Per-group codebooks (g=32), Hadamard pre-processing
RMSNorm weights	FP16	Preserved at full precision
Embedding table	FP16	Preserved at full precision (tied input/output vocab)

• Bundle size: ~2.1 GB (BF16 original: ~4 GB, ~1.9× compression)
• Generation quality: Awaiting gen_quant_eval audit. Method reference (qwen3-0.6b_q8): token overlap 0.6429, exact prefix fraction 0.3854, logprob delta +0.00685 — near-lossless, negligible drift from FP16
• Calibration-free: Hadamard pre-processing + uniform quantization, no task-specific calibration data required
• Near-lossless recipe: This is the highest-fidelity quantized Qwen3.5-2B bundle. Uniform 8-bit per-group quantization (g=32) preserves near-FP16 generation quality with logprob delta near zero. For the best quality-size trade-off, use qwen3.5-2b_q326_channel. For the smallest bundle, use qwen3.5-2b_q4

Model Architecture

Qwen3.5-2B employs a dense Transformer decoder architecture with a 3:1 DeltaNet linear-attention to full-attention layer mix:

Parameter	Qwen3-0.6B	Qwen3.5-2B
Layers	28	24
Hidden size	1,024	2,048
FFN intermediate size	2,816	6,144
Attention heads (Query)	16	8
Attention heads (KV)	8 (GQA, group 2)	2 (GQA, group 4)
Head dimension	128	256
Activation	SiLU (SwiGLU-style gating)	SiLU (SwiGLU-style gating)
Position encoding	RoPE (θ = 1,000,000)	RoPE (θ ≈ 1,000,000)
Normalization	RMSNorm (pre-norm)	RMSNorm (pre-norm)
Vocabulary size	151,936	248,320
Max context length	32,768	262,144
Input/output vocab	Tied	Tied

Design highlights (shared with Qwen3 family):

• GQA (Grouped Query Attention): 2 KV heads serving 8 query heads (group 4) — KV Cache memory is 4× smaller than full attention
• RoPE high base frequency (1M): Native support for up to 256K context length
• DeltaNet + full-attn hybrid: 3:1 linear-attention ratio enables efficient sub-quadratic long-context processing without sacrificing short-context quality
• Tied vocab: Input embedding and output projection weights are shared, saving ~509M parameters

Key architecture differences from Qwen3-0.6B:

• Mixed attention architecture (DeltaNet linear + full-attn hybrid vs pure full-attn in Qwen3) — the defining Qwen3.5 innovation for efficient long-context inference
• Double hidden width (2,048 vs 1,024) and ~2.2× FFN width (6,144 vs 2,816) — the primary source of the 1.4B parameter increase
• 256K native context vs 32K — 8× longer, enabled by DeltaNet sub-quadratic attention complexity
• Larger head_dim (256 vs 128) — deeper per-head representations
• Larger vocab (248K vs 152K) — improves multilingual tokenization efficiency
• Fewer query heads (8 vs 16) and KV heads (2 vs 8, group 4) — more aggressive GQA compensates for double head_dim in KV cache

Bias, Risks, and Limitations

Limitations

• Reasoning depth: Multi-step logical reasoning (≥3 steps) is moderately capable for a 2B-class model, but still falls short of 7B+ frontier models. Verify outputs in high-stakes scenarios.
• Mathematics: GSM8K and MATH performance is adequate for simple problems; still modest for advanced quantitative reasoning. Use larger models for mathematical precision.
• Code generation: Capable of multi-line function completions and basic algorithmic snippets; unreliable for full program synthesis or complex architectures.
• Factual knowledge: Improved world knowledge over sub-1B models due to doubled hidden width and FFN capacity, but still limited compared to 7B+ models. Always verify factual claims against authoritative sources.
• Instruction following: Handles moderate multi-constraint prompts reliably. Complex, highly constrained instructions may degrade at longer contexts.
• Quantization drift: Uniform 8-bit per-group quantization with g=32 is the near-lossless recipe — exact prefix fraction 0.3854 on method reference, with logprob delta +0.00685, negligible drift from FP16. Greedy sampling rarely forks on ambiguous continuations. Use qwen3.5-2b_q4 if volume is the priority; qwen3.5-2b_q326_channel for the best quality-size trade-off.

Bias and Risks

• Bias: As with all large language models trained on web-scale data, Qwen3.5 may reflect societal biases present in its training corpus. Evaluate outputs before deployment in sensitive domains (hiring, healthcare, law).
• Toxicity: The base model has been safety-aligned with refusal training. However, no safety filter is exhaustive. Consider an additional output classifier in high-risk environments.
• Hallucination: May generate plausible-sounding but factually incorrect information. Implement output verification for critical applications.
• Dual-use risk: Text-generation capabilities could be misused for spam, disinformation, or impersonation. Deploy responsibly and in accordance with the Apache 2.0 license terms.

Recommendations

Users (both direct and downstream) should be made aware of the above risks, biases, limitations, and constraints of the model. We recommend:

• Adding a lightweight output safety classifier for user-facing deployments
• Verifying factual claims with external knowledge bases
• Not using the model for high-stakes decisions without human review