Irfanuruchi/Qwen3-4B-Computer-Science-OpenVINO-INT8

Qwen3-4B Computer Science — OpenVINO INT8

OpenVINO INT8 weight-compressed export of Irfanuruchi/Qwen3-4B-Computer-Science.

This repository contains the OpenVINO Intermediate Representation (IR) of the fine-tuned Qwen3-4B model, intended for local inference and Intel hardware experimentation.

The model was converted and benchmarked natively on IULinux using OpenVINO 2026.4 and Optimum Intel.

Model Details

Property	Value
Architecture	Qwen3-4B
Parameters	~4B
Source checkpoint	Irfanuruchi/Qwen3-4B-Computer-Science
Source precision	BF16
Export format	OpenVINO IR
Compression	INT8 asymmetric, per-channel
Framework	OpenVINO 2026.4
Compression library	NNCF
Model size	~3.8 GB
License	Apache 2.0

This is a deployment conversion of the original fine-tuned checkpoint. No additional fine-tuning was performed during export.

Conversion

The model was exported using Optimum Intel:

optimum-cli export openvino \
  --model Irfanuruchi/Qwen3-4B-Computer-Science \
  --task text-generation-with-past \
  --weight-format int8 \
  Qwen3-4B-Computer-Science-OpenVINO-INT8

Compression Statistics

NNCF reported:

Metric	Result
Weight compression	INT8 asymmetric
Granularity	Per-channel
Eligible layers	253
Compressed eligible layers	253
Eligible-layer coverage	100%

Compression applies to eligible weights. This does not mean every operation or activation executes in INT8 precision.

Installation

pip install openvino optimum-intel transformers

Inference

from transformers import AutoTokenizer
from optimum.intel import OVModelForCausalLM

model_id = "Irfanuruchi/Qwen3-4B-Computer-Science-OpenVINO-INT8"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = OVModelForCausalLM.from_pretrained(
    model_id,
    device="CPU",
    ov_config={"PERFORMANCE_HINT": "LATENCY"}
)

messages = [
    {
        "role": "user",
        "content": "Implement binary search in Python and explain its complexity."
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=False
)

generated = outputs[0][inputs["input_ids"].shape[1]:]

print(tokenizer.decode(generated, skip_special_tokens=True))

Hardware Validation

The exported model was tested on the following system:

Component	Specification
System	ASUS ROG Strix G16
Processor	Intel Core Ultra 9 275HX
Operating system	IULinux
Python environment	Python 3.14
Inference device	Intel CPU
Runtime	OpenVINO 2026.4
Model format	OpenVINO INT8

OpenVINO detected CPU, GPU, and NPU devices.

CPU inference was successfully validated. GPU and NPU inference have not yet been tested with this exported model.

Performance

CPU Benchmark

The model was tested with the following prompt:

Write a Python function is_prime(n) using an efficient algorithm. Explain its complexity.

Configuration	Tokens	Time	Throughput
Battery, power-saver mode	160	26.08 s	6.14 tok/s
AC, performance mode	160	10.17 s	15.73 tok/s

Benchmark Method

The AC-powered run used:

• Intel Core Ultra 9 275HX
• IULinux
• OpenVINO CPU backend
• Performance power profile
• OpenVINO LATENCY performance hint
• 16-token warm-up generation
• Greedy decoding
• 160 generated tokens

The battery test did not use identical warm-up and runtime settings.

These results represent individual end-to-end generation measurements rather than statistically controlled benchmarks. Throughput includes generation overhead and should not be interpreted as steady-state decoding speed.

Functional Validation

The model generated a working Python primality-testing function using:

• math.isqrt for the integer square root
• Explicit handling of integers below 2
• Even-number elimination
• Odd-divisor iteration

The implementation has O(sqrt(n)) time complexity and O(1) auxiliary space complexity.

This is a functional smoke test, not a comprehensive accuracy evaluation.

Available Model Precisions

Additional deployment variants of the same fine-tuned checkpoint are available:

Precision	Repository
BF16 source	Qwen3-4B-Computer-Science
OpenVINO INT4	Qwen3-4B-Computer-Science-OpenVINO-INT4
OpenVINO INT8	This repository
OpenVINO FP16	Qwen3-4B-Computer-Science-OpenVINO-FP16

INT8 vs FP16 CPU Performance

Both variants were benchmarked on the Intel Core Ultra 9 275HX using the OpenVINO CPU backend.

Precision	Model size	Throughput
INT8	~3.8 GB	15.73 tok/s
FP16	~7.6 GB	7.70 tok/s

INT8 achieved approximately 2.04 times the generation throughput of FP16 in these single-run measurements.

These results do not establish overall model quality or generalized performance across different workloads.

Intended Use

• Programming assistance
• Computer science education
• Algorithm explanations
• Code generation
• Debugging assistance
• Software engineering research
• Local language-model inference
• OpenVINO deployment experiments

Limitations

• INT8 compression may change outputs relative to the original BF16 model.
• Comprehensive accuracy and regression testing have not been completed.
• Generated code may contain incorrect or insecure implementations.
• Generated code should be reviewed and tested.
• Performance depends on hardware, power settings, runtime configuration, and input length.
• Intel GPU and NPU compatibility have not been validated for this export.
• Results from the tested system may not generalize to other hardware or operating systems.

References

• Fine-tuned checkpoint: Irfanuruchi/Qwen3-4B-Computer-Science
• Qwen3: Qwen
• OpenVINO: Intel OpenVINO
• Optimum Intel: Hugging Face Optimum Intel
• NNCF: Neural Network Compression Framework

License

Apache License 2.0.

Refer to the source checkpoint for training information, original model details, and additional limitations.