Irfanuruchi/Llama-3.2-1B-Computer-Engineering-OpenVINO-FP16

Llama-3.2-1B Computer Engineering - OpenVINO FP16

OpenVINO FP16 representation of Irfanuruchi/Llama-3.2-1B-Computer-Engineering-LLM, a 1B-parameter model specialized for Computer Engineering.

The model was converted and tested on IULinux with OpenVINO 2026.4 across three Intel execution backends:

• Intel Core Ultra 9 275HX CPU
• Intel Graphics integrated GPU
• Intel AI Boost NPU

The resulting OpenVINO model directory is approximately 2.4 GB.

Prompt format

This model works best as a completion-style Q/A model rather than as a general-purpose chat assistant.

Recommended format:

Q: <computer engineering question>
A:

Example:

Q: Explain the purpose of CPU cache memory.
A:

Recommended generation settings:

max_new_tokens = 200
temperature = 0.7
top_p = 0.9
do_sample = True
repetition_penalty = 1.1

Short, direct technical questions generally work better than role-based conversational prompts.

Usage

Install OpenVINO GenAI:

python -m pip install "openvino==2026.4" "openvino-genai==2026.4"

Example:

import openvino_genai as ov_genai

model_path = "./Llama-3.2-1B-Computer-Engineering-OpenVINO-FP16"

device = "CPU"

pipe = ov_genai.LLMPipeline(
    model_path,
    device,
)

config = ov_genai.GenerationConfig()

config.max_new_tokens = 200
config.do_sample = True
config.temperature = 0.7
config.top_p = 0.9
config.repetition_penalty = 1.1
config.apply_chat_template = False

question = "Explain the purpose of CPU cache memory."

prompt = f"""Q: {question}
A:"""

result = pipe.generate(
    prompt,
    config,
)

print(result)

Available OpenVINO devices can be checked with:

import openvino as ov

core = ov.Core()
print(core.available_devices)

On the IULinux system used for this release:

['CPU', 'GPU', 'NPU']

Conversion

The published parent model is a genuine BitsAndBytes INT8 checkpoint.

Inspection of the parent checkpoint showed INT8 linear weights, SCB scaling tensors, and 112 Linear8bitLt modules.

The conversion path was:

meta-llama/Llama-3.2-1B
        ↓
Computer Engineering fine-tuning
        ↓
BitsAndBytes INT8 parent checkpoint
        ↓
Dequantized dense FP16 representation
        ↓
OpenVINO FP16

After dequantization, the intermediate checkpoint contained:

146 FP16 tensors
0 BitsAndBytes linear modules
0 INT8 weight tensors
0 SCB tensors

The OpenVINO export used:

optimum-cli export openvino \
  --model ./source-dequant-fp16 \
  --task text-generation-with-past \
  --weight-format fp16 \
  ./Llama-3.2-1B-Computer-Engineering-OpenVINO-FP16

This repository contains an OpenVINO FP16 representation of the reconstructed dense checkpoint.

It is not the original pre-quantization FP16 model.

The published parent checkpoint had already undergone BitsAndBytes INT8 quantization before this conversion workflow. Dequantization reconstructs floating-point tensors but cannot restore information lost during the original INT8 quantization.

Size

Observed local sizes during conversion:

Variant	Approximate size
OpenVINO FP16	2.4 GB
OpenVINO INT8	1.2 GB
OpenVINO INT4	758 MB

Benchmarks

Benchmarks were run on IULinux using the same prompt and generation workload across CPU, integrated GPU, and NPU.

System

Component	Configuration
OS	IULinux 0.1 Development
Kernel	7.0.0-31-generic
CPU	Intel Core Ultra 9 275HX
GPU	Intel Graphics (iGPU)
NPU	Intel AI Boost
OpenVINO	2026.4
OpenVINO GenAI	2026.4
Power profile	Performance
Power source	AC

OpenVINO exposed:

CPU: Intel(R) Core(TM) Ultra 9 275HX
GPU: Intel(R) Graphics (iGPU)
NPU: Intel(R) AI Boost

Method

Prompt tokens: 26
Generated tokens: 128
Warm-up runs: 2
Measured runs: 5
do_sample: false
ignore_eos: true

Benchmark prompt:

Q: Explain how CPU cache memory improves processor performance, including its relationship with RAM, latency, and memory locality.
A:

ignore_eos=True was used so every backend performed the same 128-token generation workload.

Greedy decoding was used only for deterministic performance measurement. The recommended sampling settings above should be used for normal model interaction.

Results

Device	Pipeline load	TTFT	TPOT	Throughput
Intel Core Ultra 9 275HX	282.66 ms	127.76 ± 3.99 ms	37.96 ± 0.12 ms/token	26.34 ± 0.09 tok/s
Intel Graphics iGPU	2537.27 ms	46.95 ± 0.53 ms	40.93 ± 0.02 ms/token	24.43 ± 0.01 tok/s
Intel AI Boost NPU	6676.38 ms	981.90 ± 1.69 ms	48.49 ± 0.20 ms/token	20.63 ± 0.09 tok/s

On this FP16 workload, the CPU produced the highest generation throughput.

The integrated GPU produced the lowest measured TTFT.

All three backends showed highly stable generation throughput after warm-up.

No device-level power measurements were collected, so these results compare latency and throughput only.

Precision comparison

All three OpenVINO variants were tested using the same benchmark methodology.

Precision	CPU	Intel iGPU	Intel NPU
INT4	62.98 tok/s	62.59 tok/s	33.89 tok/s
INT8	50.64 tok/s	41.86 tok/s	34.30 tok/s
FP16	26.34 tok/s	24.43 tok/s	20.63 tok/s

INT4 produced the highest measured CPU and integrated-GPU throughput.

INT8 produced the highest measured NPU throughput, although the difference between INT8 and INT4 on the NPU was small.

FP16 provides the uncompressed OpenVINO floating-point representation of the reconstructed dense checkpoint, but it required substantially more storage and produced lower throughput in this test.

IULinux

The complete conversion and validation workflow was performed on IULinux.

The same installation exposed all three Intel OpenVINO execution backends:

CPU
GPU
NPU

The Intel AI Boost NPU used the Intel NPU Linux driver stack and Level Zero runtime.

The integrated GPU used the existing i915 kernel driver together with the Intel OpenCL compute runtime.

No NVIDIA driver modification was required for this OpenVINO deployment.

Quality checks

The model behaves substantially better using its intended completion-style Q/A format:

Q: <question>
A:

The OpenVINO FP16 variant produced an average repeated 4-gram ratio of:

0.002

For comparison:

Variant	Average repeated 4-gram ratio
OpenVINO FP16	0.002
OpenVINO INT8	0.000
OpenVINO INT4	0.003

During deterministic quality-control testing, the OpenVINO FP16 representation also matched the reconstructed dense FP16 model prompt-for-prompt.

These checks are diagnostic and should not be interpreted as formal model-quality scores.

Files

The repository contains the OpenVINO model, tokenizer and detokenizer IR files, configuration files, benchmark data, environment information, and SHA-256 checksums.

Benchmark and reproducibility files are stored under:

benchmarks/

Limitations

This is a specialized Computer Engineering completion model and is not intended to behave like a general-purpose conversational assistant.

Technical answers can contain factual errors and should be independently verified for critical engineering work.

The model may continue generating additional Q/A pairs after completing an answer. Reducing max_new_tokens can help control output length.

This FP16 representation was reconstructed from a previously quantized BitsAndBytes INT8 checkpoint.

It must not be interpreted as the original pre-quantization FP16 checkpoint.

Quantization is lossy, and dequantization cannot recover precision that was discarded during the original INT8 conversion.

Benchmark results apply only to the tested hardware and software configuration.

License

This model is governed by the Llama 3.2 Community License.

Use, modification, and redistribution must comply with the Llama 3.2 Community License and its associated requirements.

Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Parent model

Irfanuruchi/Llama-3.2-1B-Computer-Engineering-LLM

Citation

@misc{uruchi2026llama32ceopenvinofp16,
  title     = {Llama-3.2-1B-Computer-Engineering-OpenVINO-FP16},
  author    = {Irfanuruchi},
  year      = {2026},
  publisher = {Hugging Face}
}