StarCoder2-3B OpenVINO (INT8)

OpenVINO IR conversion of bigcode/starcoder2-3b with weight-only INT8 quantization, produced on CPU-only Kaggle with optimum-cli (fully offline, token-free).

Base model: bigcode/starcoder2-3b (Starcoder2ForCausalLM, 3B params)
Task: text-generation (code + text)
Precision: weights INT8 (NNCF), activations FP16/FP32 as traced
Converted with: optimum-intel 2.2.0, openvino 2026.4.1, transformers 5.16.1
16384 context, FIM support
3 IR parts: openvino_model, openvino_tokenizer, openvino_detokenizer

Usage

optimum-intel

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model = OVModelForCausalLM.from_pretrained("Nekodeus/starcoder2-3b-openvino")
tokenizer = AutoTokenizer.from_pretrained("Nekodeus/starcoder2-3b-openvino")
inputs = tokenizer("def fibonacci(n):", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=64)[0]))

OpenVINO GenAI

ov::genai::LLMPipeline pipe("path/to/starcoder2-3b-openvino", "CPU");

Reproduce

optimum-cli export openvino \
  --model bigcode/starcoder2-3b \
  --task text-generation \
  --weight-format int8 \
  starcoder2-3b-openvino

Note: built with transformers==5.16.1 pinned (Kaggle image ships 5.20.2, which removed VisionRotaryEmbedding that optimum-intel 2.2.0 imports).

Downloads last month: -