Nekodeus/baichuan2-7b-chat-openvino

Baichuan2-7B-Chat OpenVINO (INT8)

OpenVINO IR conversion of baichuan-inc/Baichuan2-7B-Chat with weight-only INT8 quantization, produced on CPU-only Kaggle (fully offline, token-free).

• Base model: baichuan-inc/Baichuan2-7B-Chat (BaichuanForCausalLM, 7.0B params, Baichuan License)
• Task: text-generation
• Precision: weights INT8 (NNCF) for linear layers, embeddings/lm_head kept in original precision (15GB IR)
• Converted with: optimum-intel 2.2.0, openvino 2026.4.1, transformers 5.16.1
• Special in-process conversion path: optimum's MAX_TRANSFORMERS_VERSION gate bypassed, Baichuan rotary caches rebuilt in bf16, tokenizer/detokenizer IR generated locally with openvino-tokenizers

Usage

optimum-intel

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model = OVModelForCausalLM.from_pretrained("Nekodeus/baichuan2-7b-chat-openvino")
tok = AutoTokenizer.from_pretrained("Nekodeus/baichuan2-7b-chat-openvino", trust_remote_code=True)
inputs = tok("<reserved_106>What is OpenVINO?<reserved_107>", return_tensors="pt")
print(tok.decode(model.generate(**inputs, max_new_tokens=128)[0]))

OpenVINO GenAI

ov::genai::LLMPipeline pipe("path/to/Nekodeus/baichuan2-7b-chat-openvino", "CPU");

Reproduce

Baichuan2 requires a non-standard conversion path on transformers 5.x (see model card notes); this repo was produced with a custom in-process export.

License

Baichuan2 Community License (commercial use allowed <100M MAU).