Bielik 11B v2 Technical Report
Authors: name: Krzysztof Ociepa; name: Łukasz Flis; name: Krzysztof Wróbel; name: Adrian Gwoździej; name: Remigiusz Kinas
Published: 2025-05-05 07:03:41

Abstract: We present Bielik 11B v2, a state-of-the-art language model optimized for
Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to
11B parameters using depth up-scaling, this model demonstrates exceptional
performance across Polish language benchmarks while maintaining strong
cross-lingual capabilities. We introduce two key technical innovations:
Weighted Instruction Cross-Entropy Loss, which optimizes learning across
diverse instruction types by assigning quality-based weights to training
examples, and Adaptive Learning Rate, which dynamically adjusts based on
context length. Comprehensive evaluation across multiple benchmarks
demonstrates that Bielik 11B v2 outperforms many larger models, including those
with 2-6 times more parameters, and significantly surpasses other specialized
Polish language models on tasks ranging from linguistic understanding to
complex reasoning. The model's parameter efficiency and extensive quantization
options enable deployment across various hardware configurations, advancing
Polish language AI capabilities and establishing new benchmarks for
resource-efficient language modeling in less-represented languages.