Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and
  Evaluation
Authors: name: Krzysztof Ociepa; name: Łukasz Flis; name: Krzysztof Wróbel; name: Adrian Gwoździej; name: Remigiusz Kinas
Published: 2024-10-24 09:16:09

Abstract: We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for
Polish language processing. Trained on curated Polish corpora, this model
addresses key challenges in language model development through innovative
techniques. These include Weighted Instruction Cross-Entropy Loss, which
balances the learning of different instruction types, and Adaptive Learning
Rate, which dynamically adjusts the learning rate based on training progress.
To evaluate performance, we created the Open PL LLM Leaderboard and Polish
MT-Bench, novel frameworks assessing various NLP tasks and conversational
abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9
percentage point increase in average score compared to Mistral-7B-v0.1 on the
RAG Reader task. It also excels in the Polish MT-Bench, particularly in
Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model
represents a substantial advancement in Polish language AI, offering a powerful
tool for diverse linguistic applications and setting new benchmarks in the
field.