absltnull/predBor-v1

predBor-v1 bs/sr/cr Small Language Model

The new and shiny version, with better coherence and more brain power, in the same size as the preview.

Overview

If you've seen the preview, you'll know exactly what this is. If not...

predBor is the first of its kind: a small HBS language model built from the ground up to natively understand Bosnian, Croatian and Serbian, with English support. Unlike competitors such as jerteh/gpt2-orao or gordicaleksa/YugoGPT, predBor is not based on any existing language model like Mistral or GPT-2; it is its own original base model.

Model Details

• Architecture: LLaMA,
• Context Length: 4096,
• Hidden Size: 1536,
• Num. Hidden Layers: 24,
• Num. Attention Heads: 16,
• Intermediate Size: 4096,
• Training Optimizer: Lion,
• Vocabulary Size: 65k

Training Data

predBor was trained on approximately 22 billion tokens of diverse, fully uncensored, cleaned, processed and deduplicated HBS and English data, making Bor-CORPUS-22B one of the largest collections of HBS data as of writing this.

The dataset spans multiple sources and fields, such as songs, news articles, Wikipedia articles, entire books, manually scraped blogs, forums, erotica, conversations from multiple social media platforms (DMs, group chats, comment sections, forum discussions, etc.), cleaned CommonCrawl crawl and FineWeb text, and so on.

Here are some sources from the dataset, as seen in the logs of my "cleaner and packer" script:

Category	Total Count	Files
Web/Crawl	28,290,391	cc-bosnian-huge.txt, dataset_done.txt, CC_2026.txt, fineweb-small.txt
General/Language	14,817,407	english.txt
Data Split	3,997,195	split.txt
News/Blogs	1,089,621	news_done.txt, oldnewspaper.txt, blogger_done.txt, news_out.txt, titlovi_blogs_output.txt
Reference	257,932	wikipedia_done.txt, wikiahh.txt, islamska_pitanja.txt, books_done.txt
Social	92,396	klixforums.txt, reddit.txt, insta.txt, discord.txt, viber.txt
Other	32,375	lrsum.txt, num_output.txt, sex_done.txt, mobi_output.txt, epub_output.txt
Multimedia	14,677	all_subtitles_combined.txt, lyrics.txt

Model Evaluation

I've tested both versions of predBor against GPT-2 Orao, and the results were rather satisfying in some benchmarks. Take a look for yourself:

Serbian-LLM-Eval by Gordić Aleksa

Out of the 9 Serbian-translated benchmarks, predBor has crushed GPT-2 Orao on 7 of them.

English Eval (Same Benchmarks)

As you can see, since GPT-2 Orao is based on GPT-2 (who would have guessed), it scores slightly higher than predBor in BoolQ, PiQA and WinoGrande, but gets absolutely demolished in ARC Easy, OpenBookQA, NQ Open and TriviaQA.

What's next?

More data pretraining + instruct tuning.

predBor-v2 will be a continued pretrain of this checkpoint, trained on a massive uncensored, cleaned and language-labeled dataset of HBS and English data.

Then, we get to instruct tuning. That's how we get the final version: Bor.

Pretraining Dataset Focus: Human-written, clean, broad, uncensored text data.
Instruct Dataset Focus: Synthetic instruct-following data, trained only on one persona, no refusals baked in.

Stay tuned. absltnull out.