---
language:
- cs
metrics:
- perplexity
pipeline_tag: text-generation
license: mit
datasets:
- BUT-FIT/BUT-LCC
---
# Czech GPT
This is our GPT-2 XL trained as a part of the research involved in [SemANT project](https://www.fit.vut.cz/research/project/1629/.en).

# <span style="color:red">BUT LM Model Roster</span>
- [BUT-FIT/CSTinyLlama-1.2B](https://huggingface.co/BUT-FIT/CSTinyLlama-1.2B)
- [BUT-FIT/Czech-GPT-2-XL-133k](https://huggingface.co/BUT-FIT/Czech-GPT-2-XL-133k)
- [BUT-FIT/csmpt7b](https://huggingface.co/BUT-FIT/csmpt7b)

## Factsheet
- The model is trained on our  `15,621,685,248 token/78,48 GB/10,900,000,000 word/18,800,000 paragraph` corpus of Czech obtained by Web Crawling.
- The original size of our corpus before deduplication and lm-filtering steps was `266,44 GB`.
- Our tokenizer size is 64k, and we use GPT-2 like BPE encoding for tokenization.
- The model is trained in GPT-2 style, the first token is an actual text token (not bos). Thus first token probability can't be computed.
- Due to the feature of our code, our model was never trained to generate [EOS]. 
- The model was trained by 133,000 update steps (~139B training tokens), before the end of the experiment.
- The model was adapted from the original GPT-2 XL, by: 
   - replacing the tokenizer,  
   - corresponding embeddings, and  
   - copying over 1,000 EN representations corresponding to the 1,000 most frequent tokens into new embeddings based on a bilingual dictionary.
- The training loss decreased steadily, and the model definitely didn't converge yet. We compare the loss to a small 124M model version.
  <img src="XL_vs_SMALL_train.png" width="600"/>
- The validation loss also decreased steadily. We had a bug in validation for early/late steps, so we released only validation from steps 46,000 to 100,000. Similarly, we compare the loss to the small 124M model version.
  <img src="XL_vs_SMALL_test.png" width="600"/>

## Training parameters
Not mentioned parameters are the same as for GPT-2.

| **Name**                   | **Value**     | **Note**                                                                                     |
|----------------------------|---------------|----------------------------------------------------------------------------------------------|
| dataset_type               | Concat        | Sequences at the model's input were concatenated up to `$max_seq_len`, divided by EOS token. |
| tokenizer_size             | 64k           |                                                                                              |
| max_seq_len                | 1024          |                                                                                              |
| batch_size                 | 1024          |                                                                                              |
| learning_rate              | 1.0e-4        |                                                                                              |
| optimizer                  | LionW         |                                                                                              |
| optimizer_betas            | 0.9/0.95      |                                                                                              |
| optimizer_weight_decay     | 0             |                                                                                              |
| optimizer_eps              | 1.0e-08       |                                                                                              |
| gradient_clipping_max_norm | 1.0           |                                                                                              |
| attn_impl                  | flash2        |                                                                                              |
| dropout                    | 0.1           | for residuals, attention, embeddings                                                         |
| fsdp                       | SHARD_GRAD_OP | (optimized for A100 40GB GPUs)                                                               |
| precision                  | bf16          |                                                                                              |
| scheduler                  | linear        |                                                                                              |
| scheduler_warmup           | 10,000 steps  |                                                                                              |
| scheduler_steps            | 200,000       |                                                                                              |
| scheduler_alpha            | 0.1           | So LR on last step is 0.1*(vanilla LR)                                                       |

## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

t = AutoTokenizer.from_pretrained("BUT-FIT/Czech-GPT-2-XL-133k")
m = AutoModelForCausalLM.from_pretrained("BUT-FIT/Czech-GPT-2-XL-133k").eval()

# Try the model inference
prompt = "Nejznámějším českým spisovatelem "
input_ids = t.encode(prompt, return_tensors="pt")
with torch.no_grad():
    generated_text = m.generate(input_ids=input_ids,
                                do_sample=True,
                                top_p=0.95,
                                repetition_penalty=1.0,
                                temperature=0.8,
                                max_new_tokens=64,
                                num_return_sequences=1)
    print(t.decode(generated_text[0], skip_special_tokens=True))
```

## Evaluation
We observed 10-shot result improvement over the course of training for sentiment analysis, and hellaswag-like commonsense reasoning. 
There were some tasks where there was no such improvement, such as grammar error classification (does the sentence contain grammatical error?).
We will release the precise results once we advance with the work on our Czech evaluation kit.


## Disclaimer
This is an intermediate result of our work-in-progress. This is a probabilistic model, and authors are not responsible for the model outputs. Use at your own risk. 
For further questions, turn to `martin.fajcik@vut.cz`.

## Acknowledgement
This work was supported by NAKI III program of  Ministry of Culture Czech Republic, project semANT --- 
"Sémantický průzkumník textového kulturního dědictví" grant no. `DH23P03OVV060` and
by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:`90254`).