|
--- |
|
language: hu |
|
license: apache-2.0 |
|
datasets: |
|
- wikipedia |
|
tags: |
|
- generated_from_keras_callback |
|
- hubert |
|
model-index: |
|
- name: hubert-tiny-wiki-seq128 |
|
results: [] |
|
--- |
|
|
|
# hubert-tiny-wiki |
|
|
|
This model was trained from scratch on the Wikipedia subset of Hungarian Webcorpus 2.0 with MLM and SOP tasks. |
|
|
|
### Pre-Training Parameters: |
|
|
|
First phase: |
|
- Training steps: 500.000 |
|
- Sequence length: 128 (the model is capable for 512) |
|
- Batch size: 1024 |
|
|
|
Second phase: |
|
- Training steps: 100.000 |
|
- Sequence length: 512 |
|
- Batch size: 384 |
|
|
|
### Framework versions |
|
|
|
- Transformers 4.21.3 |
|
- TensorFlow 2.10.0 |
|
- Datasets 2.4.0 |
|
- Tokenizers 0.12.1 |
|
|
|
# Acknowledgement |
|
[](https://mi.nemzetilabor.hu/) |
