File size: 756 Bytes
2435d82
17078d9
 
 
 
2435d82
 
17078d9
2435d82
9819829
2435d82
 
 
 
 
17078d9
2435d82
17078d9
2435d82
17078d9
 
a523d3c
17078d9
2435d82
17078d9
 
 
 
2435d82
 
 
 
 
 
 
17078d9
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
---
language: hu
license: apache-2.0
datasets:
- wikipedia
tags:
- generated_from_keras_callback
- hubert
model-index:
- name: hubert-tiny-wiki
  results: []
---

# hubert-tiny-wiki

This model was trained from scratch on the Wikipedia subset of Hungarian Webcorpus 2.0 with MLM and SOP tasks.

### Pre-Training Parameters:

First phase: 
- Training steps: 500.000
- Sequence length: 128
- Batch size: 1024

Second phase: 
- Training steps: 100.000
- Sequence length: 512
- Batch size: 384

### Framework versions

- Transformers 4.21.3
- TensorFlow 2.10.0
- Datasets 2.4.0
- Tokenizers 0.12.1

# Acknowledgement
[![Artificial Intelligence - National Laboratory - Hungary](https://milab.tk.hu/uploads/images/milab_logo_en.png)](https://mi.nemzetilabor.hu/)