initial commit

FIxed language and team members.

Model at 210k steps, mlm acc 0.6537

Merge branch 'main' of https://huggingface.co/flax-community/bertin-roberta-large-spanish into main

Adding pad_to_multiple_of=16

Merge branch 'main' of https://huggingface.co/flax-community/bertin-roberta-large-spanish into main

Model at 182k steps, mlm acc 0.6494

Changed print to logger

Preparing code for final runs

Improved version of conversion script Flax → PyTorch

Fixed widget example

Fix config for checkpoint

Changed and added vocab and tokenizer

Merge branch 'main' of https://huggingface.co/flax-community/bertin-roberta-large-spanish into main

New Flax model

Fixes to mc4 fork

Fixes treatment of jsonl

Fix format for filepaths

Merge branch 'main' of https://huggingface.co/flax-community/bertin-roberta-large-spanish into main

Adding reading streaming files from local disk

Base model at 105k steps

Update .gitattributes

Fixes and defaults

Adding Numpy random number generator

Merge branch 'main' of https://huggingface.co/flax-community/bertin-roberta-large-spanish into main

Adding random sampling

Adding config and models for the hub widget

Adding missing import

Adding base config and organizing configs

Merge branch 'main' of https://huggingface.co/flax-community/bertin-roberta-large-spanish into main

Adding sampling to mc4

Merge branch 'main' of https://huggingface.co/flax-community/bertin-roberta-large-spanish into main

New tokenizer

Adjust batch size for extrating tokens

Scripts for perplexity sampling and fixes

Remove unused imports

Merge branch 'main' of https://huggingface.co/flax-community/bertin-roberta-large-spanish into main

Add script to generate dataset of embeddings and perplexities. Add script to generate t-SNE plot for embedding and perplexity visualization.

Adding correct models 10k steps

Updating run script

Adding checkpointing, wandb, and new mlm script

Epoch 1 Flax model

Changed batch size

Changed execution mode

Initial test with BETO's corpus

:sparkles: Added test_script and a folder for scripts

:see_no_evil: Added .gitignore file

Update README

README

initial commit