Article
Scaling robotics datasets with video encoding
ā¢
39
Charles Moslonka
CharlesMoslonka
Ā·
AI & ML interests
Information Retrieval & RAGs
Recent Activity
upvoted
an
article
22 days ago
Scaling robotics datasets with video encoding
liked
a model
24 days ago
EuroBERT/EuroBERT-210m
liked
a model
24 days ago
EuroBERT/EuroBERT-610m
Organizations
CharlesMoslonka's activity
upvoted
an
article
22 days ago
liked
3
models
24 days ago
upvoted
a
paper
24 days ago
reacted to
tomaarsen's
post with š„
24 days ago
Post
6607
An assembly of 18 European companies, labs, and universities have banded together to launch šŖšŗ EuroBERT! It's a state-of-the-art multilingual encoder for 15 European languages, designed to be finetuned for retrieval, classification, etc.
šŖšŗ 15 Languages: English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, Hindi
3ļøā£ 3 model sizes: 210M, 610M, and 2.1B parameters - very very useful sizes in my opinion
ā”ļø Sequence length of 8192 tokens! Nice to see these higher sequence lengths for encoders becoming more common.
āļø Architecture based on Llama, but with bi-directional (non-causal) attention to turn it into an encoder. Flash Attention 2 is supported.
š„ A new Pareto frontier (stronger *and* smaller) for multilingual encoder models
š Evaluated against mDeBERTa, mGTE, XLM-RoBERTa for Retrieval, Classification, and Regression (after finetuning for each task separately): EuroBERT punches way above its weight.
š Detailed paper with all details, incl. data: FineWeb for English and CulturaX for multilingual data, The Stack v2 and Proof-Pile-2 for code.
Check out the release blogpost here: https://huggingface.co/blog/EuroBERT/release
* EuroBERT/EuroBERT-210m
* EuroBERT/EuroBERT-610m
* EuroBERT/EuroBERT-2.1B
The next step is for researchers to build upon the 3 EuroBERT base models and publish strong retrieval, zero-shot classification, etc. models for all to use. I'm very much looking forward to it!
šŖšŗ 15 Languages: English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, Hindi
3ļøā£ 3 model sizes: 210M, 610M, and 2.1B parameters - very very useful sizes in my opinion
ā”ļø Sequence length of 8192 tokens! Nice to see these higher sequence lengths for encoders becoming more common.
āļø Architecture based on Llama, but with bi-directional (non-causal) attention to turn it into an encoder. Flash Attention 2 is supported.
š„ A new Pareto frontier (stronger *and* smaller) for multilingual encoder models
š Evaluated against mDeBERTa, mGTE, XLM-RoBERTa for Retrieval, Classification, and Regression (after finetuning for each task separately): EuroBERT punches way above its weight.
š Detailed paper with all details, incl. data: FineWeb for English and CulturaX for multilingual data, The Stack v2 and Proof-Pile-2 for code.
Check out the release blogpost here: https://huggingface.co/blog/EuroBERT/release
* EuroBERT/EuroBERT-210m
* EuroBERT/EuroBERT-610m
* EuroBERT/EuroBERT-2.1B
The next step is for researchers to build upon the 3 EuroBERT base models and publish strong retrieval, zero-shot classification, etc. models for all to use. I'm very much looking forward to it!
- 1 reply
liked
a
model
about 1 month ago
Use of this dataset for Financial Fraud Detection
2
#1 opened about 1 month ago
by
aryaman27
updated
a
dataset
about 1 month ago
[bot] Conversion to Parquet
#1 opened about 1 month ago
by
parquet-converter
liked
a
dataset
about 1 month ago
[bot] Conversion to Parquet
#1 opened about 2 months ago
by
parquet-converter
published
a
dataset
about 2 months ago
liked
2
models
about 2 months ago
upvoted
an
article
about 2 months ago
Article
Finally, a Replacement for BERT: Introducing ModernBERT
ā¢
592
liked
a
model
2 months ago
liked
2
datasets
2 months ago