Question:
In my model card, I used to be able to run the hosted inference successfully, but recently it prompted an error: "<mask>" must be present in your input.
My model uses RoBERTa MLM and BERT Tokenizer. So the mask token is actually “[MASK]”. I have already set it in tokenizer_confg.json but the inference API still mismatches with that.
In the past it is OK but recently it turns to prompt an error. Seems like the front-end start to double-check the mask token. How can I set the mask token in an appropriate way? Is it documented to set mask token in inference API?
Thanks!

To reproduce
Steps to reproduce the behavior:

Go to ethanyt/guwenbert-base · Hugging Face 5

Run an example with “[MASK]”

Expected behavior
In the past it was OK. See snapshot in  guwenbert/README_EN.md at main · Ethan-yt/guwenbert · GitHub

Context: Solved Mask token mismatch with the model on hosted inference API of Model Hub · Issue #11884 · huggingface/transformers · GitHub 5


Sshleifer/student_blarge_12_3 does not have a tokenizer_config.json file

Question:
Hello @sshleifer I was getting the following error
Can’t load tokenizer for ‘sshleifer/student_blarge_12_3’. Make sure that:

‘sshleifer/student_blarge_12_3’ is a correct model identifier listed on ‘Hugging Face – The AI community building the future.’

or ‘sshleifer/student_blarge_12_3’ is the correct path to a directory containing relevant tokenizer files

I figured out that tokenizer_config.json was missing in the files and versions tab.
Could you please fix this?

Context: @valhalla Could you s3cmd cp the roberta tokenizer → sshleifer/student_blarge_12_3?


How to use model card in case of multitask learning?

Question:
T5 (Text to Text Transfer Transformer) can be trained in multitask learning. In case we have such a model (suppose trained for Summarization (Summarize English[en])and translation task (translate English[en] to Deustch[de]). In that case, how should one prepare the model card that it would understand that the text given in the widget would be used for translation or summarization?
For a single task we do it something like this:
----------------------------
language: French Spanish
tags:
- translation French Spanish  model
datasets:
- dataset1 dataset2
widget:
- text: "some French text"
----------------------------

Context:
The hub will probably assigns the Text2TextGeneration pipeline for T5 models, in that case it won’t use any prefix automatically. So you should put the prefix text  for the tasks in the widget text. And add a note in the model card that to use the model for different task the user should put the prefix himself.
See for reference valhalla/t5-small-qa-qg-hl · Hugging Face 28


[Open-to-the-community] Robust Speech Recognition Challenge

Question:
Robust speech recognition in 70+ Languages

Hi all,
We are scaling multi-lingual speech recognition systems - come join us for the robust speech community event from Jan 24th to Feb 7th. With compute provided by OVHcould, we are going from 50 to 70+ languages, from 300M to 2B parameters models, and from toy evaluation datasets to real-world audio evaluation.

What it is about

The goal of the event is to provide robust speech recognition systems in as many languages as possible to the community. We hope that especially low-resource languages will profit from this event.
The main components of the speech recognition event consist of:

Meta AI’s state-of-the-art XLS-R model 165
Common Voice newest datasets 7 & 8 136
Language model boosted decoding with Wav2Vec2 116
Real-world audio for evaluation

How does it work

Participants have two weeks to build as many robust speech recognition systems in as many languages as they want. In general, speech recognition systems can consist of:

Fine-tuned speech recognition checkpoints (e.g. XLS-R)
Language model boosted decoders (e.g. pyctcdecode + n-gram)
Pre- and post-processing modules, such as noise-canceling, spelling correction, …

During the event, you will have the opportunity to work on each of these components to build speech recognition systems in your favorite language!
Each speech recognition system will automatically be evaluated on real-world audio (if available for the language). After the fine-tuning week, the best-performing systems of each language will receive  SWAG.

What do I need to do to participate

To participate, simply fill out this short google form 382. You will also need to create a Hugging Face Hub account here 37 and join our discord here 146 - when joining the event’s discord channel please make sure to click on the  emoji under the first message to access all relevant information. OVHcloud 42 kindly offered to provide a limited about of GPUs for participants if needed - if you would like to have access to a GPU, please join the discord for more information*. Here 23 are a some in-detail videos on how to get started with setting up an OVHcloud account.
This fine-tuning week should be especially interesting to native speakers of low-resource languages. Your language skills will help you select the best training data, and possibly build the best existing speech recognition system in your language.
More in-detail information will be announced in the discord channel. We are looking forward to seeing you there!

What do I get

enjoy a bit of Hugging Face vibe
learn how to build state-of-the-art speech recognition systems
free compute to build a powerful fine-tuned model under your name on the Hub
hugging face SWAG if you manage to have the best performing model in a language
100 GPU hours from OVHcloud 42 if you manage to have the best performing model in a language

Open-sourcely yours,
Anton, Omar, Nico & Patrick

Context:
Hey, is there any blog/resource available where I can learn how to build an Audio dataset for your own native language. So, that I can building speech recognition system during the event on the language that I want to work on.
Also very excited for the event


Question:
Salam!!!
Welcome! This is the introduction thread for Persian! Please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Persian NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, or anything else

Context:
Salam!
I’m AliAkbar, just a simple tech! I have been practicing NLP for over two years now and currently working on chatbots, semantic search, sentence paraphrasing, and NLP in complex network tasks like influence maximization.
We (me and @mmsamiei) have a Telegram channel about NLP, and we will be honored to have you join us.
Our Telegram channel: NLP stuff – Telegram 9
My Github: AliAkbarBadri (AliAkbar Badri) · GitHub 6


Question:
This is the introduction thread for Dutch NLP practitioners.
Welcome! Please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Dutch NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else really

Context:
To kick things of, let me introduce myself a bit
I’m Thomas Dehaene, and I work as an ML Engineer at a company called ML6.
In my day-to-day job, I work a lot with Dutch NLP related topics of all sorts (summarization, entity extraction, text-generation, general text data analysis, etc.). In general I’m super interested in all things multilingual NLP!
Some recent work I helped on (together with the team) involved:

A small NLP analysis on some political documents (link 11)
Finetuning a Dutch GPT2 model (link 10)

Also up for a chat , you can ping me on:

here, obviously
Twitter (@TDehaene)
Github (TDehaene)

Eager to connect with you all  !


Question: Hello Everyone! I am planning to work on fine-tuning XLSR-Wav2Vec2 in Kannada. Would love to collaborate with others working on it too. We can share and discuss ideas here.

Context: I will also be working on Kannada


Question: excuse me does the decoder of the language model deal with words or sentences to do the captioning . I mean, if i use BERT, will i use it for word embedding or sentence embedding … Thanks in advance

Context:
Hello,
AFAIK you can use a GPT-2 for the decoder. Here’s an example 3 with ViT + GPT-2.


Question: Hi guys! I am planning to work on fine-tuning XLSR-Wav2Vec2 in Hindi. Would love to collaborate with others working on it too. We can share and discuss issues here.

Context: hii @hadarishav i am also working on it tooo.


[Open-to-the-community] XLSR-Wav2Vec2 Fine-Tuning Week for Low-Resource Languages

Question:
Speech-To-Text in 60 languages

Hi all,
We organize a community week (Mar 22th to Mar 29th) to fine-tune the cross-lingual speech recognition model XLSR-Wav2Vec2 297 on all languages of the crowd-sourced Common Voice dataset 397.
What it is about
The goal of the event is to provide state-of-the-art XLSR-Wav2Vec2 speech recognition models in as many languages as possible to the community. We hope that especially research in speech recognition for low-resource languages can profit from it.
How does it work
Participants have one week to fine-tune as many XLSR-Wav2Vec2 models as they want on as many of the ~60 Common Voice’s languages 397 as they want. Each fine-tuned model should then be evaluated on Common Voice’s test data of the respective language. All data can be used as training data, except the official test data (which will be checked by the Hugging Face team). After the fine-tuning week, the best performing models of each language will receive  SWAG.
What do I need to do to participate
All you need is a google colab account to be able to run the fine-tuning script provided by us. If you can train the model on a local GPU - even better, but a free google colab is enough. This fine-tuning week should especially be interesting to you if you are a native speaker of a low-resource language because your language skills can help you get a better data processing pipeline than the competition.
If you want to participate all you need to do is to sign-up to the  hub here 405 and post your name and your  hub username in this thread. The hugging face team will then add you to an internal Slack channel where you receive more in-detail information!
What do I get

enjoy a bit of HuggingFace vibe by joining the fine-tuning week
a fine-tuned model under your name on the  hub
hugging face SWAG if you manage to have the best performing model in a language

Open-sourcely yours,
Patrick & Suraj

Context:
Name : Laxya Agarwal
Username :laxya007
Very excited to be part of this competition.


Question:
Hello there! This is the introduction thread for Indonesian NLP enthusiasts.
Welcome, and please feel free to introduce yourself with any of the following:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Indonesian NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, or anything else

Context:
Hi there Mas @cahya, thanks for initiating this thread!
I’m Wilson Wongso (@w11wo in Github and HF), a year-2 Computer Science undergraduate student from Jakarta, Indonesia. I’m still relatively new and am still learning NLP/Hugging Face, but am having fun thus far!
I trained some small language models with GPT-2 and RoBERTa recently as my side project during semester break and am interested to create language models for native Indonesian languages like Javanese, Sundanese, Medanese, etc.
Looking forward to connecting with the community here!


Question:
I am a MSc student in University of Siegen and have keen desire and interest in training NLP models specific to german language. It would be really great to meet people wanting to contribute models or datasets in german language. As of now I have trained and shared 3 models for german langauge such as Question answering models and NER model for legal domain in german.
Feel free to experiment and share your reviews .

      huggingface.co

Sahajtomar (Sahaj Tomar) 10

More models to come

Context:
Hi @Sahajtomar,
I am interested in German/English models that are useful for sentiment analysis and text classification in general (topic detection). I Have labeled datasets for both tasks.
Can you recommend any available model in the repository?
I am facing a blocker issue on a MultiLabel Text Classificatin task (described in a different issue item) would you happen to know how this could be implemented with the current version of Transformers (there seem to be some breaking change around version 3 which make it difficult to review a working sample  )…
Thanx Dirk


Question:
Introduction to NLP in Turkish
Turkish in Perspective
Turkish alphabet consists of 29 letters, 8 of them being vowels, written with the Latin alphabet with an addition of letters “ç”, “ö”, “ü”, “ğ”, “ı”, “ş”. Basic word order of a sentence in Turkish is subject-object-verb. It’s an agglutinative language that uses suffixes on top of stems to create new words or conjugate existing ones. There’s no grammatical gender in Turkish unlike Germanic languages, there’s a pronoun called “o” to refer to all genders and objects. The original Turkish doesn’t contain suffixes to indicate gender in jobs and articles in front of nouns. For formal referrals, second-person plural pronoun “siz” is used.
Turkish NLP with Hugging Face
Currently there are 48 models 30 that can make predictions on Turkish language, and 41 datasets 13 that include Turkish examples. One of the most used models is the bert-base-turkish-cased 13 by Munich Digitization Center, being used as a popular language model base for Turkish NLP tasks.
Loading a Dataset
Let’s dive into a dataset! Loading a dataset is easy enough with the datasets 8 library.
We will load the dataset ‘turkish_product_reviews’ and fine-tune “savasy/bert-base-turkish-sentiment-cased“ with it.
# uncomment and install datasets, if not installed

# !pip install datasets

from datasets import load_dataset

dataset = load_dataset('turkish_product_reviews', split = "train")

Let’s examine the dataset:
In [1]:dataset

Out[1]:Dataset({

features: ['sentence', 'sentiment'],

num_rows: 23516

})

In [2]:dataset[0]["sentence"]

Out[1]:’beklentimin altında bir ürün kaliteli değil’

This dataset does not contain a separate test set, so we divide the training set twice, once for test set and once for validation set.
from sklearn.model_selection import train_test_split

train_texts, test_texts, train_labels, test_labels = train_test_split(texts, labels, test_size=.2)

Using a Pre-Trained Model
Below model is Turkish BERT fine-tuned on sentiment analysis, we will fine-tune this model on the above dataset.
You can see the model here: model page 14
from transformers import AutoTokenizer, AutoModel
model_name = 'savasy/bert-base-turkish-sentiment-cased'
tokenizer = AutoTokenizer.from_pretrained('model_name')
model = AutoModel.from_pretrained('model_name')

Check out example notebook 88 for full application of above process.
To-do:

Add Turkish translation
Add comments in Turkish to notebook
Add native Tensorflow implementation

Context:
The ProductReviews dataset used here was parsing all data positively labeled due to a file extension error. This bug has been fixed and now parses negative and positive as it should.
Upgrade your library to version 1.9.0

      github.com/huggingface/datasets

      Fixed label parsing in the ProductReviews dataset 2

      huggingface:master ← yavuzKomecoglu:fixed-turkish-product-reviews

        opened

        Jun 22, 2021

          yavuzKomecoglu

          +13
          -4

    Fixed issue with parsing dataset labels.


Question:
Hello there! This is the introduction thread for Turkish NLP practitioners.
Welcome, and please introduce yourself below:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Turkish NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, or anything else

Context:
Merhaba
I’m Merve, a Google Developer Expert in ML and have been practicing NLP for over two years now, making chatbots and other NLP tools. I’m currently working on an information extraction model, I’m modeling it with domain-specific QNLI models in huggingface. Will post my super-messy notebooks once I get them tidied up!


Turkish NLP - Showcase Projects, Resources

Question:
Merhaba
Please share your notebooks  , medium blogs , github projects done with huggingface  libraries (or any other library), if you have an open-source dataset, please share here (all should be done on Turkish NLP).
I will share mine once I clean my notebooks.

Context:
Hi, we contribute to the field of Turkish natural language processing by producing resources both commercially and personally as a Kodiks company.
As an example of these, we added 6 new Turkish datasets in the last sprint.
Blog post about added datasets.
With the models we trained using these datasets, we quickly prepared demos thanks to Spaces.
Demos:

Demo: Turkish News Classification 1
– Blog Post: Turkish News Category Classification Tutorial
– Source Code: GitHub - kodiks/turkish-news-classification: Turkish News Category Classification Tutorial

Demo: Turkish Automatic Speech Recognition 1

Spaces is great


Question:
Hi everyone,
Opening this thread to discuss about or collaborate on fine-tuning XLSR-Wav2Vec2 for Kyrgyz.

Context:
Results of the first run:
{
“epoch”: 30.0,
“eval_loss”: 0.4920000433921814,
“eval_mem_cpu_alloc_delta”: 118951837,
“eval_mem_cpu_peaked_delta”: 44984344,
“eval_mem_gpu_alloc_delta”: 0,
“eval_mem_gpu_peaked_delta”: 6326542848,
“eval_runtime”: 145.0179,
“eval_samples”: 1503,
“eval_samples_per_second”: 10.364,
“eval_wer”: 0.5470570952893938,
“init_mem_cpu_alloc_delta”: 10150057,
“init_mem_cpu_peaked_delta”: 18306,
“init_mem_gpu_alloc_delta”: 1261919232,
“init_mem_gpu_peaked_delta”: 0,
“train_mem_cpu_alloc_delta”: 2815161,
“train_mem_cpu_peaked_delta”: 300144618,
“train_mem_gpu_alloc_delta”: 3787529728,
“train_mem_gpu_peaked_delta”: 14746535424,
“train_runtime”: 13357.7567,
“train_samples”: 3466,
“train_samples_per_second”: 0.245
}
I forgot to remove " character in the preprocessing stage. I’ll try to remove it and run it again.
Hyperparams were the same as in Turkish.


Question: In this thread we will collect the Arabic NLP resources. Mainly, notebooks, tutorials and articles that discuss Arabic NLP.

Context:
A collection of some of the notebooks and code samples I know for Arabic NLP

ARBML contains around 10 [notebooks](https://Languages 6 at Hugging Face) discussing different NLP tasks.

tkseem has 5 notebooks 3 for sentiment classification, translation, etc.

AraBERT has many notebooks 7 for fine-tuning on different tasks.

CAMeL Tools has many useful code samples on their docs website 3.

SOQAL has a notebook 1 for question answering.

AraVec has a notebook 3 for extracting embeddings for words.

farasapy has a notebook 1 for using FARASA for many tasks like segmentation, stemming, etc.

Mawdoo3 has some code samples 5 for running their multi-dialect Arabic bert model.

Alyahmor has many code samples for morphology generation.

pyarabic has many code samples 3 for prerpcoessing Arabic text.


Question:
This is the introduction thread for Urdu (or Roman Urdu) NLP practitioners.
Welcome! Please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Urdu NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else reall

Context:
This is pretty nice to see that we also have a Urdu introductions post. Despite the fact that it is among the top 10 most spoken languages in the world (link 2) it is severely under-represented.
Kya mazaq hai yeh


Using Wav2Vec in speech classification/regression problems

Question:
Hi Wav2Vec enthusiasts,
I created a script for using Wav2Vec 2.0 in speech classification/regression problems. I tested the model on Persian and Greek and got significant results even way better than what they proposed in their papers.
I’m going to share this script/model with you to take advantage of it in your research, and let me know your results and feedback.
Repo: GitHub - m3hrdadfi/soxan: Wav2Vec2 for speech recognition and classification 171
Notebook: https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb 107

Context:
Hello @m3hrdadfi,
Great work, I see that you created a script which can decide regression or classification is going to be used by looking the “num_labels” extracted from csv files.
I am trying to estimate some neurological scores from sound for Parkinson’s disease patients.
And I am going to try to build a six dimensional regression model. I will give to wav2vec model a wav file and give six floating point labels means an array of six elements and want to make the model predict these labels.
The question is how can I prepare the CSV files and feed the model?
And do I need to make any change to existing model? You have added a classification layer at the top of the model which has two parameters config.hidden_size and config.num_labels. I think I have to change these parameters right?
Could you give some help please?
Regards,


Question:
This is the introduction thread for Japanese NLP practitioners.
Welcome! Please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Japanese NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else really

Context:
I’m Yusuke Mori.
My Github account: forest1988 9
My interest in Japanese NLP:
Although I mainly work with English texts as my research subject, my first language that I use daily is Japanese.
I was involved in the Japanese translation project of spacy-course. I would be pleased if I could make some contribution in the “Languages at Hugging Face”.
Projects I am working on or interested in starting:
I’m now working on the documentation of BERT Japanese.

      github.com/huggingface/transformers

      Improve coverage of the documentation 10

        opened

        Dec 11, 2020

          sgugger

  Currently, some public classes are not documented anywhere because we didn't create the corresponding doc pages. Those missing pages are:
Benchmark classes
Bert...

    Documentation
    Good First Issue

Any other languages:
I am not very good, but I can communicate in English to some extent.


Question:
Greetings!
I’m starting this casual thread in the hopes of better connecting with other fellow Korean NLP enthusiasts. To get started, please feel free to introduce yourself with any of the following:

Your name, GitHub, Hugging Face, and/or Twitter handle
Some projects you are working on or interested in starting
Any potential directions you have in mind for the Hugging Face Korean community (HF-KR)
Anything else you’d like to share!

Thank you to Hugging Face organizers and coordinators for opening up this new section in the forum. Excited to see where this will take us!

Context:
Hi!
I’m working on finetuning Wav2vec2 on Korean – currently using the Kspon dataset.
@jaketae What projects are you working on?


Question:
This is the introduction thread for Arabic NLP practitioners.
Welcome! Please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Arabic NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else really

Context: Hey, I am Zaid Alyafeai, my handle is @zaidalyafeai in all platforms. I have been working on Arabic NLP for a while now tackling different problems like text generation, evaluating tokenizers for Arabic, compact language modelling and distillation. I am currently working on improving language modelling for Arabic. I and my friend Maged co-founded arbml which is library that revises the nlp pipeline for Arabic starting from preprocessing, tokenization, training and deployment ARBML · GitHub 11.  I speak kind of English and mostly I like to work on GANs in my free time.


To be in the club, to be in the model hub

Question:
I’m writing to ask what it means that some of the most advanced language models are available on Hugging Face’s model hub.  And I’m writing to ask what it means if your language does not appear anywhere in the model hub.
What does it mean that I can download Google’s BERT, OpenAI’s GPT2 or Facebook’s M2M 100 and run them on my laptop?  What does it mean that Hugging Face has made these models so easy to use that I can batch translate a whole document in just a few lines of code?
For the millions of underserved people who speak one of the top 100 languages, this is wonderful!  We can translate books, translate Wikipedia and create an endless stream of educational resources.
Meanwhile the millions of people who don’t speak one of those 100 languages won’t get any new textbooks.  For educational materials, they’ll remain dependent on a foreign language – usually the language of their colonizer – until they start translating.
But they can start translating!  A few dedicated people could translate enough sentence pairs to train a basic machine translator.  Then with back-translation, multilingual translation and other tricks, they could get it up to a respectable quality.
Will they start translating?  Or will those languages wither away because they’re not in today’s top 100?  What’s the significance of your language being included in these models?  What does it mean if your language is not included?

Context:
Hi Eryk,
Indeed there’s a “winner takes it all phenomenon” for languages, and building models that only support the most well-resources languages participates in that trend. But so does our usage of English on this forum!
On a more positive note, the 1000+ models released by Helsinki-NLP using OPUS include some pretty low-resource languages.
Another idea would be to fine-tune BERT to translating between a low-resource language and English, making use of the low-resource language similarity with other top 100 languages. For example, the Tetun language (which I work with) is low-resource, but has similarities with both Portuguese and Indonesian, which are part of BERT. Is that something that you’ve considered?


Question:
Greetings !
This is the introduction thread for Vietnamese NLP practitioners.
Welcome, and please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Vietnamese NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else really

Context:
Hi, my name is Minh. I work as AI engineering in Ha Noi, Vietnam. My interest in ML / NLP started in early 2020.
Currently, I like to experiment with Fine-tune Sentiment analysis, NER, Pos tag and Build Pretrain.
In parallel with Roberta, we are also exploring language models, currently training a Electra and having fun with various experiments with Onnx runtime, distill.
I hope we could connect and work together on interesting Vietnamese nlp projects.
My handles:

Github: BinhMinhs10  3


Question:
Hello everyone.
I’m fine-tuning XLSR-Wav2Vec2 in Dhivehi (dv) using Common Voice. There’s 18 hours of validated data in Common voice.
I followed the notebook on Turkish and got WER 0.54 (Trained for 60 epochs). Changing learning rate to 5e04 did not improve WER but learning rate 1e04 improved slightly WER to 0.52. These are trained for 30 epochs.
Now I’m looking into how I can improve this.
Would love to collaborate, discuss or any help on improving.

Context:
Update:
I have tried with different learning rates and dropout but WER did not improve much.
So far I have found learning rate 1e-4 works best for me.
Though there isn’t a huge improvement in WER, WER and validation loss does goes down with steps.
Still the best WER I have got is 0.52.
Dhivehi is a very small dataset and unfortunately there isn’t any other dataset available for Dhivehi other than Common voice.


Question:
Hey, I fine-tuned a XLSR-Wav2Vec2 in Turkish.
My training code can be found here: https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_Tune_XLSR_Wav2Vec2_on_Turkish_ASR_with_🤗_Transformers.ipynb 70

Training details:
Data: Official Common Voice 4 Train + Validation dataset amounting to 3478 data samples
Data preprocessing: I removed all special characters like .,!?
Hyperparameter tuning:

Learning rate: A LR of 1e-5 didn’t work well. 3e-4 worked well
Dropout: I disabled (setting to 0.0) / enabled (setting to 0.1) multiple combinations of attention_dropout, hidden_dropout, feat_proj_dropout

Layerdrop: A rate of 0.1 worked well for me
mask_time_prob: I tried 0.1 and 0.05 → 0.05 seemed to work better

Time:

I trained for 30 epochs and saw the WER nicely decreasing. I could have probably trained a bit longer to get better results.

Training Graph:
You can see my some training stats here: https://wandb.ai/patrickvonplaten/huggingface/reports/Project-Dashboard–Vmlldzo1[…]539b1pmkfbfrd96pxtvzuycfkdc13sp1cy3wl161g2s9whrcikebv20rte35o9le 32
Any ideas on how training could be improved?

Context: Thank you very much for sharing the notebook, gonna work for my fine tuning process using your notebook.


Question:
This is the introduction thread for Hebrew NLP practitioners.
Welcome! Please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Hebrew NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else really

As for me, I’m Vladimir Gurevich (@imvladikon on GitHub, linkedin 2) and I’m a data scientist in Israel,
and I’m interested in Hebrew NLP  ,  in making transformers efficient and in making knowledge enriched models.
Nice to e-meeting you all!
Useful links:

      github.com

NLPH/NLPH_Resources 22

  A comprehensive list of Hebrew NLP resources. Contribute to NLPH/NLPH_Resources development by creating an account on GitHub.

      github.com

avichaychriqui/HeBERT 19

  HeBERT: Pre-training BERT for modern Hebrew. Contribute to avichaychriqui/HeBERT development by creating an account on GitHub.

      huggingface.co

Hugging Face – On a mission to solve NLP, one commit at a time. 4

Context:
Yo Hello!
Eyal Gruss / @eyaler (twitter, github) / https://eyalgruss.com 1 / https://linkedin.com/in/eyalgruss 2
Passionate about algorithmic Hebrew poetry.
I am limit to 2 links so I will share some resources in the following…


Question:
Hello everybody! Creating a thread to organize the work on Russian ASR
Data:

Common voice (111 hours validated)
CSS10 Russian: Single Speaker Speech Dataset (small, 440 utterances, available at Russian Single Speaker Speech Dataset | Kaggle 2)
Open STT (GitHub - snakers4/open_stt: Open STT 3 very large, ~20k hours, multi-domain, probably too use in full)

Progress:
So far, @gorodecki has begun training on a subset of the common voice dataset. I tried briefly to run on the whole dataset on colab, but quickly ran out of memory so I’ll have to revisit that.
Another thought I had that could be a stretch goal was to use this Russian model as a base from which to further fine-tune on similar slavic languages/dialects that may be lower resource. I was thinking of Belarusian, Ukrainian, Kazakh Russian, any others?
Tagging those involved so far (tag anybody else who would like to join us as well!): @gorodecki @vladdy

Context:
Hello!
My result is 0,38 WER on part of common voice.


Question:
Hi there, I’m on my way to train a wav2vec2 Nahuatl (which is a one of the most known native language in MX), I have used a CC share alike no profit base dataset (hope that is OK?).
For the moment taking as base the example of wandb so being my first time trying to fine tune a model for ASR thanks for the base… hopefully something useful come out of this (don’t know if will get something “usable” at end).
Have also have requested it on Weights & Biases - Hugging Face xls.

Context:
Hi there this is the first model uploaded tyoc213/wav2vec2-large-xlsr-nahuatl · Hugging Face 3 with wer of 69.11 (I hope to improve it).
Now that I have something working and knowing that Nahuatl is a binding language and it has long vowels so there is o and ooooo (double time, but can change meaning of a word) also to express it some ones write o:… and in some translations it was not writed because europeans didn’t know long vowels and couldn’t detect them so transcriptions don’t have it.
Do you know if in your language you have similar things, if so, how you handle?

long vowels annotation (make them é or how tou write it??)
binding of words affect this model at all?

My first 2 test when uploaded showed me this
said “amo tlen” which  means “youare welcome”… I didn’t say the rest… I wonder what is that ?

The second one didn’t go that well "Tlazohcamati" (thanks)-> "'kilawtsokama tihss"


Portuguese ASR: Fine-Tuning XLSR-Wav2Vec2

Question: Hi guys, I started this Topic regarding fine tuning Wav2Vec2 in Portuguese language. Let’s exhange knowledge and help to implement this model at HuggingFace

Context:
I’m training on Google Cloud with 2 T4 GPUs, which are quite inexpensive and using screen to keep the code running, even if SSH disconnects for any reason. Steps to run screen properly:
$ script /dev/null
$ screen

ENTER
$ python training.py

CLOSE WINDOW
To check what part of the code is running, SSH again an run:
$ screen -r

3310.pts-6.training

$ screen -d -r 3310
To quit screen:
$ screen -X -S 3310.pts-6.training quit


Question:
¡Hola y bienvenidos!
This is the introduction thread for Spanish NLP practitioners. Please introduce yourself by telling us a little bit about:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Spanish NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, or anything else

As for me, I’m Lewis (@lewtun on GitHub /  Hub, @_lewtun 6 on Twitter) and I’m a data scientist in Switzerland where I build ML-powered applications for startups and enterprises, mostly in the domains of NLP , time series , and topological data analysis .
I spent a big chunk of my childhood living in Chile, which is why I’m interested in Spanish NLP
My most recent project with Transformers involved applying XLM-R to improve the routing accuracy of customer support tickets in the 3 major Swiss languages (German, French, and Italian). I’m personally interested in making Transformers efficient through techniques like distillation, quantization, and pruning, because deploying these large models in the real-world is no mean feat!
Looking forward to e-meeting you all!

Context:
Hola a tod@s!
Thanks @lewtun for kicking off this thread introducing yourself!
I’m Maria and my handle is @mariagrandury on GitHub 4, Hub and LinkedIn 4.
I’m from Spain - from Ponferrada, in the northwest - and currently I’m living in Berlin. I’m a Research Engineer at a startup here called neurocat and focused on AI quality and trustworthiness (i.e. explainability, robustness, performance and ethics).
I’m very interested in the intersection of all these topics with NLP and also in the application of AI to biomedicine, in particular the use of NLP techniques and architectures like Transformers to work with text-based representations of molecules (e.g. SMILES).
I’m looking forward to reading your introductions and seeing this community grow!
Hasta pronto!


Additional sound files using speech synthesizer

Question:
Hi, I created a script to generate additional sound files using google text to speach service. The quality of google wavenet sound files are really good that I think it could be used as additional sound files to the Common Voice dataset for the training. Here is the link to the script if you want to try it

      github.com

cahya-wirawan/artificial-commonvoice 18

  Common Voice Generator using Speech Synthesizer. Contribute to cahya-wirawan/artificial-commonvoice development by creating an account on GitHub.

Context: This is interesting! What languages have you tried it with? I’m particularly curious about lower-resource languages - For example I once tried Google Translate with Swahili, and the speech synthesis on that was really robotic-sounding.


Question: Hello all, I planning to work on fine-tuning XLSR-Wav2Vec2 for Tamil. Did not find a thread for Tamil hence creating one, would love to collaborate with others working on it. Open for discussions! வாங்க பழகலாம்!

Context: Any normalizers specific for Tamil? I found this Normalizer 4 helpful. Interested to see other sources!


Question:
Hey everybody,
I’m planning on fine-tuning XLSR-Wav2Vec2 in German. I’m happy about any tips / collaborations with the community

Context:
Hi Patrick!
I will work on the german dataset, too, and would very much appreciate discussions and collaborations, because - to be honest - I have never done it before.


Question:
Hello,
I’m planning to fine-tune XLSR-Wav2Vec2 for Spanish, following the notebooks and methodology shared by @patrickvonplaten. See for example the Turkish one 15.
I’m just getting started and still getting a feel for the data I downloaded from Common Voice. I’ve seen sentences referring to words in other languages, for example:
"hay pues dos pronunciaciones posibles para 日本 nihon o nippon"
"más adelante los kanjis cambiaron a 蝦夷"

This increases the vocab substantially (230 characters after basic punctuation removal), and probably adds a lot of noise. Is this a scenario that occurs in other Common Voice datasets?
I’m currently considering the exclusion of these samples, but I haven’t yet measured how many of them there are in the dataset.

Context:
Hey @pcuenq,
Very good question & It’s great that you already took a look at the dataset. I think you have the following three options here:

Remove all characters which clearly don’t belong to the Spanish languages from both the training and the test data.

Don’t remove those characters from the training and test dataset, but remove them from the vocabulary. In this scenario make sure to add the "[UNK]" token to the vocab and define it as unk_token="[UNK]" when instantiating the tokenizer. This way the model will simply learn to classify all such tokens to [UNK], but you don’t have to significantly change the training data.

Just add all such tokens to the vocab.

Obviously all methods have their advantages and disadvantages. I would tend to either option 2) or 1) with option 2) probably being my preferred option.
The reason is that removing all those characters might change the meaning of the sentence so that the not-removed part of the sentence doesn’t make that much sense anymore. Teaching the model to simply classify unknown sounds (symbol of other language) to unknown symbols (the "[UNK]" token representing the symbol of the other language) makes most sense here IMO. Also it shouldn’t really affect your final WER as the model would have in the most likely scenario not classified those symbols correctly anyways.
Option 1) is also very much possible by just fully removing those data samples. This should very much be feasible for high resource languages like Spanish.
Option 3) has the big advantage of making training slow and less stable. This option really only makes sense if each of the added characters occurs a significant amount in your training which I believe is not the case here.


Question:
Hi everyone,
I’m planning to fine-tune XLSR-Wav2Vec2 in Indonesian. We have around 9 hours of validated voice data and 17 hours of overall voice data for Indonesian according to Common Voice 3. Let’s share tips/collaborations here

Context:
I am running the fine tuning in Colab, it should take around 4 hours and 30 minutes. Let see tomorrow when it is successfully done
WER after 90 minutes is about 0.48. Don’t know what is a good WER for indonesian ASR.


Question:
Hi,
This topic is for Marathi ASR. The dataset I am planning to use is openslr.org 4
Looking forward to discussing and collaborating with more people. Please add any more datasets or any specific details you come across for fine-tuning on Marathi

Context: It has only female speakers data


Question:
Hi Persian speakers,
Let’s do something cool for Persian together. Let me start first, I fine-tuned the model on only 40% of the whole dataset and got a 0.5 WER score. The preprocessing step considers only 36 characters in Farsi, which I think is covered all.

Context:
After a few more epochs and training the whole data, I got 32.09% WER and 8.23% CER on the test set.
Give it a try! I provided the preprocessing and the training arguments in the following link.

      huggingface.co

m3hrdadfi/wav2vec2-large-xlsr-persian · Hugging Face 9


Question:
Hi!
I’m planning on fine-tuning XLSR-Wav2Vec2 in Arabic using Common Voice. I already made code changes to transformers examples that should help with Arabic (e.g., Buckwalter orthography): transformers/examples/research_projects/wav2vec2 at master · huggingface/transformers · GitHub 16
You may also start with my pre-trained model: elgeish/wav2vec2-large-xlsr-53-arabic · Hugging Face 14 – it should be easy to tweak the sample script to transliterate into Buckwalter (to match its vocab). I’m happy to help and collaborate. Good luck and happy fine-tuning!

Context:
Interested! I’d like to fine tune on the EveryAyah and Tarteel Quranic Audio dataset.
Should be interesting to see results with noisy + recited (Mujawwad) arabic.
I’m still learning how to use HuggingFace datasets, but once I wrap my head around this, I can publish an EveryAyah dataset.


Weights & Biases - Supporting Wave2Vec2 Finetuning!

Question:
Hey folks!
First off huge thanks to @patrickvonplaten and @valhalla for putting this challenge together, really excited to do some Irish language training!
I’m also really happy to say that the team here at Weights & Biases would like to support the community with their training as much as we can!
(If you haven’t used Weights & Biases before you can find brand new docs for our Hugging Face integration here 6 or check out the XLSR colab linked below)
Quick Summary

Weights & Biases will create public language-specific W&B Projects so that multiple people can collaborate effectively on the same language, add your language here 11

We have a beta dataset visualisation 5 feature that I think is suuper useful to explore speech datasets
We have a W&B XLSR Colab 21 to show off how best to instrument your code to log your models training progress as well as upload and version your datasets, tokenizers, processors and models (before logging your best model to the HF Model Hub  ).

Language-specific W&B Projects - just ask

In order to help organise multiple people working on the same language, we are happy to create public language-specific W&B Projects that anyone will be able to log their results, datasets, models, tokenizers etc to. This way folks working on the same language can work as a team and can easily share results and see the configs and hyperparameters that were used for specific model runs.
Go here to add the language-specific project you’d like us to request 11
W&B Dataset Visualisation [Beta]
I’m suuuper excited about using this feature to quickly explore speech datasets
With this new W&B feature (still in Beta) you can easily explore rich media tables to better understand your speech dataset. I’ve made a quick video demo 5 which I think will best explain the value of this feature for EDA of rich media such as audio and video.
To see the code that created this rich media table in W&B Artifacts, see the W&B XLSR Colab 21
This is still in beta and the team would love to hear any feedback you have on it, please feel free to ping me about it and I can pass it on to the team  Docs are here 1 for more info.
W&B XLSR Colab
We have also created a W&B XLSR Colab 21 with setup and training from top to bottom to show how you can get the most out of Weights & Biases. Get wandb setup to log your models training as well as version your datasets, tokenizers, processors and models!
To make finding relevant wandb code a bit easier, the relevant headings in the notebook start with “WANDB: …” Just search “wandb”  to jump through and find the wandb code you’re looking for!
Let us know how it goes!
Let use know how integrating and using W&B goes and whether you have any issues! I’ll be active here and in the Hugging Face XLSR slack channel to answer any Weights & Biases questions you might have! @boris will also be able to help out too!
Best of luck with the challenge everyone!!

Context:
FYI I just fixed a permissions issue with the Weights & Biases projects created so far  if anyone had trouble logging to a project already created please try again, it should be working now
Check out the post above for more info on W&B Projects and feel free to drop a post here or DM me directly with any questions you have


Question:
Bonjour à tous!
This is the introduction thread for French NLP enthusiasts. Let’s share on experiences, models, datasets, research, … or any topic related to NLP and la langue de Molière!
To get started, please introduce yourself with any of the following:

Your name, Github, Hugging Face, and/or Twitter handle
Some projects you are working on or interested in starting
Any potential directions you have in mind for the Hugging Face Italian community
Anything else you’d like to share!

E.g. I’m Thomas Vrancken (ThomasVrancken on Github, @VranckenThomas on Twitter - to make it original ). I’m an MSc in AI graduate (Maastricht University), and I have experience in NLP research (Philips R&D), data science consulting (YGroup) and now I’m in the coolest ML consulting firm in Belgium - ML6!
I’m absolutely passionated about ML and NLP and pretty active in our NLP chapter. We do a lot of nice stuff, feel free to check our bi-weekly tips-of-the-week 8 (we’ve been doing it internally for a while and recently started going public with some, more to come soon!) and trained non-English transformers models 5 (currently in Dutch/German but FR on the backlog!).
Looking forward to meet more FR-NLP enthusiasts!

Context:
Haha with all of us French people working at Hugging Face you’d think we would have started this thread earlier! “Les cordonniers sont les plus mal chaussés”
Welcome and thanks for starting this group!


Question:
Hello everybody, I’m planning on fine-tuning XLSR-Wav2Vec2 in Bemba. I’m happy to collaborate with anybody willing to join me. So I have created this thread so that we can share and discuss issues here.
Summary dataset details:

Language: Bemba (or Icibemba) language of Zambia

Dataset: BembaSpeech (If interested, you can check out the paper for more details).

Duration: The dataset has the total duration of 24hrs of read speech already preprocessed and partitioned into train, dev and test sets.

Size: 2.8Gb

Subset [optional]: There is also a 17hrs subset of the BembaSpeech here consisting of audio files less than 10 seconds.

Progress:

So far, I just quickly tried to fine-tune on the 17hrs subset using the parameters that came with @patrickvonplaten `s notebook 4  but ran into vanishing/exploding problem. So yeah, need to twerk a few parameters. So get in touch if you are willing to join in… I`m happy to collaborate with anyone in the community.

Cheers!

Context:
Update on my trainings progress:

So I have been training using the 17hrs subset [optional] of the BembaSpeech: train, dev and test.

To test the waters, first I trained using the dev and test sets only. The trainign went on without a problem. However, when I decided to include the training and evaluate on the dev set… I started getting the nan results.


Question:
Ciao a tutti!
Following the example of other language communities, this is the introduction thread for Italian NLP practitioners using HuggingFace libraries! To get started, please feel free to introduce yourself with any of the following:

Your name, Github, Hugging Face, and/or Twitter handle
Some projects you are working on or interested in starting
Any potential directions you have in mind for the Hugging Face Italian community
Anything else you’d like to share!

As for me, I am Gabriele (@gsarti on Github, gsarti_ on Twitter), I recently graduated from a Data Science MSc program at University of Trieste/SISSA and I’m currently working for an AI startup + looking for NLP PhD opportunities!
As for my recent projects using HF libraries, I recently used the allenai/unifiedqa-t5-large model alongside the new prefix-constrained generation capabilities introduced by @nicola-decao for multiple choice QA!
Looking forward to e-meet you all!

Context:
Ciao Gabriele,
good idea to open the thread, we need to strengthen the nlp community in Italy
I’m Enrico Bertino, mathematical engineer passionate about language (@EnricoBerto91 on twitter).
I co-founded Indigo.ai, a startup in Ai with a focus on R&D in the NLP field.
Speaking about HF, I use transformers in practically all the task I handle. In Indigo, we love “light” models and we pre-trained distilBERT in Italian 16 (it was the only chance in my life to call a model like me ) and this year we aim to train some even more compressed models in order to allow the use of transformers in Italian also with low computational demand.
In parallel with BERT, we are also exploring the world of “next word predictor” language models, currently training a GPT-2 and having fun with various experiments with GPT-3! With this in mind, it would be interesting to organize a discussion on GPT-3 and its use in Italian, I think there are ideas from both a technical and social point of view!
In ogni caso sono super carico per fare una comunità HF italiana!!


Question:
Oh, benvinguts, passeu passeu
This is the introduction thread for Catalan NLP enthusiasts. Albeit a “small” language, it has a strong internet presence, such as Viquipèdia and even some HF models 3! So let’s keep the wheel turning.
To get started, please introduce yourself with any of the following:

Your name, Github, Hugging Face, and/or Twitter handle
Some projects you are working on or interested in starting
Any potential directions you have in mind for the Hugging Face Italian community
Anything else you’d like to share!

My name is Pere-Lluís (@PereLluis13 | Github | Twitter 1), I come from Reus, hometown of Gaudí, the guy who built all the cool stuff in Barcelona but nothing in my hometown, however we still have the best vermouth.
I am currently doing a PhD at Babelscape and la Sapienza, in Rome. My work leverages NLP with Knowledge Graph related tasks such as Relation and Entity extraction, with a particular focus on Multilinguality. I also got a couple of publications during my MSc that used Huggingface, you can check them in Scholar.
I think Transformers are a given in any NLP research nowadays, and that includes Huggingface, so I am constantly using the library in my research. I am particularly interested in autoregressive models as well as interpretability.
Una abraçada

Context:
Hi everybody,
My name is Albert Villanova del Moral (@albertvillanova , GitHub 1, Twitter 3), and I am from Alcoi (València, Spain), currently living in Paris (France).
I am a Data Scientist, and I collaborate with Hugging Face. My background is in Theoretical Particle Physics (PhD) and Computer Science (BSc).
In a broad sense, my main interests in NLP revolve around the development of Open Source language technologies (datasets and models) created collaboratively by and for the community, with a special emphasis on diversity, bias and underrepresented languages.
I hope that we manage to create and develop an interesting active community around the Catalan language, where we will be able to coordinate and collaborate so that our language does not miss this important train of the technology age.
Abraçades!


Question:
This is the introduction thread for Hindi (or Hinglish) NLP practitioners.
Welcome! Please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Hindi NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else really

Context:
Laxya Agarwal , laksh9950 , laxya007 , @Laxya_Agarwal .
Hindi NLP is the main part of my job. We train  all kinds of sentiment analysis , text generation models.
I speak Engilsh , Hindi , Russian .


Question:
สวัสดีครับ!
Following the example of other language communities, this is the introduction thread for Thai NLP practitioners using HuggingFace libraries! To get started, please feel free to introduce yourself with any of the following:

Your name, Github, Hugging Face, and/or Twitter handle
Some projects you are working on or interested in starting
Any potential directions you have in mind for the Hugging Face Thai community
Anything else you’d like to share!

For me, I am Sakares ( @sakares on Github), I work for NLP in the broad spectrum from reading and re-implement some part of papers, making a pull-request back to Huggingface/other NLP open sources, try out Kaggle competitions in spare time for some NLP challenge.
This is my following PR back to HF

      github.com/huggingface/transformers

      Add missing token classification for XLM 1

      huggingface:master ← sakares:add-XLMForTokenClassification

        opened

        Mar 15, 2020

          sakares

          +169
          -22

Currently, I am working in the Lab team from Digital Office, SCG from Thailand. Most of my works focus on POCing and delivering NLP solutions to various existing gap in business. My current project focuses on the Speech-to-Text model/app development and also the Question-Answering domain.
Yeah, nice to e-meet you all

Context:
Hi, K. @sakares ! This is Jung from ThaiKeras and Kaggle 1 krub
I have not been in particular working on Thai NLP but NLP in general with Tensorflow.
Will be very happy if I can help anything krub.
I have contributed two Tensorflow models on Huggingface which are
TFDPR

      github.com/huggingface/transformers

      Add TFDPR 2

      huggingface:master ← ratthachat:master

        opened

        Oct 31, 2020

          ratthachat

          +1133
          -0

and
TFRAG

      github.com/huggingface/transformers

      Add TFRag 1

      huggingface:master ← ratthachat:tfrag-draft-new

        opened

        Dec 9, 2020

          ratthachat

          +3039
          -19

The two combined consume me 500+ hours already plus I just finished one tough Kaggle competition, so I am currently on a month break krub


Question:
Greetings
Would like to contribute to Amharic (less-resource language) I noticed resources for this language is next to nil.
Where do I start and build up?
1st datasets?
2nd tokenizers ?
3rd transformers?

      github.com/huggingface/tokenizers

      ByteLevelBPETokenizer output seems weird 7

        opened

        Mar 24, 2020

          seyyaw

  I use the ByteLevelBPETokenizer to train a custom tokenizer for Amharic language (less-resource language).
tokenizer = ByteLevelBPETokenizer(lowercase=False)
tokenizer.train(files=paths, vocab_size=32000, min_frequency=3, special_tokens=[
 "<s>",
 "<pad>",
...

Thanks!

Context:
Hello @yosiasz , welcome!
Amharic is definitely an under-served language and one for which we’re working on improving coverage, but we do have some resources to get you started.
First, you can look at the list of datasets that have some Amharic text here 11
For example, you can get about 28M words of Amharic text from the CommonCrawl in the OSCAR dataset 6 with:
import datasets

oscar_am = datasets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])

We don’t have any models trained exclusively on Amharic so far, but we do have a few translation models as well as a Language-Agnostic Sentence Encoder 1 you can check out:
import torch
from transformers import BertModel, BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained("setu4993/LaBSE")
model = BertModel.from_pretrained("setu4993/LaBSE")
model = model.eval()

am_sentences = oscar_am["train"][:4]["text"]
am_inputs = tokenizer(am_sentences , truncation=True, return_tensors="pt", padding=True)

with torch.no_grad():
    am_outputs = model(**am_inputs )

This will give you vector representations of all of the texts in am_sentences
Hope that helps get you started, would you like to tell us more about what you want to do in Amharic so we can help further?


Question:
This is the introduction thread for Bengali NLP practitioners.
Welcome! Please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Bengali NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else really

Context:
I am Tanmoy Sarkar
[Github link] 6  [HuggingFace ID] 5 [Twitter] 7
I am final year undergrad computer science and engineering student. My area of research is Deep learning, accelerated computing. Currently learning NLP, Quantum computing.
Why BengaliNLP
NLP in Bengali comes with different set of challenges. Bengali has large number of letters, vowels. Unlike English or other majority languages it can form conjugate letters combining two or more letters. In spite of being member of Indo-European linguistics Bengali has different set of characteristics, way of expressions, structure of sentences. And the most important thing may be if we can solve NLP in Bengali we can even solve other languages like Hindi, Sanskrit etc. as these are all similar and solving this would be a huge achievement as there are approximately 1 billion native speaker in Bengali and Hindi combined in India, Bangladesh, Pakistan etc.
Projects I am currently working on
In spite of being a rich language there are very few good open source dataset available on Bengali. So I have decided to create my own dataset. We have formed a small community where there are some developers working on gathering quality datasets from classic novels, articles etc. along with the dataset annotation. It might take some time to finish that task but we are pretty much excited thinking about the possibility of NLP in Bengali Language and this dataset. Also I am planning some other community projects which I will be talking about later.
A little more about me
I am from India and I speak in Bengali(Native), English and Hindi. I am still new to NLP and I am very much focused towards it. Other than that I like doing literally everything like from blockchain to deep learning, from accelerated computing to Quantum ML  also I love music


MRLs (Morphologically Rich Languages) NLP

Question: This space is for the discussion of morphologically rich languages in a cross-lingual fashion. MRLs are languages that have rich morphology causing recent advances in NLP to lag behind in terms of learning useful word or subword representations.

Context:
Hey everyone, we could start by mentioning some examples in each language and why it is considered an MRL language. I will start by mentioning some examples in Arabic .
Basically, the complexity of Arabic comes from the problem of non-concatenative morphology which happens when morphemes are not added in a linear fashion. I will mention to examples of concatenative and non-concatenative morphology

Concatenative morphology: Let us consider the verb يعلم (to know) in which we added the morpheme ي to the stem علم to represent the present tense . We can also construct plurals like يعلمون (they know) which adds prefixes: ي and suffixes ون to the same stem.

Non-concatenative morphology: happens mostly when the stem is modified/added to a template in a non-linear fashion. For example, broken or irregular plurals happens a lot in Arabic which adds the stem of the word into a template to create the plural. For instance, the word كاتب (writer) whereas the plural form is كتاب (writers).


Question:
Hello NLP fellows,
I hope you and your family is doing good.
Recently I came across a project and thought that it’s highly likely someone would be working on something similar and would be excited to solve the problem.
Let’s connect with each other:

Share your GitHub, Twitter and LinkedIn profile.

Which NLP project you are currently working on.

Anything that you would like to share with the community.

As of me I am an NLP Engineer @Gnani.ai. Also significant time is spend on learning and trying NLP in multi lingual setting.
LinkedIn : https://www.linkedin.com/in/beingprofess/ 8
Github: theainerd (SHYAM SUNDER KUMAR) · GitHub 12
Twitter: https://twitter.com/shyam_sunder_kr?s=08 1
Have a nice day.

Context: Welcome! This is great, feel free to also start per-language intro threads for language-specific projects


Question:
Hello everyone,
This is the introduction thread for Igbo NLP enthusiasts.
Welcome, and please feel free to introduce yourself with any of the following:

Your name, Github, Hugging Face, and/or Twitter handle.
Your interest in Igbo NLP or Igbo language in general.
Some projects you are working on or interested in starting.
Any other languages that you speak, any personal interests, or anything else.

Context:
My name is Chris Emezue
I am generally passionate about NLP for African languages.
Lately I have co-worked on building a machine translation model for Fongbe, ASR for Igbo, NER for Igbo, and now looking into speech translation and modelling for African languages, starting with Igbo.
Igbo is my mother-tongue, although I hardly spoke it as a child unfortunately. Now however, I am working towards changing that and making it used widely. For example, I designed the Edemede keyboard 7, which contains NLP features like auto-complete. More on the way!
Please feel free to connect with me on or Twitter 4 or here on HuggingFace.


Question:
This is the introduction thread for Amharic NLP practitioners.
Welcome! Please introduce yourself and let us know:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Amharic NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else really

Context:
Name: Josiah Solomon, ዮስያስ ሰለሞን

      huggingface.co

yosiasz (Tito) 3

Submitted Amharic support in spaCy: Add Amharic አማርኛ Language support by yosiasz · Pull Request #6583 · explosion/spaCy · GitHub 2


Question:
Greetings !
This is the introduction thread for Yorùbá NLP practitioners.
Welcome, and please introduce yourself and let us know:
Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Yorùbá NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, anything else really

Context:
I’m David Adelani, dadelani on Github, @Davlan on HuggingFace, and @davlanade
on Twitter.
Very interested in the development of NLP resources for African language. I was involved the development of FastText embeddings for Yorùbá, text classification and named entity recognition datasets  Hugging Face – On a mission to solve NLP, one commit at a time.
Working on NER datasets  and developing BERT LMs for African languages
I speak Yorùbá, English and a little German


Question:
This is the introduction thread for Swahili NLP enthusiasts.
Welcome, and please feel free to introduce yourself with any of the following:

Your name, Github, Hugging Face, and/or Twitter handle
Your interest in Swahili NLP
Some projects you are working on or interested in starting
Any other languages that you speak, any personal interests, or anything else

Context:
Hello
My name is Davis David from Tanzania. I’m happy to be here.
My current project is creating a Swahili news dataset supported by AI4D. The train set is currently available in the hugging Face package called “datasets”.
Github Username: Davisy
Twitter handle @Davis_McDavid


Arabic NLP - Tutorial - الدورة التعليمية

Question:
NLP in Arabic with HF and Beyond
Overview
Arabic language consists of 28 basic letters in addition to extra letters that can be concatenated with Hamza (ء) like أ ، ؤ ، ئ that are used to make emphasis on the letter. Moreover, there are special characters called diacritics to compensate for the lack of short vowels in the language. This increases the number of letters to reach over 40 if we consider diacritics. In Arabic writing script, the words are usually connected together (ligature) which makes it challenging for many tasks like OCR. From a language construction perspective, Arabic is morphologically rich where multiple prefixes, suffixes and infixes can be added to a single stem. There are two types of morphology in Arabic: derivational and inflectional. Derivational morphology is common in Arabic which allows deriving multiple words from a single stem. On the other hand inflectional morphology can add multiple morphemes to a single stem to create much longer and complex words. As a result, Arabic verbs can have up to 5,400 different forms. This rich morphology gave rise to completely different and complex dialects around the Arabic world. Moreover, the complexity of the language makes it challenging for language modelling in general.
State of NLP in Arabic
Before the recent advances in NLP there was some focus on the analysis of morphology in Arabic. Notably, a lot of tools have been created for morphological analysis like segmentation : FARASA 6 and MADAMIRA 8. Later, after the success of word embeddings like Word2Vec, AraVec 9 was created as a very large word embedding for modern standard arabic and dialects. Recently with the advances in transformers many models have been pre-trained for Arabic like BERT based model i.e AraBERT 5, and improved variants MARBERT & ARBERT 5, a GPT-2 based model AraGPT-2 2, ELECTRA based model AraELECTRA 3 and GigaBERT 1 using zero shot cross lingual understanding from English. Not to mention the task specific models for different tasks like dialect identification, fake news detection, sentiment analysis, translation, etc.
Datasets and Models in HF
Currently there are around 50 datasets 26 for Arabic in different tasks like sentiment analysis, text classification, question answering, translation, unsupervised training, localization, etc. All the transformer models in the previous section are available in the huggingface model hub. In addition there are currently over 70 models 12 for Arabic ranging from monolingual to multilingual models which tackle MSA and dialects.
Arabic Transformers in HF
We can use the datasets module to load any available dataset for Arabic. In this example we use the MetRec dataset which contains poems labeled in terms of meters. In the code snippet below we can load the dataset
from datasets import load_dataset
dataset = load_dataset('metrec')

As we can see if we print the variable we see the dataset contains 47,124 records for training and 8,316 records for testing
DatasetDict({
train: Dataset({
features: ['text', 'label'],
num_rows: 47124
})
test: Dataset({
features: ['text', 'label'],
num_rows: 8316
})
})

Transformers with tokenizers can be used also with ease. Here is a simple example for loading some pretrained Arabic models and tokenizers using transformers
from transformers import AutoTokenizer, AutoModel

PRE_TRAINED_MODEL_NAME = 'aubmindlab/bert-base-arabertv01'

tokenizer = AutoTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME)
bert_model = AutoModel.from_pretrained(PRE_TRAINED_MODEL_NAME)

You can do some complex tasks like training and fine-tuning as you can see in this tutorial 29.
Relevant Resources
As the transformers and tools of hugging face expand to multiple languages, the community is building tools that build on top of that. In Arabic, Tnkeeh 18 (تنقيح) is an Arabic text cleaning library that can be used for removing diacritics, normalization and segmentation and many other preprocessing tasks. The library can process instances of the datasets module. Also CAMeL Tools 10 have incorporated transformers in their api that can be used for many tasks like sentiment classification.

Context:
معالجة اللغة الطبيعية باستخدام مكتبة هقنق فيس huggingface
تمت  الترجمة بواسطة عبد الإله المسفر
المقدمة
تتكون أحرف اللغة العربية من ٢٨ حرفا بالإضافة الى أحرف أخرى تتشكل عند جمعها مع الهمزة (ء). مثلا: أ، ؤ، ئ. مايميز اللغة العربية آيضا هو وجود التشكيل، مايجعل للكلمات معان مختلفة على الرغم من مشاركتها نفس الأحرف (عِقْد، عَقّد، عُقْد). مما يجعل عدد الأحرف يتخطى ٤٠ حرفا اذا ما اخذنا التشكيل بعين الإعتبار. عند كتابة اللغة العربية، تتصل الأحرف ببعضها بطريقة تختلف عن الشكل الأساسي للأحرف على حدة. مثلا: (ح ر ف) تتشكل عند الكتابة الى (حرف)، مما يجعل بعض مهام تعلم الآلة صعبة، مثل التعرف على الأشكال (OCR). من منظور بناء اللغة ، تعتبر اللغة العربية غنية من الناحية الشكلية, حيث يمكن إضافة العديد من البادئات واللواحق إلى جذع واحد. هناك نوعان من التشكل في اللغة العربية: الاشتقاقي والتصريف. علم التشكل الاشتقاقي شائع في اللغة العربية والذي يسمح باشتقاق كلمات متعددة من جذع واحد. من ناحية أخرى ، يمكن أن تضيف مورفولوجيا التصريف العديد من الأشكال إلى جذع واحد لإنشاء كلمات أطول ومعقدة. نتيجة لذلك، يمكن أن تحتوي الأفعال العربية على 5400 صيغة مختلفة. أدت هذه التركيبة الغنية إلى ظهور لهجات مختلفة تمامًا ومعقدة في جميع أنحاء العالم العربي. علاوة على ذلك ، فإن تعقيد اللغة يجعل نمذجة اللغة مهمة صعبة.
تطورات معالجة اللغة الطبيعية في اللغة العربية
قبل التطورات الحديثة في معالجة اللغة الطبيعية، كان هناك بعض التركيز على تحليل التشكل في اللغة العربية. والجدير بالذكر أنه تم إنشاء الكثير من الأدوات للتحليل الصرفي مثل التجزئة في فراسة FARASA 1 ومداميرا MADAMIRA 3. في وقت لاحق، بعد نجاح تقنيات تضمين الكلمات مثل Word2Vec ، تم إنشاء AraVec 5 ككلمة كبيرة جدًا يتم تضمينها في اللغة العربية واللهجات القياسية الحديثة. في الآونة الأخيرة مع التقدم في المحولات TRANSFORMERS ، تم تدريب العديد من النماذج مسبقًا على اللغة العربية مثل النموذج القائم على BERT ، مثل  AraBERT 4، والمتغيرات المحسنة MARBERT & ARBERT 3 ، وهو نموذج يعتمد على GPT-2  AraGPT-2 1 ، ونموذج يعتمد على ELECTRA AraELECTRA 3و GigaBERT باستخدام لغة بدون طلقة فهم من اللغة الإنجليزية. ناهيك عن نماذج المهام المحددة لمهام مختلفة مثل تحديد اللهجة واكتشاف الأخبار المزيفة وتحليل المشاعر والترجمة وما إلى ذلك.
بيانات ونماذج اللغة العربية في مكتبة هقنق فيس huggingface
يوجد حاليًا حوالي 50 مجموعة بيانات 1 للغة العربية تختص في مهام مختلفة مثل تحليل المشاعر ، تصنيف النص ، والإجابة على الأسئلة ، والترجمة ، والتدريب غير الخاضع للإشراف ، والتخصيص ، وما إلى ذلك. جميع نماذج المحولات في القسم السابق متوفرة في محور نماذج 8 مكتبة هقنق فيس huggingface. بالإضافة إلى ذلك ، يوجد حاليًا أكثر من 70 نموذجًا للغة العربية تتراوح من النماذج أحادية اللغة إلى النماذج متعددة اللغات التي تتعامل مع اللغة العربية الفصحى واللهجات.
المحولات العربية في مكتبة هقنق فيس huggingface
يمكننا استخدام وحدة مجموعات البيانات لتحميل أي مجموعة بيانات متاحة للغة العربية. في هذا المثال ، نستخدم مجموعة بيانات MetRec التي تحتوي على قصائد مصنفة من حيث الأمتار. في مقتطف الشفرة أدناه يمكننا تحميل مجموعة البيانات.

from datasets import load_dataset
dataset = load_dataset('metrec')

كما يمكننا أن نرى أن مجموعة البيانات تحتوي على 47124 سجلًا للتدريب و 8316 سجلًا للاختبار

DatasetDict({
train: Dataset({
features: ['text', 'label'],
num_rows: 47124
})
test: Dataset({
features: ['text', 'label'],
num_rows: 8316
})
})

يمكن أيضًا استخدام المحولات ذات الرموز المميزة بسهولة. فيما يلي مثال بسيط لتحميل بعض النماذج العربية التي تم اختبارها مسبقًا والرموز المميزة باستخدام المحولات.

    from transformers import AutoTokenizer, AutoModel
PRE_TRAINED_MODEL_NAME = 'aubmindlab/bert-base-arabertv01'

tokenizer = AutoTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME)
bert_model = AutoModel.from_pretrained(PRE_TRAINED_MODEL_NAME)

يمكنك القيام ببعض المهام المعقدة مثل التدريب والضبط كما يظهر في هذه الشروحات 20.

مصادر ذات صلة
نظرًا لتوسع محولات وأدوات تعانق الوجه إلى لغات متعددة ، يقوم المجتمع ببناء أدوات تعتمد على ذلك. في اللغة العربية ، مكتبة تنقيح 3 لتنظيف النصوص العربية يمكن استخدامها لإزالة علامات التشكيل والتطبيع والتجزئة والعديد من مهام المعالجة المسبقة الأخرى. يمكن للمكتبة معالجة مثيلات وحدة مجموعات البيانات. كما قامت CAMeL Tools 12 بدمج المحولات في واجهات برمجة التطبيقات الخاصة بها والتي يمكن استخدامها في العديد من المهام مثل تصنيف المشاعر.


Question:
This category should be used to propose and join existing projects that make use of  Flax/JAX and  Transformers. Make sure that your project proposal includes all relevant information (as listed below) for people to join/comment/improve your idea.
Feel free to also post any questions you might have about  Transformers flax examples 28. If you found issues in one of the flax examples, please submit an issue to the transformers repo instead.
Relevant information when proposing a / project:

Description: What is the project about? E.g. Train a CLIP model in Swedish for zero-shot image classification.
Model: What model do you propose to use? E.g. openai/clip-vit-base-patch32 · Hugging Face 3 in Flax/JAX
Dataset: What dataset do you want to use? E.g. GitHub - google-research-datasets/wit: WIT (Wikipedia-based Image Text) Dataset is a large multimodal multilingual dataset comprising 37M+ image-text sets with 11M+ unique images across 100+ languages.

Training scripts: What training scripts do you want to use? E.g. Training scripts will be created as part of the project
Expected result: What do you expect the outcome of the project to be? E.g. A zero-shot image classification pipeline for CLIP

Context: Hi - I wanted to propose a project. But the submitted projects are really interesting. Is it possible to join more than one ?


Pretrain GPT-Neo for Open Source GitHub Copilot Model

Question:
Open Source GitHub Copilot for auto generating code
I would like to train an open source version of the new awesome GitHub Copilot AI tool, which is based on GPT3. Similar to the awesome people behind GPT-Neo, having such an open source model would greatly help researchers understand what this type of biases and limitations this kind of code autocompletion model might have such as generating insecure code (i do research in this area and i know my team would love an open sourced version to run experiments on, i.e. try and break it )

2. Language
The model will be trained on different programming languages such as C, C++, java, python, etc.

3. Model
GPT-Neo

4. Datasets
Datasets that contain hopefully high quality source code
Possible links to publicly available datasets include:

code_search_net · Datasets at Hugging Face 48
Hugging Face – The AI community building the future. 26

Some additional datasets may need creating that are not just method level.

5. Training scripts
I believe the standard CLM language model script would do for this.
We can make use of transformers/run_clm_flax.py at master · huggingface/transformers · GitHub 46

6. (Optional) Challenges
The data additional data may be a challenge. From what I can see in copilot, it looks to be training on entire files, not code snippets. There are file level datasets that exist but they are a few years old and i don’t think they cover many programming languages. The ones I listed above have multiple languages but are only methods.
However, githubs API is pretty easy to use and so it would be pretty easy to create one from scratch, especially if we get some insights into how the copilot dataset was generated

7. (Optional) Desired project outcome
I’d love to have this open source model setup in a similar Visual Studio Code extension to the GitHub Copilot one. I’ve actually made a tutorial on doing this using the GPT-Neo model, so we could easily clean it up and release it free of charge forever because from what I’ve seen on Twitter the GitHub Copilot might eventually be put behind a paywall .

8. (Optional) Reads
The following links can be useful to better understand the project and
what has previously been done.

Introducing GitHub Copilot: your AI pair programmer | The GitHub Blog 34

Deep Learning to Autocomplete Code: Part #2 Autogenerating Code - VS Code Tutorial - YouTube 32  (tutorial on how we could setup the demo of the model once it’s done cooking)

Context: Very much interested to be part of this project.


Question: Been digging through the HF repo and haven’t been able to locate the training scripts for flax / jax models.  Are there scripts for training the various jax model implementations from scratch?

Context: Hi there! You can find the official examples at https://github.com/huggingface/transformers/tree/master/examples/flax 4 as well as notebooks at https://github.com/huggingface/notebooks/tree/master/examples 3.


Image captioning for French with pre-trained vision and text model

Question:
Image captioning with pre-trained vision and text model
For this project, a pre-trained image model like ViT can be used as an encoder, and a pre-trained text model like BERT and/or GPT2 can be used as a decoder.

Model
Pre-trained ViT 8, BERT 3, and GPT2 4 models can be found on the model hub

Datasets
The WIT 3 dataset can be used for this task. It has almost over 1M image-text examples for the French language.

Available training scripts
As this will be a Seq2Seq model, the run_summarization_flax.py 4 script can be used for training this model with some modifications.

(Optional) Desired project outcome
The desired outcome is to see if pre-trained vision and text models can be leveraged for image captioning and also train captioning models in French languages.  This can be showcased directly with a streamlit  or gradio app.

(Optional) Challenges
This model will require some modifications to the existing text models. Specifically, as this will be a seq2seq model, we’ll need to add a randomly initialized cross-attention layer in BERT or GPT2 to use it as a decoder in the encoder-decoder setting.

(Optional) Links to read upon
This keras example presents an excellent example of how image encoder and transformer can be used for image captioning.

      keras.io

Keras documentation: Image Captioning 13

Context:
Hi, I am a bit late for deciding the project to work on. This one seems a good fit for me. Is it possible?
And one more question, there is another project named “Multilingual Image Captioning”. It has already 8 participants, so I don’t expect to be a new member for that project. However, would it be possible to join their discussions?


Question:
PreTrain Wav2Vec2 in German
There is currently only a multilingually pretrained model for German Wav2Vec2. Let’s make a Wav2Vec2 only pretrained on German.

Model
A randomly initialized Wav2Vec2 model

Datasets
One can make use Common Voice 1 the dataset is also available through the datasets library here: common_voice · Datasets at Hugging Face 2.

Available training scripts
FlaxWav2Vec2 will be merged soon: [Flax] Add wav2vec2 by patrickvonplaten · Pull Request #12271 · huggingface/transformers · GitHub 13 and a pretraining script should be relatively easy to be merged.

(Optional) Desired project outcome
The best German ASR model.

(Optional) Challenges
It might make sense to use more data than just common voice.

Context:
I’d actually be happy to join this team
Also pinging @flozi00 @stefan-it in case you guys are interested.


Question:
PreTrain Wav2Vec2 in Swedish
There is currently only a multilingually pretrained model for Swedish Wav2Vec2. Let’s make a Wav2Vec2 only pretrained on Swedish.

Model
A randomly initialized Wav2Vec2 model

Datasets
One can make use Common Voice the dataset is also available through the datasets library here: common_voice · Datasets at Hugging Face.

Available training scripts
FlaxWav2Vec2 will be merged soon: [Flax] Add wav2vec2 by patrickvonplaten · Pull Request #12271 · huggingface/transformers · GitHub 3 and a pretraining script should be relatively easy to be merged.

(Optional) Desired project outcome
The best Swedish ASR model.

(Optional) Challenges
It might make sense to use more data than just common voice.

Context: Maybe also interesting for @marma ?


Question:
Pretrain Wav2vec2 in Russian
There is currently only a multilingually pretrained model for Russian Wav2Vec2. Let’s make a Wav2Vec2 only pretrained on Russian.

2. Language
The model will be trained in Russian.

3. Model
A randomly initialized Wav2Vec2 model

4. Datasets
Part of CommonVoice - CommonVoice 1 - 35h
SberGolos - Golos 2 - 1095h
SovaDataset - SovaDataset 1 - 400h

5. Training scripts
We can make use of run_wav2vec2_pretrain_flax 13 to train the model.

6. (Optional) Challenges
Dataset have many hours, training will take a long time

7. (Optional) Desired project outcome
The best Russian ASR model.

Context: @anton-l


Question:
PreTrain Wav2Vec2 in Dhivehi
There is currently only a multilingually pretrained model for Dhivehi Wav2Vec2. We would like to make a Wav2Vec2 only pre-trained on Dhivehi.

Model
A randomly initialized Wav2Vec2 model

Datasets

commonvoice has 18hrs in the last released dataset. [ 32hrs+ if mid 2021 dataset released in time]
podcast data [30hr]
others []

Available training scripts
FlaxWav2Vec2 will be merged soon: [Flax] Add wav2vec2 by patrickvonplaten · Pull Request #12271 · huggingface/transformers · GitHub 4 and a pretraining script should be relatively easy to be merged.

(Optional) Desired project outcome
The best Dhivehi ASR model

(Optional) Challenges
scraping publicly available Dhivehi audio from various sources

Context: Hi, I would like to join this project.


Question:
RoBERTa/BERT for Hindi
Currently, there is only a very limited amount of BERT-like models for Hindi on the hub: Hugging Face – The AI community building the future. 2 . For this project, the goal is to create a RoBERTa/BERT model for just the Hindi language.

Model
A randomly initialized RoBERTa/BERT model

Datasets
One can make use OSCAR 4 the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face.

Available training scripts
A masked language modeling script for Flax is available here 5. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a strong RoBERTa/BERT model in Hindi.

(Optional) Challenges
The OSCAR dataset might be too small (it has < 10GB of data for Hindi). Also it might be important
to find datasets the BERT-like model can be evaluated on after pretraining in Hindi. Having found a dataset to fine-tune the pretrained BERT-like model on, one can make use of the text-classification script here 5

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory 6

Context:
I am planning to work on this project.
So, here is my first question
As the current dataset size is too small for it to effectively train BERT and planning on building my own dataset, by scrapping website that host content in Hindi.
The first thing that comes to mind is scrapping news article, which perpetual source of data, we can also tap into some social networks which have more content in Hindi, such as ShareChat. Hindi/Hindustani Literature websites
Does this approach sounds reasonable, or is too weak because common crawl could have already covered such websites?


Train a VAE to interpolate on English sentences

Question:
Transformer-VAE
Convert a T5 model into a variational autoencoder for text.
I have already made a project that does this in PyTorch but its never been trained at scale.
This project is to convert the autoencoder into Flax so it can be trained efficiently on a TPU to train the largest ever Transformer-VAE!

Language
The model will be trained in english.

Model
Build on T5-base, this will match with the Optimus model.
Only additional parameters come from a small Autoencoder module that will go between the encoder and decoder.

Datasets
Use the wikipedia sentences dataset from OPTIMUS.
This comes tokenized so we’ll need to use its tokenizer with T5.

      github.com

    ChunyuanLI/Optimus/blob/master/download_datasets.md 6

    # Download/Pre-process Datasets

## Wikipedia

Download processed files (11.78G) below, and unzip it (298 files)

https://textae.blob.core.windows.net/optimus/data/datasets/wikipedia_json_64_filtered.zip

Download raw file (11.79G):

https://textae.blob.core.windows.net/optimus/data/datasets/wikipedia.segmented.nltk.txt

Our pre-processing protocal: We split the original wiki text into 298 files, and loop over files in one epoch.

We filter each sentence in wiki based on two constraints: (1) The sentence length is smaller than 64. (2) The tokenized sentence length is smaller than 256 (so that the encoder can take the entire sentence).

To filter the sentence, please change the data folders and run the script:

    sh scripts/scripts_local/run_data_filtering_wiki.sh

  This file has been truncated. show original

Training scripts
The original PyTorch training script 17 is adapted from the old Huggingface clm training script so using the flax clm script 3 should be a good base to build on.

Challenges
The original model was made with PyTorch so there will be some features that can’t be ported over. E.g. I added a prism layer to the PyTorch code which requires FFTs.

Desired project outcome
A colab notebook where people can explore the Transformer-VAE’s latent space.

Interpolate between sentences.

Transfer the style/content of one sentence to another.

Do gradient descent on a sentences latent code to get the desired sentiment, classification score, etc.

Reads
Here are some background links to understand the context behind this project:

Initial Transformer-VAE post 24

and the improvements post 9.

MMD-VAE 9, the MMD loss that Transformer-VAE uses.

OPTIMUS 10 Current SOTA text-vae will give a sense of the outputs would should expet.

Context:
How does this work?
Currenlty the takes a transformer encoder and decoder and puts a VAE between them.

.1554×742 43.5 KB

The VAE forms a compressed latent code which allows interpolating on the training data.

.1540×926 69 KB

For regulatisation an MMD loss is used instead of KL Divergence which reduces posterior collapse.

.1528×916 60.8 KB


PreTrain ELECTRA from scratch in Portuguese

Question:
ELECTRA for Portuguese
Currently, there is no ELECTRA or ELECTRA Large model that was trained from scratch for Portuguese on the hub: Hugging Face – The AI community building the future. 3. For this project, the goal is to create a ELECTRA model for just the Portuguese language.

Model
A randomly initialized ELECTRA model

Available training scripts
A masked language modeling script for Flax is available here 4. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a strong ELECTRA model in Portuguese.

Context: ELECTRA actually uses a special pretraining script, which would need to be written during the sprint - if you are very motivated I’m willing to confirm this “single-person” project and give you a TPU VM. Please leave a comment on the official googe sheet in this case: Confirmed teams for Flax/JAX community week - Google Sheets 5


Question:
T5 for Arabic
Currently there is a fair amount of Encoder-only and Decoder-only models for Arabic (AraBERT, AraElectra, AraGPT2, etc.), but there aren’t any seq2seq models. The goal of this project is to pretrain a T5 language model for the Arabic language.

Model
A randomly initialized T5 model. I’m not sure yet what the biggest model size that can be trained during a one week period however. Ideally, I think we should train the biggest variant possible, given that bigger models converge faster.

Datasets

The Arabic subset of the mC4 dataset 7

The Arabic subset of the Oscar dataset 3

The Arabic subset of mC4 should be more than enough though. It contains 57 Billion tokens (according to the mT5 paper) and the English T5 model was trained on 2^{35}  (about 34 Billion) tokens (T5 paper).

Training scripts
We can use the recently released T5 pretraining script 13

Challenges
I have never worked with datasets this huge.

Desired outcomes

An Arabic T5 model with reasonable results when fine-tuned for summarization, question generation, question answering and so on.
Learning some JAX and having fun

Team members:

ManarAli
Onyx
salti

Context:
I am very interested in this project.
Are you still looking for partners?


Train a GPT2 and ViT model to detect online harm

Question:
Project Description
The aim of the project is to improve online safety for vulnerable populations (e.g. adolescents). We would like to train a NLP and CV model to detect hate speech and memes, toxic comments, and potentially identify the presence of online predators as users engage in conversations or browse through a website.
Language
The model will be trained in English.
Model
We will finetune pre-trained GPT2 and ViT models for text and image classification respectively, but are also open to testing other models like ELECTRA and RoBERTa.
Datasets
Possible links to publicly available datasets include:

https://hatespeechdata.com/ 4
Toxic Comment Classification Challenge | Kaggle 3
Hateful Memes Challenge and dataset 2

Training scripts
We can start with existing Flax scripts for sequence classification (transformers/run_flax_glue.py at master · huggingface/transformers · GitHub 1) but will likely have to create our own for ViT.
(Optional) Challenges
One challenge will be to appropriately identify the intent of a given statement within the broader content of the visited site. For example, the inclusion of profanities can massively impact the evaluation of a sentence even if the overall sentence is meant to convey a positive sentiment.
A second challenge will be the ability of the model to correctly identify forms of toxicity that it has yet to encounter, and also circumvent biases (e.g. ethnicity) that could arise as a result of the data.
(Optional) Desired project outcome
We would like to have a demo that can run in near real-time. Future work can similarly adopt a multimodal approach to tackle videos and other forms of media.
(Optional) Reads
The following links can be useful to better understand the project and what has previously been done.

Detecting predatory conversations in social media by deep Convolutional Neural Networks: (PDF) Detecting predatory conversations in social media by deep Convolutional Neural Networks 1

Contextualizing Hate Speech Classifiers with Post-hoc Explanation: [2005.02439] Contextualizing Hate Speech Classifiers with Post-hoc Explanation 1

Detecting Online Hate Speech Using Context Aware Models: https://arxiv.org/pdf/1710.07395.pdf

Team
@diffindiff 1
@krecceg
@draziert

Context: Should be an interesting project!


Question:
Hi,
So am fairly new at NLP, so im just getting used to HF and NLP in general. My project is at img-caption in spanish 24. Thus following questions,

IT looks like wit 14 dataset isnt on HF library, so downloaded it on the TPU. But files are on tsv and the HF parsers I found support csv, py dicts, json etcs. What should be the fastest way here to parse correctly? any HF library for TSVs?

Valhalla 6 my proj lead suggested using run_summarization_flax.py 9 script, but I am a bit confused. How would I go about specifying ViT as encoder; and how would I go about adding my cross-attn layer to BERT and specifying it to that script ? I see only one model_dir and tokenizer_name which I am not sure for my use case.

I just want to start with pre-trained networsk and once it works, go with fine tuning etc. Thanks

Context:
Hi there,
Regrading the dataset:
Yes, the dataset is not available in datasets lib. You will need to download the tsv and the prepare the dataset by downloading the image. The tsv file for wit contains the image URLs and other metadata.
This script might help 8. It’s for downloading conceptual captions data, but you could re-purpose it to download WIT.
Regarding model:
There is no off-the-shelf model for this in transformers (yet!). What we need here is a Seq2Seq model. Where that image encoder is a pre-trained vision model like ViT or CLIP's vision model and the decoder is any pre-trained text model (BERT/ROBERTa).
To do this we will need to modify the BERT/ROBERTa model and add a cross-attention layer in it.
The encoder will the encode images, the decoder will take the decoder_input_ids and the encoder_hidden_states.
This should be similar to encoder-decoder model in pytorch transformers/modeling_encoder_decoder.py at master · huggingface/transformers · GitHub 59.
@gchhablani  is also working on a similar project and might have the modeling code ready
Let me know if you have any more question.


Pre-training/fine-tuning Seq2Seq model for spelling and/or grammar correction in English

Question:
Pre-training/fine-tuning Seq2Seq model for spelling and/or grammar correction
For this project, one can use a randomly initialized or a pre-trained BART/T5 model.

Model
Pre-trained BART 16, T5 8 models can be found on the model hub.

Datasets
The dataset for this model can be prepared as described in this blog post 50.
One can make use OSCAR 13 . The dataset is also available through the datasets library here: oscar · Datasets at Hugging Face 7.

Available training scripts
As this will be a Seq2Seq model, the run_summarization_flax.py 31 script can be used for training.

(Optional) Desired project outcome
The desired outcome is to train a spelling correction model for the English language. This can be showcased directly on the hub or with a streamlit  or gradio app.

(Optional) Challenges
Implementing the dataset noising function would be the challenging part of the project.

(Optional) Links to read upon

      Microsoft Research – 8 Feb 21

Speller100 expands spelling correction technology to 100+ languages 50

  Researchers in Microsoft Search and AI created Speller100 to expand spelling correction technology to 100-plus languages. Learn how it uses the concept of language families and zero-shot learning to make spelling correction more inclusive of...

Context: I have been working on automatic punctuation prediction so I am keen to explore other aspects of language correction as well. This project would be great for that. Count me in!


PreTrain RoBERTa from scratch in Portuguese

Question:
RoBERTa for Portuguese
Currently, there is only a very limited amount of RoBERTa or RoBERTa Large model that was trained from scratch for Portuguese on the hub: Hugging Face – The AI community building the future. 3. For this project, the goal is to create a RoBERTa model for just the Portuguese language.

Model
A randomly initialized RoBERTa model

Available training scripts
A masked language modeling script for Flax is available here 3. It can be used pretty much without any required code changes.

Dataset
We can use dataset from BrWaC (Brazilian Web as Corpus) a large Portuguese corpus or Wikidata in Portuguese.

Challenges

Data pre-processing
Sample preparation for LM

(Optional) Desired project outcome
The desired project output is a strong RoBERTa model in Portuguese.

Context: This is an amazing idea! I’m really looking foward to be part of this team!


Fine-tune CLIP on satellite images+captions

Question:
Fine-tune CLIP on satellite image data

Description
Fine-tune CLIP on remote sensing image data to enable zero-shot satellite image classification and captioning.

Language
The model will be trained in english.

Model
CLIP 43

Datasets
RSICD 62 + any extra data we can find

RSICD is used for remote sensing image captioning task. more than ten thousands remote sensing images are collected from Google Earth, Baidu Map, MapABC, Tianditu. The images are fixed to 224X224 pixels with various resolutions. The total number of remote sensing images are 10921, with five sentences descriptions per image.

Possible links to publicly available datasets include:

GitHub - 201528014227051/RSICD_optimal: Datasets for remote sensing images (Paper:Exploring Models and Data for Remote Sensing Image Caption Generation) 62

Training scripts
The training script for CLIP is on the way (PR 126).

Desired project outcome
An example notebook/script for CLIP domain adaptation.
A zero-shot satellite image captioning/classification demo app.
A zero-shot text-to-image search app: finding relevant areas given bird-view images and text query.

Reads
The following links can be useful to better understand the project and
what has previously been done.

Exploring Models and Data for Remote Sensing Image Caption Generation 28

Context:
Sounds very interesting! May I join the project?
-sujit


Pretrained language model that enables non-autoregressive generation

Question:
Description: What is the project about?
Pre-train a language model for non-autoregressive generation at scale.
Recent pre-trained language models, e.g. BERT, ELECTRA, boost the performance of various natural language tasks. These approaches focused on scaling up massive training data and efficient pre-training. However, the meaning of their objective is not clear and prevents the further step beyond them. Therefore, we have to design another way to build up pre-trained language model with better interpretability.
The ability to generate language could improve the interpretability of a language model by inspecting the generated results. Some language models with generating language in an auto-regressive fashion, e.g. XLNet, are proposed, but it generates tokens one by one, which incurs slow decoding process.
In this project, we implement a (1) theoretically motivated language model with (2) highly parallelized model training while (3) enabling fast decoding scheme. We follow up a recent paper 22 and move this technique to the pre-training stage. We believe that this model could provides better understanding about the research community while efficiently utilizing state-of-the-art accelerator such as TPUv3.

Model: What model do you propose to use?
Transformer layers with conditional random fields. The detailed architecture is inside this paper 22.
The model architecture huggingface provided is used and the additional conditional random field layer will be implemented as a part of our project.

Dataset: What dataset do you want to use?
The same dataset trained for BERT. It would be wikipedia dataset.

Training scripts: What training scripts do you want to use?
Training scripts will be created as part of the project

Expected result: What do you expect the outcome of the project to be?
A new pre-trained architecture that generates long language quickly with non-autoregressive decoding.

Context: Good idea! I want to join