Question:
Use the Funnel Transformer + T5 model from the huggingface hub with some subclassing to convert them into a VAE for text.
The current SOTA VAE is OPTIMUS which still suffers from some posterior collapse.

      Microsoft Research

OPTIMUS: Organizing Sentences via Pre-trained Modeling of a Latent Space -... 52

  When trained effectively, the Variational Autoencoder (VAE) (Kingma and Welling, 2013; Bowman et al., 2016) can be both a powerful generative model and an effective representation learning framework for natural language. In this paper, we propose the...

Its training data is all open source.

      github.com

    ChunyuanLI/Optimus/blob/master/download_datasets.md 8

    # Download/Pre-process Datasets

## Wikipedia

Download processed files (11.78G) below, and unzip it (298 files)

https://textae.blob.core.windows.net/optimus/data/datasets/wikipedia_json_64_filtered.zip

Download raw file (11.79G):

https://textae.blob.core.windows.net/optimus/data/datasets/wikipedia.segmented.nltk.txt

Our pre-processing protocal: We split the original wiki text into 298 files, and loop over files in one epoch.

We filter each sentence in wiki based on two constraints: (1) The sentence length is smaller than 64. (2) The tokenized sentence length is smaller than 256 (so that the encoder can take the entire sentence).

To filter the sentence, please change the data folders and run the script:

    sh scripts/scripts_local/run_data_filtering_wiki.sh

  This file has been truncated. show original

From my experiments an MMD-VAE doesn’t suffer form as much posterior collapse in smaller scale models, why not try & scale up?

      Fraser Greenlee – 13 Aug 20

Transformers as Variational Autoencoders 48

  Avoid posterior collapse even with a huge model.

I’ve actually already got the code working to turn them into a Pytorch MMD-VAE so why not just convert it to JAX/Flax?

      github.com

    Fraser-Greenlee/transformer-vae/blob/master/transformer_vae/model.py 53

    """
    Base transformer-VAE model.
"""
import torch
from torch import nn
from typing import Dict, Any
from transformers.utils import logging
from transformers.modeling_utils import PreTrainedModel
from transformers.modeling_outputs import BaseModelOutput
from transformers.models.funnel.modeling_funnel import upsample
from transformers import AutoModelForSeq2SeqLM, AutoModelForMaskedLM

from transformer_vae.custom_t5 import modify_t5_stack
from transformer_vae.autoencoders import VAE_ENCODER_MODELS, VAE_DECODER_MODELS, EncoderDecoderVAE
from transformer_vae.critic import CRITIC
from transformer_vae.model_outputs import BaseTransformerVAE_Output
from transformer_vae.config import Funnel_T5_VAE_Config

logger = logging.get_logger(__name__)

  This file has been truncated. show original

Context:
Maybe this could make the basis for a new kind of search engine?

      Fraser Greenlee – 11 Mar 21

Interpolating the internet 20

  What happens when every plausible idea is an inference away.


Question:
Project Recreate a mini-version of DALL-E
Useful links:

Project Report 40
Demo 133
Github Repo 56

Original post:
It would be cool to recreate something like DALL-E 33.
I can see some challenges though:

difficult paper though the main concepts could be simplified
availability/collection of dataset
the model used was GPT-3, not sure if interesting results can be obtained with smaller models
may require more time to implement

Also I don’t believe a public version is available so maybe it will be easier to just do it in Pytorch (I have no experience yet with Flax/JAX).

Context: We Could train NLP based models to generate embedding for sentences and then use this embeddings as input embedding for gan model to generate images. This is pretty simple idea just vice versa of image captioning, but I think it can work pretty well given the performance of transformers based models.


HuggingArtists | Train a model to generate lyrics

Question:
logo3584×1792 183 KB

In HuggingArtists 11, we can generate lyrics by a specific artist. This was made by fine-tuning a pre-trained HuggingFace Transformer 1 on parsed datasets from Genius 3.
As well, we use Weights & Biases integration to automatically log model performance and predictions.
 2
Project repository:
 11

About
Create training script for TPU with Flax/JAX. So users can choose GPU or TPU in Colab.

Model
A pre-trained GPT2 model in English and fine-tune it to learn English lyrics.
The GPT2 model, pre-trained in English: link 3.

Datasets
Parsed with this script 4 from Genius 3.

Training scripts
GPU with PyTorch is available in project Colab:
 3
So we need to create for TPU with Flax/JAX.

Project outcome
Create training script for TPU with Flax/JAX. So users can choose GPU or TPU in Colab.

Challenges

I am new with Flax/JAX, so I need some help.
There is no other challenges

Conclusion
We can integrate this project with Video and Music generation so we can create our AI-Artist
I hope to find like-minded people and create a cool project

Context: Wow this is really cool!


Question:
GPT2 for Bengali
Currently, there is no GPT2 model that was trained from scratch for Bengali on the hub. For this project, the goal is to create a strong language generation model for Bengali using GPT2 Model.

2. Language
Bengali.

3. Model
A randomly initialized GPT2 model.

4. Datasets
One can make use of OSCAR the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face 4. The total Bengali resource in OSCAR is 11 GB.
Another source can be the mC4 dataset which is available in AllenAI 2. The resource size is 29GB.

5. Training scripts
A causal language modeling script for Flax is available here 10. It can be tweaked for training GPT2.

6. Challenges

Fix a good tokenizer that covers Bengali vocabulary properly and make sure that the LM doesn’t become character-level LM.

7. Desired project outcome
The desired project output is a GPT2 model that is able to generate Bengali language.

8. Reads
The most important read would be the following colab:

Google Colaboratory  13

Other reads that might be interesting include:

GPT-2 no habla español: Artificial Intelligence, Anglocentrism, and the non-human side of DH | | Vanderbilt University  2
https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171  2

Context: I am also a Bengali speaker. I am in!


Pretrain GPT-J-6B from scratch on Arabic

Question:
The idea behind this project is to pretrain a GPT-J-6B on an Arabic corpus using JAX and TPUs of course.
Some resources:

Automate Text-based Workflows at Scale 15
GPT-J-6B: 6B JAX-Based Transformer – Aran Komatsuzaki 23

Context:
@ yalouini
Interesting how to particpate


Auto-generation of Messages to Commits | Project Proposing

Question:
0_gHL7902eGDXwmR0Y800×449 68.1 KB

Description
PreTrain language model for automatic generation of messages to commits.
Version control systems are used in the development of many projects, so the tool can be relevant for a wide range of developers.

Model
Here we need to use the Text2Text/Sequence2Sequence or Text Generation model. Better to discuss it all together.

Dataset
Not sure that there are existing datasets for us. But we can get everything from GitHub API.

How to collect data

Get all branches from the repository:
https://api.github.com/repos/huggingface/transformers/branches?per_page=100&page=2

Result (master as example):
[...
   {
       "name": "master",
       "commit": {
         "sha": "3ff2cde5ca4a2d3c622b827d9edf7e3d0b7f4fb7",
         "url":    "https://api.github.com/repos/huggingface/transformers/commits/3ff2cde5ca4a2d3c622b827d9edf7e3d0b7f4fb7"
       },
       "protected": true
   },
...
]

Get all commits from the branch:

page=1 and sha=7a8d6b19767a92b1c4ea45d88d4eedc2b29bf1fa as example

https://api.github.com/repos/huggingface/transformers/commits?per_page=100&sha=3ff2cde5ca4a2d3c622b827d9edf7e3d0b7f4fb7&page=1

Result:
An array of commits. Json of each commit is huge, so I will not paste it. From each commit we need a url.
https://api.github.com/repos/huggingface/transformers/commits/24cbf6bc5a0b6a9bb5afdda6bb1a329ac980fa4b

Get commit message and patches:

Message - $.commit.message

Patches - array of changed files $.files, where each file has $.patch

Patch example:

image1523×570 37.9 KB

@@ -193,7 +193,7 @@ It is recommended to pre-train Wav2Vec2 with Trainer + Deepspeed (please refer t\n Here is an example of how you can use DeepSpeed ZeRO-2 to pretrain a small Wav2Vec2 model:\n \n ```\n-PYTHONPATH=../../../src deepspeed --num_gpus 2 run_pretrain.py \\\n+PYTHONPATH=../../../src deepspeed --num_gpus 4 run_pretrain.py \\\n --output_dir=\"./wav2vec2-base-libri-100h\" \\\n --num_train_epochs=\"3\" \\\n --per_device_train_batch_size=\"32\" \\

Training scripts
I think we would use our own script for this project. As well, we can refer to any example script.

Expected result
Ready to go model. So we can integrate it with source code editors and browser extensions.

About
I am really interested in this project. I hope to find like-minded people and create a cool project
Even if you don’t want to participate - like and reply to this topic so that more people will see it!

Context:
Hello, Aleksey. This sounds interesting.
Something that might be useful to you, CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model 5.
I might be able to help you very passively. Let me know. They have a codebase alongside the dataset over here 3.
Let me know.
Best Wishes !
PS : Look at this community Code.AI 3, you are more likely to find collaborators over there.


Pre-training/fine-tuning Seq2Seq model for spelling and/or grammar correction in French

Question:
Pre-training/fine-tuning Seq2Seq model for spelling and/or grammar correction
For this project, one can use a randomly initialized or a pre-trained BART/T5 model.

Model
Pre-trained BART 1, T5 4 models can be found on the model hub.

Datasets
The dataset for this model can be prepared as described in this blog post 12.
One can make use OSCAR 2 . The dataset is also available through the datasets library here: oscar · Datasets at Hugging Face 4.

Available training scripts
As this will be a Seq2Seq model, the run_summarization_flax.py 4 script can be used for training.

(Optional) Desired project outcome
The desired outcome is to train a spelling correction model for the French language. This can be showcased directly on the hub or with a streamlit  or gradio app.

(Optional) Challenges
Implementing the dataset noising function would be the challenging part of the project.

(Optional) Links to read upon

      Microsoft Research – 8 Feb 21

Speller100 expands spelling correction technology to 100+ languages 12

  Researchers in Microsoft Search and AI created Speller100 to expand spelling correction technology to 100-plus languages. Learn how it uses the concept of language families and zero-shot learning to make spelling correction more inclusive of...

Context: Hi @valhalla I have previously worked on grammar correction for English using T5, and it gives a great result. It would an exciting task to do same on French, would like to be part of this project.


Wav2vec2 for dementia screening based on spontaneous speech

Question:
wav2vec2 for dementia screening
Dementia is hard to diagnose. And there is no known cure, maybe because it too late by the time the diagnosis is confirmed. It is understood that the development of dementia commences 10 to 15 years before the symptoms first appear.  Also, language, especially spontaneous speech, is a promising indicator/biomarker for diagnosing dementia and other cognitive disorders.

2. Language
The model will be trained on English audio.

3. Model
As far as I know, wav2vec2 is the best candidate.

4. Datasets
There is dementia classification from dementia bank. However, it is a binary classification dataset for dementia and no dementia.
To predict dementia 10 to 15 before the onslaught of symptoms, we would need longitudinal data on individuals who develop dementia. The only such available datasets I know is the Framingham heart study, and it is text only.
I have been building a list of public figures diagnosed with dementia and scrapping videos of youtube into different categories; after symptoms, two years before symptoms, five years before symptoms etc. Over the next week, I will build a streamlit app to extract 8 to 10 secs of audio files of the person of interest from the video.
Maybe we can use data from dementia bank or other sources for no dementia class??
Possible links to publicly available datasets include:

Dementia Bank

Google sheet 2 with list of public figures with dementia and YouTube urls

5. Training scripts
TODO

6. Challenges
The dataset is too small, too noisy?
No dataset on time for no dementia

7. Desired project outcome
A proof of concept streamlit app that this works?

8. Reads
The following links can be helpful to better understand the project and
what has previously been done.

IBM efforts using Farmingham data for dementia prediction

Context: Really like this project! Hope more people will be interested


Question:
Motivation
Generative representation learning has seen incredible success in natural language processing, but it remains under-explored in computer vision, where contrastive methods (e.g. SimCLR, MoCo, Swav, DINO, etc.) are dominant. In this project, we will reproduce the state-of-the-art in generative representation learning for vision (BEIT 14) and hopefully extend upon it.

Description
For this project, we will replicate and hopefully extend BEIT 14.
The first and most important part of the project will be reproducing the results of BEIT on the tasks of image classification and semantic segmentation. This entails pre-training on ImageNet for 800 epochs and then finetuning on ImageNet/ADE20K for 100 epochs.
The BEIT architecture is straightforward: it is a ViT that predicts visual tokens given by a VQ-VAE. We will use the Huggingface transformers ViT implementation and the VQ-VAE from DALL-E (or possibly the VQ-GAN from the Taming Transformers 2 paper, we shall see). Personally, I have have a lot of experience with these types of models, but I do not have much experience with Jax/Flax, so this will be a learning experience.
The BEIT training procedure, data loading, and image augmentations are similar to those used in popular vision transformer papers (ViT, DeiT, etc.). All hyperparameters are listed in the appendix of the paper.
After reproducing the BEIT results, I have a number of ideas for extending the architecture. First, I’d like to explore ways of combining contrastive self-supervised objectives (e.g. SimCLR, MoCo, DINO, etc.) with BEIT. Appendix D in the BEIT paper explored one way of combining BEIT with DINO, but I think there could be better ways of combining the masked language model and contrastive objectives.

Notes on the Paper
Some short notes on the paper are here 6.

Context: Officially defining it!


BERTIN: PreTrain RoBERTa-large from scratch in Spanish

Question:
BERTIN
The idea is to use the Spanish portion of mC4 12 (which roughly amounts for 1TB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512. It might be a good idea to start in 128, as suggested here 13.

2. Language
The model will be trained in Spanish (regardless of variety).

3. Model
RoBERTa-large

4. Datasets
Spanish portion of mC4 of about 1TB of uncompressed data.

5. Training scripts
There are already Flax scripts to pre-train RoBERTa that we can easily use:
https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)

6. Challenges
It is too much data. We need a way to reduce the amount of data to finish on time. Options:

Random sampling.
Perplexity sampling using percentiles and a Spanish language model. One option here is to use a 5-gram Kneser-Ney model as implemented in the KenLM library (Heafield, 2011) and released by Facebook.

7. Desired project outcome
A Spanish monolingual well performing model on the usual benchmarks.

8. Reads
- https://arxiv.org/pdf/1907.11692.pdf
- https://arxiv.org/pdf/1911.00359.pdf
- https://www.aclweb.org/anthology/W11-2123.pdf

Context: So interesting! I am in!


Question:
Image captioning for Spanish with pre-trained vision and text model
For this project, a pre-trained image model like ViT can be used as an encoder, and a pre-trained text model like BERT and/or GPT2 can be used as a decoder.

Model
Pre-trained ViT 16, BERT 6 models can be found on the model hub. We could use multi-lingual BERT/ROERTa models for the Spanish language.

Datasets
The WIT 14 dataset can be used for this task. It has almost over 1M image-text pairs for Spanish.

Available training scripts
As this will be a Seq2Seq model, the run_summarization_flax.py 23 script can be used for training this model with some modifications.

(Optional) Desired project outcome
The desired outcome is to see if pre-trained vision and text models can be leveraged for image captioning and also train captioning models in for the Spanish language.  This can be showcased with a streamlit  or gradio app.

(Optional) Challenges
This model will require some modifications to the existing text models. Specifically, as this will be a seq2seq model, we’ll need to add a randomly initialized cross-attention layer in BERT or GPT2 to use it as a decoder in the encoder-decoder setting.

(Optional) Links to read upon
This keras example presents an excellent example of how image encoder and transformer can be used for image captioning.

      keras.io

Keras documentation: Image Captioning 37

Context:
Hi,
I would like to join this project. I am not experienced enough with NLP to create a project on my own. But I do have experience with Computer Vision overall, and would like to try and see how well spanish text is generated for the image-captioning.
I am David and I am in  the Amsterdam timezone(CET). I have some experience with other frameworks, and training/evaluating object detection networks, but not with NLP. I can contribute by putting in my previous knowledge and discipline towards achieving this goal


Question:
Korean-GPT
There are already published Korean GPT2 models, but these models are limited in their use in the industry due to license restrictions. Recently, a large number of South Korean public natural language datasets such as KLUE dataset, AI-Hub dataset, and Modu Corpus have been additionally released. My goal is to train a Korean GPT2 model that anyone can use for any purpose, using only large publicly accessible datasets, including recently added datasets.

2. Language
The model will be trained in Korean

3. Model
GPT2

4. Datasets
KLUE Dataset
KorQuAD Dataset
Modu Corpus(모두의 말뭉치)
Korean Text Datasets from AI-hub
+ Maybe extra publicly accessible Korean datasets…

https://klue-benchmark.com
https://korquad.github.io
https://corpus.korean.go.kr 1
https://aihub.or.kr 1

5. Training scripts
A causal language modeling script for Flax is available here  7

6. Challenges
It will take time and effort to preprocess each of the various public Korean datasets in the most helpful form for pre-training.

Context: I think this will be an amazing challenge!  Wish I can join this project!


Question:
Scandinavian RoBERTa
Rather than training separate models for Swedish, Norwegian, Danish and Icelandic [1][2][3], we could probably produce a better model by pretraining a Scandinavian model and then finetuning to one of the four, considering how similar the four languages are (in its written form that is, cough cough Danish).
We can train a RoBERTa-large model on the combined mC4 dataset 5, containing 386 GB uncompressed text (179 GB Swedish, 107 GB Danish and 100 GB Norwegian). Furthermore, there are gigaword datasets in both Swedish, Danish and Icelandic that we could use. As suggested in [1], we could start training the model with a sequence length of 128, then 256 and lastly 512.

2. Language
The model will be trained in Swedish, Danish, Norwegian and Icelandic.

3. Model
RoBERTa-large

4. Datasets

Swedish:

mC4 (179 GB)
Gigaword [4] (~9 GB compressed)

Danish

mC4 (107 GB)
Gigaword [5] (~2 GB compressed)

Norwegian

mC4 (100 GB)

Icelandic

mC4 (9 GB)
Gigaword [6] (~14 GB compressed)

5. Training scripts
There are already Flax scripts to pre-train RoBERTa that we can easily use:
https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling 14

6. Challenges

Will the data be enough to be able to train a good model?
Will all the languages be well-represented?

7. Desired project outcome
A Scandinavian language model which performs well on the usual benchmarks, on each of the four languages.

8. Reads
[1] https://discuss.huggingface.co/t/pretrain-roberta-large-from-scratch-in-swedish
[2] https://discuss.huggingface.co/t/pretrain-gpt2-from-scratch-in-swedish
[3] https://discuss.huggingface.co/t/pretrain-roberta-from-scratch-in-norwegian
[4] The Swedish Culturomics Gigaword Corpus | Språkbanken Text 3
[5] https://gigaword.dk/ 2
[6] Icelandic Gigaword Corpus

Context: This makes sense to me.


Question:
GPT2 for Tamil
We aim to create a GPT2 model for tamil language.

Model
A randomly initialized GPT2 model

Datasets
OSCAR dataset (https://oscar-corpus.com/ 3) has about 9 GB of training data and Indic-NLP (IndicCorp | AI4Bharat IndicNLP 5) has 500 million token dataset.

Available training scripts
A causal language modeling script for Flax is available [here ]It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Tamil language.

Context: @rays2pix : Hi Deepak. I was hoping that someone would propose a project for Tamil. I would like to collaborate on this!


Question:
GPT2 for Swedish
Currently, there is no GPT2 model that was trained from scratch for Swedish on the hub: Hugging Face – The AI community building the future. 1 . For this project, the goal is to create a strong language generation model for Swedish.

Model
A randomly initialized GPT2 model

Datasets
One can make use OSCAR 4 the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face 1.

Available training scripts
A causal language modeling script for Flax is available here 3. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Spanish language. A nice generation demo can be created for this.

(Optional) Challenges
It might be possible that there is not enough data for the model to perform reasonably well on text generation. In this case, one would have to look at other datasets as well, like mc4.

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory 8

Context:
patrickvonplaten:

Google Colaboratory

This would be awesome!


Pre-train ALBERT from scratch for Persian/Farsi language

Question:
1. Pre-train Albert from Scratch for the Persian Language
Currently, there are some open-source language models for the Farsi language. We Crawl some new samples as a new dataset which I want to use for pre-training Albert for the farsi language from scratch.

2. Language
Persian/Farsi

3. Model
ALBERT xLarge

4. Datasets
Wikipedia dump
Common Crawl dump
random web scraps

5. Training scripts
There are already Flax scripts to pre-train Albert that we can easily use:
transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHub 27

6. Challenges
Using Dataset pipeline (ETL approach) for new txt crawled file

Context: Sounds amazing, I’m also interested in participating.


[Open-to-the-community] Community week using JAX/Flax for NLP & CV

Question:
Learn how to use JAX/Flax with Transformers  +

We partnered-up with Google’s Flax, JAX, and Cloud teams to organize a new community week from July 7th to July 14th. We want to teach you how to effectively use JAX/Flax for Natural Language Processing (NLP) and Computer Vision (CV).
Free access to a TPUv3-8 VM will kindly be provided by the Google Cloud team !
We can guarantee TPU access for the first 400 participants, so it might be worth to sign-up quickly .
TLDR; All important announcements will be made in an internal Slack channel. To get access to the Slack, you can sign-up here 751. Important information is summarized here 426  .

What it is about

The goal of the JAX/Flax community week is to make compute-intensive NLP and CV projects (like pre-training BERT, GPT2, CLIP, ViT) practicable for a wider audience of engineers and researchers.
To do so, we will teach participants how to effectively use JAX/Flax on TPU and help them define a fun project to complete during the community week.

How does it work

Participants can propose ideas for an interesting NLP and/or CV project. Teams of 2 to 5 will then be formed around the most promising and interesting projects.
Make sure to read this 276 document on how to propose projects, comment on other participants’ project ideas, and create a team.
To help each team to successfully finish their project, we have organized talks by leading scientists and engineers from Google, Hugging Face, and the open-source NLP & CV community. The talks will take place before the community week from June 30th to July 2nd. Make sure to attend the talks to get the most out of your participation!
Each team is then given free access to a TPUv3-8 VM from July 7th to July 14th. In addition, we will provide training examples in JAX/Flax for a variety of NLP and Vision models to kick-start your project. During the week, we’ll make sure to answer any questions you might have about JAX/Flax and Transformers. We will try to help each team as much as possible to successfully complete their project!
At the end of the community week, each team can submit a demo of their project. All demonstrations will be evaluated by a jury and the top-3 demos will be awarded a prize! In a few days, we will share a document that gives a couple of propositions on how to submit your project with a nice demo.

What do I need to do to participate

An interest in JAX/Flax and ideally an idea for a fun NLP/CV project! Compute in the form of a TPUv3-8 will kindly be provided by Google’s Cloud team. To sign up for the event, please use this 751 Google form to fill in your name, email address as well as your Hugging Face account name. You will then be added to an internal Slack channel in which all the relevant information will be given.
An overview of the event is shown here 426.

What do I get

enjoy a bit of Hugging Face vibe by joining the JAX/Flax community week.

learn about JAX/Flax and how to effectively use it for compute-intensive training.

interesting presentations by leading researchers and engineers in NLP & CV.

a prize if you are under the top-3 projects!

Summary - Timeline

23.06. - Official announcement of the community week. Make sure to sign-up in this 751 Google form.
23.06. - 30.06. Participants will be added to an internal Slack channel. Project ideas can be proposed here and groups of 2-10 are formed. Read this 276 document for more information.
30.06. - Release of all relevant training scripts in JAX/Flax as well as other documents on how to set up a TPU, how to use the training scripts, how to submit a demo, tips & tricks for JAX/Flax, tips & tricks for efficient use of the  hub.
30.06. - 2.07. Talks about JAX/Flax, TPU, Transformers, Computer Vision & NLP will be held.
7.07. Start of the community week! Access to TPUv3-8 will be given to each team.
7.07. - 14.07. - The Hugging Face & JAX/Flax & Cloud team will be available for any questions, problems the teams might run into.
15.07. Access to TPU is deactivated and community week officially ends.
16.07. Deadline for each team to submit a demo.
Open-sourcely yours,
The , , Flax and Cloud team

Context: Really cool Patrick! @thomasdehaene might be interested to pre-train on Dutch data.


MedClip - Pretraining CLIP on medical data

Question:
MedClip
Advancements in computer vision and deep learning techniques carry the potential to make significant contributions to healthcare. The current state-of-the-art models for automated diagnosis and outcome prediction using medical imaging tend not to consider additional information such as medical reports.
A multimodal model like CLIP pre-trained in medical data could allow new medical applications that combine text and image.

Model
Pre-trained ViT 3 and SciBERT 6 models can be found on the model hub.

Dataset
The MIMIC-CXR 11 dataset can be used for this task. For privacy reasons, the dataset in question has restricted access. Anyone who wants to participate in this project must obtain the necessary credentials to access the dataset.
In my experience, getting access to MIMIC-CXR is not particularly complicated, it’s necessary to accept the terms of the license and take a short course on medical data management. It normally takes ~2 weeks to get such credentials.

Available training scripts
A training script for this will be provided soon. (see PR  17)

Challenges
Carrying out a proper evaluation of the model may be difficult

Context: I would like to join this project!


A Text2Text model for semantic generation of building layouts

Question:
The goal of the project would be to fine tune GPT-Neo J 6b on the task of semantic design generation.
The model will learn to transform natural language prompts into geometric descriptions of designs. It will then be deployed in a simple interface that allows for users to immediately see their generated design, creating a semantic generation tool out of Neo.
I already have a dataset of paired annotations-design pairs and have started experimentations with the smaller Neo models as we speak.
Let me know if you’re interested

Context:
That’s a cool idea !
Could you post an example of language prompt => geometric descriptions => generated design ? Thanks.


Question:
StyleGAN2 for medical datasets
In this project, we would train a StyleGAN2 model for medical datasets. This could be beneficial for synthetic data augmentation, or potentially encoding into and studying the latent space could be useful for other medical applications.

Datasets
Personally, I am more interested in histopathological datasets:

BreCaHAD 7
PANDA 6

TCGA 1 (needs more data wrangling though)
PatchCamelyon 5

However, I don’t mind looking into other medical datasets and training a StyleGAN2 for them:

NIH Chest X-rays 3

Training scripts
This repository 13 has a JAX model implementation of StyleGAN2 but unfortunately not a training script.

Desired project outcome
At the end of the project we would hopefully have:

A JAX-based pipeline for training StyleGAN2 models on arbitrary datasets and for exploring the latent space of these models.
A trained StyleGAN2 model for the selected datasets (with an online demo)

Let me know if you find this project interesting and if you have any related ideas…

Context:
Hey @tmabraham, super interested in being part of this project. I’m working in the intersection on Deep Learning and Medical imaging and have worked on histopathological datasets.
Really looking forward to train StyleGAN2 using JAX and see how the latent space on medical imaging pans out for other downstream tasks.


BigBirDNA - Pretraining BigBird on DNA sequences

Question:
BigBirDNA
Pretraining BigBird on DNA sequences. This provides a base model for downstream DNA sequence analysis tasks

2. Language
The model will be trained in DNA

3. Model
BigBird.

4. Datasets
All the available DNA sequences.
Possible links to publicly available datasets include:

www.ncbi.nlm.nih.gov/genbank/ 5
Others can be found on www.insdc.org 2

5. Training scripts

There is a Flax BigBird implementation at transformers/modeling_flax_big_bird.py at master · huggingface/transformers · GitHub 11

There is transformers/run_mlm_flax.py at master · huggingface/transformers · GitHub 3 for word masking which we may be able to tweak for bigbird
There is GitHub - jerryji1993/DNABERT: DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome 9
that can be used to copy/tweak downstream tasks

6. (Optional) Challenges

There will be quite a bit of preprocessing involved
There will be quite a lot of data involved (about half a TB compressed)
Besides NLP skills there may be some bioinformatics skills required

7. (Optional) Desired project outcome

Pretrain a transformer that can emulate or improve on the downstream tasks mentioned reads below

8. (Optional) Reads
The following links can be useful to better understand the project and
what has previously been done.

https://arxiv.org/pdf/2007.14062.pdf 14
https://www.biorxiv.org/content/10.1101/2020.09.17.301879v1.full 10

Context:
kees:

Pretraining RoBERTA on DNA sequences. This provides a base model for downstream DNA sequence analysis tasks

Sounds interesting @kees, I was infact working on something similar → a jax/flax implementation of ProteinBERT 4. Pretraining RoBERTa seems like a good strategy. I’m definitely interested.


Question:
Pre-training GPT2 for Persian:
Description
We want to pre-train GPT-2 from scratch, as stated in the OpenAi paper, with high-quality data that will suit text generation task in the Persian language. There is 2 version of GPT-2 which are somehow small model, but we want to take a further step and train GPT-2 on a more extensive data-set and with more diverse and quality text.
Data-set
As part of the research effort,  we are preparing a Persian MEgA data-set, including 50~70 GB of text, which is still in progress. We may also use the Oscar data set, which is already available on Huggingface’s data sets.
Challenges
Besides creating the data-set, the main challenge is to choose a tokenizer that suits our purpose in the Persian language and doesn’t mess up with the literature. We are still doing various research in this field, but up until now, our pick is the BPE tokenizer which is GPT’s default tokenizer.
Training scripts
We will use some ideas like (here 7) and here 6 and also do some modification to suits our purpose.
Insight
Besides generating high-quality text, with a big GPT model as a reference, we can fine-tune it on several tasks like chatbots(the data for this task will publish soon by our team).

Context:
@patrickvonplaten
First of all, thank you and Huggingface so much for this opportunity.
As I mentioned, we already prepare some data, and the rest is still in progress. I was wondering is it possible to add additional data to the OSCAR data-set and what is your insight on doing such a thing.
Thanks again


PreTrain RoBERTa from scratch in Marathi

Question:
RoBERTa/BERT for Marathi
Currently, there are no BERT-like models for Marathi on the hub. For this project, the goal is to create a RoBERTa/BERT model for just the Marathi language.

Model
A randomly initialized RoBERTa/BERT model

Datasets

      GitHub

The C4 Multilingual Dataset · Discussion #5265 · allenai/allennlp 1

  The wait has been long, but we are finally able to release the C4 multilingual dataset! We now have almost 27TB of clean-ish data, in 101 different languages (plus the "undetected" langua...

One can make use A colossal, cleaned version of Common Crawl’s web crawl corpus - Marathi  1  dataset. Marathi corpus is ~70GB so should be great start for training a new language model.

Available training scripts
A masked language modeling script for Flax is available - huggingface/transformers repo: examples/flax/language-modeling/run_mlm_flax.py. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a strong RoBERTa/BERT model in Marathi.

(Optional) Links to read upon
The most important read would be the following colab:

Download C4 dataset - allenai/allennlp/discussions/5056

Context:
We are pre-training RoBERTa for Hindi… maybe I can help…
How are we gonna evaluate this? Any benchmark datasets available for Marathi?


Question:
Train T5 from scratch in Italian on mC4
Currently the Italian Hub 4 has only autoencoding (e.g. bert-base-italian  1,  BERTino) and autoregressive (e.g. GePpeTto 1) language models. For this project, the goal is to create a strong Text2TextGeneration model for seq2seq tasks in Italian.

Model
We’ll be using a random T5 3 model with a configuration similar to the one of t5-base, since the Flax implementation was recently made available on the Hub.

Datasets
Recently the multilingual C4 dataset 1 was made available, and it surpasses by a large margin the size of the OSCAR corpus (590GB cleaned vs. 69GB). Given the new streaming capabilities for large datasets this should be easily feasible. Another option is to use multiple large datasets (mC4, OSCAR, Italian Wikipedia) with ìnterleave_datasets to mix them.

Training scripts
We can leverage the T5 pretraining script that is currently being prepared 8 to train the model, and run_summarization_flax.py 4 to test downstream performances, e.g. on SQUAD-it converted in text2text format.

Challenges
The project involves a pre-training of a large LM, so it will inevitably take some time to reach competitive performances. Most scripts and implementations are very new, and as such it is possible that we’ll incur in some issues to be fixed.

Desired project outcome
The main project outcome, besides producing a trained T5 model for Italian, is to have fun and get our hands dirty with some ! We could optionally test the validity of our results at a later stage by fine-tuning the model on some downstream tasks.

Reads

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer 2

Context: Think this is a cool project - let’s see if we get some more participants


Question:
There are currently no Dutch seq2seq models on the hub. There exist multilingual seq2seq models such as mbart and mt5, but no Dutch (nl) only models. This project aims to pre-train T5 from scratch using the Dutch segment of mC4.

Language
Dutch

Model
Randomly initialized T5 model

Datasets
Our primary dataset for pre-training is the Dutch part of multilingual C4, and separately downloaded and curated Dutch news site documents from Common Crawl. We re-use these datasets that have been already cleaned for the project in the FlaxJax week that trains BigBird from scratch on Dutch. The cleaning uses an adapted version of the clean script used for the english c4, that filters badwords and content that are not sentences. Details and source of the data cleaning will be provided on the model hub for this project.
The dataset for fine-tuning will be CNN and XSUM translated to Dutch, and news summaries from the Dutch nu.nl website downloaded from CommonCrawl.

Training script
The starting point for the training will be transformers/run_t5_mlm_flax.py at master · huggingface/transformers · GitHub 62
This will result in a model that needs to be fine-tuned on a downstream task.
For the demo we will fine-tune the model to perform summarization of news articles in Dutch using transformers/run_summarization_flax.py at 7d6285a921a23c06169e2d90c94faa0d92d00d78 · huggingface/transformers · GitHub 9

Challenges
Adapting the training script for our use case and custom datafiles.

Desired project outcome

A T5 model that can be finetuned for Dutch seq2seq tasks
A T5 model that can summarize news articles in the Dutch language
A streamlit app that demonstrates news article summarization

Context: Same team!


Question:
Pre-train RoBERTa from Scratch for Georgian Language
Currently, there are no open-source language models for the Georgian language. I have not so large dataset which I want to use for pre-training RoBERTa for the Georgian language from scratch.

2. Language
The model will be trained in Georgian Language

3. Model
RoBERTa

4. Datasets
wikipedia dump
Common Crawl dump
random web scraps

5. Training scripts
There are already Flax scripts to pre-train RoBERTa that we can easily use:
transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHub 23)

Context: cool defined it!


Question:
Pretrain T5 for Chinese
Currently there is No T5 pretrained model for Chinese language on the Huggingface hub. The goal is to train a T5-base model in Chinese and finetune it on Chinese QA dataset CMRC2018 to see its performance compare to other non-T5 model.
Model
A randomly initialized T5 model
Datasets
CLUECorpus2020
OSCAR
mC4
CMRC2018 (CMRC 2018)
Available training scripts
Will be using standard T5 training script from Huggingface
(Optional) Desired project outcome
A ChineseT5 model that is competitive with other similar size/speed non-T5 model (Roberta, GPTs etc） on Chinese QA dataset
(Optional) Challenges
Chinese character is non-spaced, might need special treatment compare to spaced language like English.

Context: sounds good - let’s define it!


Question:
GPT2 for Sinhala
Currently, there is no GPT2 model that was trained from scratch for Sinhala on the hub: Hugging Face – The AI community building the future. 1 For this project, the goal is to create a good language generation model for Sinhala.

Model
A randomly initialized GPT2 model

Datasets
OSCAR has ~850 MB dataset and it is also available through the datasets library here: oscar · Datasets at Hugging Face.
C4M by AllenAI has a 3GB dataset which could be another potential dataset.

Available training scripts
A causal language modeling script for Flax is available here. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Sinhala language. A nice generation demo with streamlit can be created for this.

(Optional) Challenges
The dataset on OSCAR is comparably small: ~ 850MB. There is an option to use the C4M dataset to supplement that.

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory  4

Context: Let’s give it a try


ALBERTI, multilingual model for poetry tasks

Question:
ALBERTI
The idea is to do further training from multilingual BERT on a multilingual corpus of stanza and verse split poetry. The goal is to perform better than monolingual models on tasks such a metrical pattern prediction, rhyme and rhythm identification, rhetorical figure detection, and stanza type classification.

2. Language
This will be a multilingual model using Spanish, Italian, French, German, Czech, Hungarian, and English.

3. Model
We can use mBERT or XLM-R as the backbone model. Ideally, given the nature of the downstream tasks, it would be ideal to have ALBERT and its sentence ordering objective.

4. Datasets
We might need to convert corpora from Averell 1 to datasets, which amounts to 25M words. There’s also a Spanish corpus called poesi.as that contains 10M words as well, a German corpus with 4M words, and a Portuguese one with 1M words. In total would be around 40M words.

5. Training scripts
There are already Flax scripts to pre-train BERT and RoBERTa that we can easily use: transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHub 5

6. Challenges
It is too little data and very short sequences. Hopefully, by using pre-existing weights we can overcome both issues. The fact that sequence lengths are generally short should make training faster as only one phase of 128 (or even 64) might be needed

7. Desired project outcome
A multilingual well performing model on the poetry downstream tasks

8. Reads
- https://research.tilburguniversity.edu/en/publications/a-hybrid-approach-to-stanza-classification-in-spanish-poetry
- https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24532

Context: This is really interesting to me. Not sure how to solve the size issue of if it’s even possible. But definitively worth exploring.


Train a GPT2 model for contextual common sense reasoning using the COSMOS QA dataset

Question:
Idea:-A unique task wherein we train a GPT-2 model for contextual common sense reasoning using the COSMOS QA dataset.The goal is the test the GPT-2 for the common sense reasoning case.
Model:-Need to add support for ‘FlaxGPT2ForMultipleChoice’ model which can be easily done using the existing ‘FlaxGPT2Model’.
Dataset:-cosmos_qa · Datasets at Hugging Face 5 available at huggingface.
Available training scripts:-Need to prepare a script for multiple choice training for our Flax Model.

Sample636×782 264 KB

We need some guidance for the project,we would love to hear feedback(@patrickvonplaten and @valhalla please do have a look) and we are very excited to contribute this task to huggingface.
We are currently a team of two:-1.Rohan.V.Kashyap(@Rohan)
2.Vivek.V.Kashyap(@Vivek)

Context: I’m in.Really excited for this upcoming project @Rohan


Question:
We’re planning to use ViT encoder, mBART decoder and train them end-to-end for image captioning in different languages.

Model
Pre-trained ViT 2, mBART 3 (will be merged soon) can be leveraged for our task.

Datasets
Currently we’re thinking of using WIT 4. COCO dataset can also be used but it has caption only in English language. Dataset can be changed later if we come across more suitable dataset for this task.

Available training scripts
Since this is a Seq2Seq model, the run_summarization_flax.py 7 script can be used for training this model with some modifications.

(Optional) Desired project outcome
Our end use case would be to run this model on a video clip or movie and make it like an accessibility tool for visually challenged people. This can be showcased with a streamlit or gradio app.
We’re also planning to benchmark its zero-shot performance on VizWiz 4

(Optional) Challenges
-Training encoder-decoder model end-to-end in FLAX/JAX will require some effort
-Data Processing incase new datasets are available only in English

Context:
I am in
@abheesht @Vaibhavbrkn @nilshoehing have also shown interest in this sheet: JAX/Flax Vision-and-Language Projects - Google Sheets 17
Others are welcome to join!


Question:
Covid19 Related Question Answering (Closed book question answering)
In 2020, COVID-19 which is caused by a coronavirus called SARS-CoV-2 took over the world. It touched the lives of many people and caused a lot of hardship for humanity. There are still many questions in regards to COVID-19 and it is often difficult to get the right answers. The aim of this project is to finetune models for closed book question answering. In closed-book QA, we feed the model a question without any context or access to external knowledge and train it to predict the answer. Since the model doesn’t receive any context, the primary way it can learn to answer these questions is based on the “knowledge” it obtained during pre-training [1]  [2].
The main goals of this project are:

Train a model for question answering in regards to COVID-19
Release the top performing models for further research and enhancement
Release all of the preprocessing and postprocessing scripts and findings for future research.

2. Language
The model will be trained in English and python language.

3. Model
Possible model candidate model is pretrained T5-large 2 or Bert variants (e.g. BioBert orBioClinicalBert) with a sequence generation head.

4. Datasets
The following datasets would be used for finetuning the model. Note that the last dataset is optional and the model is evaluated only using Covid-QA.

Covid-QA 4
CDC-QA 5
Optional - Trivia-QA 2

5. Training scripts
We can make use of :

For preprocessing and mixing datasets 3
For T5 training 2

8. (Optional) Reads
The following links can be useful to better understand the project and
what has previously been done.

How Much Knowledge Can You Pack Into the Parameters of a Language Model? 3

Context: That’s a super nice & in-detail description! I’d really like this project to take place  Are there other ways we can promote this project?


Unsupervised Code-Code Translation based on TransCoder

Question:
The Goal of this project is to train an Unsupervised Code-Code Translation Model. There are a lot of different approaches which modeled Supervised with parallel corpora. There is this particular paper([2006.03511] Unsupervised Translation of Programming Languages), which tried to build a Code-Code Translation with no parallel corpora, between languages. This would be a great addition to the Hub too. Let me know if you’re interested.
Code - GitHub - facebookresearch/TransCoder: Public release of the TransCoder research project https://arxiv.org/pdf/2006.03511.pdf 4

image853×369 59.8 KB

Context:
I’m interested – would the pretraining script on Facebook’s GitHub be enough for reference for us to implement? Is it doable in 7 days?
EDIT: Although I am interested - I may not be able to join/contribute much don’t want to over-commit


Fine-Tuning a T5 model for sentence splitting in English

Question:
Fine-Tuning a Seq2Seq model for sentence splitting in English.
Sentence fusion is the task of dividing a long sentence into multiple sentences. E.g.: the sentence:
Mary likes to play football in her freetime whenever she meets with her friends that are very nice people.

could be split into
Mary likes to play football in her freetime whenever she meets with her friends. Her friends are very nice people.

Currently there is only one model on the hub for sentence splitting:

      huggingface.co

google/roberta2roberta_L-24_wikisplit · Hugging Face 16

The goal of this project is to have the best sentence splitting model for English on the hub.

Model
One use one or multiple of the pretrained T5 models:

google/byt5-base · Hugging Face 3
google/t5-v1_1-base · Hugging Face 1
t5-base · Hugging Face 3

Datasets
The Wikisplit dataset can be used:  WikiSplit – Google Research 6

Available training scripts
As this will be a Seq2Seq model, the run_summarization_flax.py 5 script can be used for training.

(Optional) Desired project outcome
The desired outcome is to have a sentence splitting model for the English language. This can be showcased directly on the hub or with a streamlit  or gradio app.

(Optional) Challenges
Preprocessing the dataset can be challenging, but should be feasible. It might even be a nice side-project to add the Wikisplit dataset: WikiSplit – Google Research 6 to the datasets library while training the model.
ALso it could be difficult to beat the existing model: google/roberta2roberta_L-24_wikisplit · Hugging Face 16

(Optional) Links to read upon

[1907.12461] Leveraging Pre-trained Checkpoints for Sequence Generation Tasks 1
google/roberta2roberta_L-24_wikisplit · Hugging Face 16

Context: Interested to be part of this project


Pretrain RoBERTa-large from scratch in Swedish

Question:
Roberta Swedish
The project idea is identical to the one for Pretraining Roberta in Spanish but instead using the Swedish dataset.
The idea is to use the Swedish portion of mC4  1 (which roughly amounts for 100GB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512. It might be a good idea to start in 128, as suggested here  3.

2. Language
The model will be trained in Swedish.

3. Model
RoBERTa-large

4. Datasets
Swedish portion of mC4 of about 100gb of uncompressed data.

5. Training scripts
There are already Flax scripts to pre-train RoBERTa that we can easily use:
https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)

6. Challenges
Will the data be enough to be able to train a good model?

7. Desired project outcome
A Swedish monolingual well performing model on the usual benchmarks.

8. Reads
- https://arxiv.org/pdf/1907.11692.pdf
- https://arxiv.org/pdf/1911.00359.pdf
- https://www.aclweb.org/anthology/W11-2123.pdf

Context: Just a suggestion: How about we pretrain a common Scandinavian model instead, as our languages are so similar? I’ve made a project suggestion here: Scandinavian RoBERTa 1.


Question:
GPT2 for English Rap
For this project, I propose to use a pretrained GPT2 model in English and fine-tune it to learn to English rap.

Model
A GPT2 model, pretrained in English can be found here: gpt2 · Hugging Face 4

Datasets
The model can be fine-tuned on a variety of publicly available lyrics of rap songs. The team can choose whatever lyrics they like.

Available training scripts
A training script to fine-tune a GPT2 model in Flax is available here 13

(Optional) Desired project outcome
The desired project output is a GPT2 model that can drop bars 8. Ideally the model is able to generate a whole rap song.

(Optional) Challenges
It might be difficult to find enough data and to homogenize it well.

(Optional) Links to read upon
There are some nice articles on GPT2 models that can rap:

GitHub - ewenme/gpt-2-raps: generating lyrics with GPT-2 9
Create a Rapping AI using deep learning | by Max Leander | Analytics Vidhya | Medium 6

Context: This is very interesting! I would like to work on it.


Question:
Fine-tuning BERT-based language model to overcome gender-bias
Transformer-based language models such as BERT, and GPT-x has pushed the state-of-the-art on many tasks including language modeling. However, thorough examination [1] of these models has shown their biases toward specific genders, events, etc. For instance, in the sentence “David is driving a car”, almost even attention is paid to different parts. Nonetheless, this does not hold for “Mary is washing the dishes”, in which model is considerably focused on the name “Mary”. We think that by refining the loss function and modifying the training schedule, this artifact might be resolved. We expect a nearly bias-free model to output the same probability for plausible candidates of a masked token.
[1] Do Neural Language Models Overcome Reporting Bias? (https://aclanthology.org/2020.coling-main.605.pdf 8)

2. Language
The model will be trained in English.

3. Model
We intend to use the bert-base-uncased model.

4. Datasets
We aim to use common datasets used for pre-training of BERT, BookCorpus, and English Wikipedia.
Possible links to publicly available datasets include:

<BookCorpus: https://github.com/soskek/bookcorpust>
the link to the Wikipedia dataset is omitted due to new user’s limitations.

5. Training scripts
We had a similar experience of fine-tuning a BERT model for the task of sentiment analysis and language modeling, and we hope by small modifications to our previous codes training for the new objective will be possible.

6. Challenges
There are two central challenges in this project: First, we need to devise a decent loss function for the training to effectively minimize the bias. Second, if redefinition of the bias did not help, we need to train the model from scratch with newly added tokens, plus the new loss function.

7. Desired project outcome
We want to test the model on the task of masked language modeling and hope to see bias-free predictions for masked tokens.

Context:
you have addressed an interesting subject in which I’m utterly enthusiastic to participate.
as you mentioned earlier, it is good to add a new token for genders to tackle the reported bias.


Question:
How to get current available memory in TPU, like nvidia-smi? I tried to use bfloat in TPU, but doesn’t seem to work. Is there ways to get TPU memory info from other process? (tensorboard didn’t work, not sure if I did something wrong, and pprof looks like it can only profile usage of the process itself. I am getting paranoid and thinking there might be TPU memory used by dead process.

I decided to use GPU to debug and get an idea of how much memory is used. I tried adafactor, and float16, and the memory saving seem small. (I used XLA_PYTHON_CLIENT_ALLOCATOR=platform,  XLA_PYTHON_CLIENT_PREALLOCATE=false so I can see the actual usage.)

How does float16/bfloat16 works? I printed the params and it’s float32, is it expected behavior? Here is the code:

import jax.numpy as jnp
from transformers import FlaxT5ForConditionalGeneration, AutoConfig
import jax

config = AutoConfig.from_pretrained('t5-base')
model = FlaxT5ForConditionalGeneration(config, seed=0, dtype=jnp.float16)
print(jax.tree_map(lambda x: x.dtype, model.params))

output: (trimmed)
{'decoder': {'block': {'0': {'layer': {'0': {'SelfAttention': {'k': {'kernel': dtype('float32')},
       'o': {'kernel': dtype('float32')},
       'q': {'kernel': dtype('float32')},
       'relative_attention_bias': {'embedding': dtype('float32')},

another float16/bfloat16 question: In the google t5x (flax t5) code 1, there is this line:

    if use_bfloat16:
      grad = jax.tree_map(lambda x: x.astype(jnp.bfloat16), grad)

But huggingface example doesn’t have this line, which one is “correct”? Is there a difference?

another question is related to training speed: I am training t5-base, with per_device_batch_size=16, which means batch_size=16*8=128 and max_seq_len=512 (same as t5 paper) and t5 paper trains 524,288 steps, my progress bar says it would take 8 days for that many steps, is this good?

Context:
A quick note on reduced precision training:
Params are typically still stored in float32. But some intermediate computations can be done in bfloat16. Most standard Flax modules have a dtype  attribute, that you can set to bfloat16 to compute the safe parts in reduced precision.


[SOLVED] Cloud VM Access Problem: WARNING: You do not appear to have access to project [hf-flax] or it does not exist

Question:
I am trying to log into the cloud VM assigned to me.
But I am facing the problem:

$ gcloud config set project hf-flax
Updated property [core/project].
WARNING: You do not appear to have access to project [hf-flax] or it does not exist.

Before trying to do this, I have followed the steps to install Google Cloud SDK from here 1.
I did not start GCoud to get started by gcloud init. Because it tells me to enter a project name. Should I do this?
After following steps 1 through 4 there, I turn back to Hugging Face’s guide 5.
I have set the account, and I have logged in through the browser. Now, I cannot set the project and face the error message that I have mentioned.

Now, when I go ahead anyway with the steps, and run
$ gcloud alpha compute tpus tpu-vm ssh <name> --zone us-central1-a --project hf-flax
# I do replace the <name> placeholder with the name supplied to me

I see this message-

WARNING: The private SSH key file for gcloud does not exist.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):

As suggested here 5, I clicked on the “Open Cloud Console” button, and I see this, as expected:

You don’t have sufficient permission to view this page

But, before all this, what I didn’t see (and still don’t see) is a pop-up as mentioned there 5-

“You should click on “Open Cloud Console” on the first mail and agree to the pop up windows that follows”

There was no pop-up window after I see the permission error. I have tried after disabling Privacy extensions on Chrome as they sometimes block pop-ups. But no luck.
This is what I see-

image874×421 39.5 KB

This is the same before and after I run the setup steps.

What should I do?

Context:
I went ahead with this prompt-

 ghosh-r:

Enter passphrase (empty for no passphrase):

Let’s say I typed in “foobarbaz”.
It is successful, and it shows-

Your identification has been saved in /home/<user>/.ssh/google_compute_engine
Your public key has been saved in /home/<user>/.ssh/google_compute_engine.pub
The key fingerprint is:
SHA256: <sha256 code> <user>@<computer-name>
The key’s randomart image is:
±–[RSA 3072]----+
|                     |
|                     |
|                     |
|                     |
|                     |
|                     |
|                     |
|                     |
|                     |
±—[SHA256]-----+

(I removed the characters)
This time, and whenever I run
$ gcloud alpha compute tpus tpu-vm ssh <name> --zone us-central1-a --project hf-flax

I get-

SSH: Attempting to connect to worker 0…
sign_and_send_pubkey: signing failed for RSA “/home/<user>/.ssh/google_compute_engine” from agent: agent refused operation
<user>@x.x.x.x: Permission denied (publickey).
Retrying: SSH command error: [/usr/bin/ssh] exited with return code [255].
SSH: Attempting to connect to worker 0…
sign_and_send_pubkey: signing failed for RSA “/home/<user>/.ssh/google_compute_engine” from agent: agent refused operation
<user>@x.x.x.x: Permission denied (publickey).
Retrying: SSH command error: [/usr/bin/ssh] exited with return code [255].
^C
Command killed by keyboard interrupt

Then I get asked to enter my password repeatedly. I type in the password, it gets accepted, then another prompt appears.
It says- “Enter password to unlock the private key, an application wants to access the private key “<user>@<computer>”, but it is locked.”

$ cat /etc/apt/sources.list.d/google-cloud-sdk.list

returns-

deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main


Pretraining RoBERTa (or another ‘flavor’) on permuted word orders

Question:
For the Community Week project I’d like to extend the work of the McGill researchers who, by inducing ‘order agnostic’ tokens via a ‘permuted’ pre-trained version of BERT, get high task scores on a wide range of benchmark tasks (GLUE, PAWS, etc.) due to the model learning to represent distributional priors rather than its ability to “discover the NLP pipeline”.
I’d like to replicate the authors’ work and perhaps extend it by looking at the ‘permuted model’s performance’ on other benchmarks / sets of NLP and ‘NLU’ tasks for which we have human-curated ‘gold standard’ performance measures.
To start, the authors pre-trained models on various permuted corpora that preserve sentence-level distributional information by randomly shuffling n-grams within the sentence where n is [1,4]. While the authors evaluated the ‘permuted’ models’ performance against ‘normally-trained’ Transformer models in a wide range of settings - GLUE, etc. - I’d be interested to see how the models perform against some of the human-curated benchmark datasets such as PAWS.

Context:
Very cool project! I think we should try to keep the scope as small as possible → so maybe:

PreTrain RoBERTa on permuted words
Evaluate “permuted words” pretrained RoBERTa against “real” pretrained RoBERTa on PAWS?

What do you think @andreaparker ?
Anyone who would like to join?


Question:
The newly release AudioClip model combines, audio, text and images.
Would it be possible to train an AudioClip model using Flax/Jax?

      GitHub

AndreyGuzhov/AudioCLIP 6

  Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043) - AndreyGuzhov/AudioCLIP

      arXiv.org

AudioCLIP: Extending CLIP to Image, Text and Audio 5

  In the past, the rapidly evolving field of sound classification greatly
benefited from the application of methods from other domains. Today, we observe
the trend to fuse domain-specific tasks and approaches together, which provides
the community with...

      paperswithcode.com

Papers with Code - AudioCLIP: Extending CLIP to Image, Text and Audio 1

  Implemented in one code library.

Context: Hi! I’d be really interested in participating in this project.


CLIP like contrastive vision-language models for German with pre-traind text and vision models

Question:
CLIP like contrastive vision-language models for German with pre-trained text and vision models
For this project, a pre-trained image model like ViT and a pre-trained text model like BERT can be used as an image encoder and text encoder respectively.

Model
Pre-trained ViT, BERT models can be found on the model hub. We could also use multi-lingual BERT/ROBERTa models for the German language.

Datasets
The WIT 3 dataset can be used for this task.

Available training scripts
A training script for this will be provided soon. (see PR 8)

(Optional) Desired project outcome
The desired outcome is to train a CLIP-like model for German language.  This can be showcased with a streamlit  or gradio app.

(Optional) Challenges
This model will require some modifications to the existing models. Specifically, we will need to add projection layers in both the text and image encoder models.

(Optional) Links to read upon

      Sachin’s Blog – 7 Mar 21

Multilingual CLIP with Huggingface + PyTorch Lightning 🤗 ⚡ 15

  Training OpenAI’s CLIP on google colab

Context: It’s quite an interesting project ,count me in


Wav2vec For Music Applications (generation, captioning, instrument classification)

Question:
Wav2vec is SoTA for ASR. It will be interesting to explore the model or other transformer architectures for Music AI applications. We can focus on of the following depending on bandwidth:
Instrument classification
Vocal separation or instrument segmentation
Emotion/rhythm/pitch analysis
Pitch shift
One thing I am curious about is:
If we train a music to lyrics model, what are the features being learnt by the layers? Can we fine-tune such a model on the other downstream tasks?

Context:
Hey @tanmaylaud,
This paper: [2105.01051] SUPERB: Speech processing Universal PERformance Benchmark 10 might be of interest. I’m also currently working on adding Wav2Vec2 in Flax, see: [WIP][Flax] Add wav2vec2 by patrickvonplaten · Pull Request #12271 · huggingface/transformers · GitHub 14. Hopefully this will be merged by next week.


Question:
GPT2 for Indonesia
Currently, there is only few GPT2 model that was trained from scratch for Indonesian (small size only) on the hub: Hugging Face – The AI community building the future. 2 . For this project, the goal is to create a strong language generation model for Indonesia.

Model
A randomly initialized GPT2 model

Datasets
One can make use OSCAR  the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face.
A portion mc4 [MC4] (The C4 Multilingual Dataset · Discussion #5265 · allenai/allennlp · GitHub 2) and
CC100 [CC-100: Monolingual Datasets from Web Crawl Data] can also be used for this task

Available training scripts
A causal language modeling script for Flax is available here  3. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Indonesia language.

(Optional) Challenges
The dataset on MC4 and CC100 is quite large → around 350 GB. One might want to try out dataset streaming to solve this problem. Dataset streaming will be merged to datasets in a couple of days. See PR here 1 and the docs here.
Since mc4 not available in datasets yet, adjustment for streaming dataset could be required

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory  2

https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171  1

Context: Thanks for creating the proposal, I am in.


Question:
CLIP like contrastive vision-language models for Italian with pre-trained text and vision models
For this project, a pre-trained image model like ViT and a pre-trained text model like BERT can be used as an image encoder and text encoder respectively.

Model
Pre-trained ViT, BERT models can be found on the model hub. We could also use multi-lingual BERT/RoBERTa models for the Italian language.

Datasets
The WIT 4 dataset can be used for this task.

Available training scripts
A training script for this will be provided soon. (see PR 15)

(Optional) Desired project outcome
The desired outcome is to train a CLIP-like model for Italian language. This can be showcased with a streamlit or gradio app.

(Optional) Challenges
This model will require some modifications to the existing models. Specifically, we will need to add projection layers in both the text and image encoder models.

Context:
That is super interesting!
We have ViT and Italian checkpoints: can’t wait to see it in action!


Question:
ViT+CLIP+NeRF: Fewshot Learning, Putting NeRF on a Diet

Is anyone interested in Computer Vision Field?
Our team project goal is the implementation of this paper:
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis :

      arXiv.org

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis 31

  We present DietNeRF, a 3D neural scene representation estimated from a few
images. Neural Radiance Fields (NeRF) learn a continuous volumetric
representation of a scene through multi-view consistency, and can be rendered
from novel viewpoints by ray...

With this project, We will code a 3D neural scene representation (NeRF: Neural Radiances Field) estimated from a few images. Which is based on extracting the semantic information using a pre-trained visual encoder such as CLIP, a Vision Transformer.

1. Languages and Skills

Languages : Python, Pytorch, JAX
Other Skills : Git, Github, Cloud Computing Experience

2. Model / Baseline Code

ViT, CLIP : https://github.com/openai/CLIP

Meatlearning-NeRF (Learned Initialization Paper : JAX Based NeRF Code) :
https://github.com/tancik/learnit

NeRF (Pytorch Based NeRF Code) : https://github.com/yenchenlin/nerf-pytorch

3. Datasets

Phototourism dataset:
https://www.cs.ubc.ca/~kmyi/imw2020/data.html

Realistic Synthetic dataset :
 https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1

DTU multi-view stereo (MVS) dataset :
http://roboimagedata.compute.dtu.dk/?page_id=36

4. Challenges

JAX based NeRF Code
Data Preprocessing
CLIP / NeRF training with TPU
Augmentation CLIP to NeRF with JAX based code

5. Project Goal Output
The desired project output is an implementation of the Fewshot learning NeRF model which is using CLIP+ViT (JAX based code)

6. References

NeRF : https://www.matthewtancik.com/nerf

Putting NeRF in a diet : https://arxiv.org/abs/2104.00677

Learned Initialization : https://arxiv.org/abs/2012.02189

7. Team Mindset
-Enjoy!
-Passionate
-Collaborative
-Learning Oriented

Context:
Hi @howtowhy ,
This appears very interesting and cool project. I would like to participate and contribute in this project. I am hoping, I will be able to learn interesting concepts from this project.


Pretrain and Fine Tune Byte-level model for multilingual extractive QA (Like ByT5)

Question:
I am really fascinated by Byte-level models which have been proposed currently such as ByT5 and Canine that get rid of tokenizers. I have tried probing the ByT5 model to check whether it learns morphological and syntactic features.
Project Proposal: I intend to pre-train the ByT5 model from scratch for a few languages(currently it is trained on mC4 corpus which has 100+ languages) and then fine-tune the model for zero extractive QA. More information can be found here 11
Goal: At the end, I would like to have insights into how does the ByT5 model performs when random syenitic noise is added to the data (See the last part of the experimental section of the paper for more information).
Languages: As this is a multilingual project, I am considering a set of languages that belong to a family(Like the Indian Languages or European Languages) and the dataset used would be part of mC4 data set(this can be changed if we get more data)
Dataset for Multilingual QA:  MLQA 3 or X
Pretraining script; The fine-tuning script is available in the Transformer’s library,  pretarning script is available in the official repo of the model 13
Challenges: Getting the pretraining script running with JAX may be a challenge, though I am not sure how challenging it would be.

Context:
Hey there,  @sidicity,
The project idea sounds  interesting   and I have been wanting to work on training Multi-lingual models myself.
Count me in!


Question:
PreTrain GPT-2 from scratch for German on novel GC4 dataset
To the best of our knowledge, there is no GPT-J model trained on a german corpus.

2. Language
The model will be trained in German.

3. Model
GPT-J

4. Datasets
The German colossal, cleaned Common Crawl corpus. GC4 Corpus — German NLP Group documentation 5
Another potential source of data is that which was used by the Digitale Bibliothek/Münchener Digitalisierungszentrum to pretrain their German GPT-2 variant

5. Training scripts

      GitHub

kingoflolz/mesh-transformer-jax 6

  Model parallel transformers in JAX and Haiku. Contribute to kingoflolz/mesh-transformer-jax development by creating an account on GitHub.

6. Challenges
Ridiculous amount of data renders both training and preprocessing challenging.

7. Desired project outcome
A competitive German language model, and a demo based on the HF SvelteKit inference demos

8. Team members
Christopher Akiki (myself) and Alina Mailach (@mailach)

Context:
More information on german models trained by Digitale Bibliothek/Münchener Digitalisierungszentrum
together with relevant papers can be found here here.  4


Question:
Multilingual Visual Question Answering
We are currently planning to use ViT + mBert for two modalities namely images and text for Multilingual Visual Question Answering task

Model
Pre-trained models  ViT 3 , mBERT , can be used for this task.

4. Datasets
Currently we came across one dataset WIT 1 which is a large multimodal multilingual dataset , We can update the datasets if we encounter more suitable dataset for this task.

5. Training scripts
Since this is a Seq2Seq model, the run_summarization_flax.py 1 script can be used for training this model with some modifications.

6. (Optional) Challenges

How to combine the two modalities i.e. image and text, some of the ways to this is like VisualBert ,Here we can use ViT for images and use the embeddings from earlier stage for mBERT along with word embeddings. One other way for interaction of two modalities can be done through coattentional transformer layer like in ViLBERT, so we need to decide which would be better suitable for our task.

7. (Optional) Desired project outcome
Final Goal is to have an end to end model where we can perform Visual QA Task in multiple languages.
We are also planning to benchmark the few-Shot/zero-shot performance on VQA 1/VizWiz 2/GQA 1

Context: Count me in!


Question:
Pre-train DistilmByT5Neo
Let’s combine ByT5, mT5, GPT-Neo and DistilBert!
Model
For initial pre-training, a randomly initialized ByT5 model which we distil after pre-training is completed.
Datasets
c4multilingual 2
The Pile 3
Training Scripts
Training scripts will be created as part of the project
Expected Result
A Distilled model that combines the power of T5 and GPT-Neo while removing the need for tokenization.
UPDATE: We could just go for it and implement this with rotary embeddings as well.

Context: We could kick this up a notch further with rotary embeddings…DistilmRoByT5Neo?


Train the Best Sentence Embedding Model Ever with 1B Training Pairs

Question:
Background
The quality of sentence embedding models can be increased easily via:

Larger, more diverse training data
Larger batch sizes

batch_size784×571 77.9 KB

However, training on large datasets with large batch sizes requires a lot of GPU / TPU memory.
TPU-v3-8 offers with 128 GB a massive amount of memory, enabling the training of amazing sentence embeddings models.
Join me and use this event to train the best sentence embedding models that ever existed.

Roadmap

Create a Jax training script for MultipleNegativesRankingLoss 82. MultipleNegativesRankingLoss is currently the best method to train sentence embeddings. As training data, we need text-pairs (textA, textB) where we want textA and textB close in vector space. This can be anything like (question, answer), (text, summary), (paper, related_paper), (input, response).
Collect suitable training data:

I already have 25 suitable training datasets, that provide 100+ million training pairs (some are listed here 68).
Mine StackExchange (title, question, best_answer) triplets from the stack exchange archive 7

Mine Conversational Datasets from Reddit: PolyAI 21 has the script ready
Extract Wikipedia intro sections for articles that are in the same category
Do you have further ideas for suitable (large scale) datasets?

The data collection should give us a train dataset of 1+ billion pairs
Train on this massive corpus and create the best sentence embedding model that ever existed

Language
The initial training will focus on English. After the event, Multi-Lingual Knowledge Distillation 37 will be used to transfer the model to 50+ languages.
If you have large scale training data for other languages, feel free to provide it and we can try to train a multilingual model too.

Output
We will train different models:

General purpose model
Dedicated model for Semantic Search / Question-Answer-Retrieval
Dedicated model for Conversational AI

Models and datasets will be shared with the community.

Model to Use
We will use RoBERTa and maybe several smaller models (Distil*-Models, MiniLM etc.).

Training Script
Training-Script is not yet available, but creating it is not too difficult: We need a the token embeddings from the model + mean pooling + cosine-similarity + CrossEntropyLoss.

You want to join?
Help is needed on different aspects of the project

Data crawling & preparation (more data is always better)
Creating a suitable JAX train script for InfoNCE Loss (I have one for PyTorch)
Create code for data loading so that we can train on 1B train pairs

Interested to join? Please send me an email to
nils@huggingface.co
so that I can invite you to the kick-off event.
We also have our own Discord server for communication:

      Discord

Join the flax-jax-community-week-sentence-embeddings Discord Server! 36

  Check out the flax-jax-community-week-sentence-embeddings community on Discord - hang out with 91 other members and enjoy free voice and text chat.

Data crawling, preparation and code writing must happen before we get the TPU compute power ( 7.07. - 14.07.)

Context:
Very cool idea! I have the impression that Transformer sentence embeddings are a bit neglected since Sentence-BERT and so…I’m interested to join this project!
Useful info about me:

I’m a young AI researcher working in Indigo.ai, a little company based in Italy and focused on conversational AI
I have experience with the  Transformers library, but I never used JAX. I think that this project could be an occasion for me to gain confidence whith this library, so I would like to work on the training script for this project, but obviously I’m open to other tasks!

If you want to know my time zone…I live in Italy!

Just one question: what’s the idea to train this model? It seems that you only need couples of similar sentences, right? No negative examples (sentences with different meanings) are required?


Question:
RoBERTa/BERT for Thai
Currently, there is only a very limited amount of BERT-like models for Thai on the hub: Hugging Face – The AI community building the future. 2 . For this project, the goal is to create a RoBERTa/BERT model for just the Thai language.

Model
A randomly initialized RoBERTa/BERT model

Datasets
One can make use OSCAR 2 the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face.

Available training scripts
A masked language modeling script for Flax is available here 2. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a strong RoBERTa/BERT model in Thai.

(Optional) Challenges
The OSCAR dataset might be too small (it has < 20GB of data for Thai). Also it might be important
to find datasets the BERT-like model can be evaluated on after pretraining in Thai. Having found a dataset to fine-tune the pretrained BERT-like model on, one can make use of the text-classification script here 2

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory 4

Context:
Hi there. I have tried with this blog 2 for PyTorch before and would love to do this in Flax/Jax too
I will take a look and update the status back soon.


Question:
Sentiment analysis of Sinhala language using deep learning networks

1. Description
The main objective of the project is to test deep learning models to identify the sentiments in Sinhala text. A Facebook dataset is used to train and test the models. The model codes are already developed and only the training and testing phases remain to be done. Since Sinhala remains as a resource poor NLP language, this project will lend a hand to improve the current tools and provide insight on the current state.
Migrating the current code into JAX with the use of Flax, Haiku and other libraries is another objective. Libraries like Trax with basic deep learning models and the trending Transformers are aimed to be tested.

2. Language
The models are trained in Sinhala Language

3. Model
The models that will be tested are

RNN
LSTM
GRU
BiLSTM
Baseline  models  with  the  combination  of  a CNN
Stacked LSTM and BiLSTM
HAHNN
Capsule networks

4. Dataset
A Facebook dataset contaning 526,732 Sinhala and English posts extracted from CrowdTangle . The dataset consists of a decade’s worth of content from Facebook pages popular in Sri Lanka.

5. Training scripts
The following links contain the model scripts
Main models 1

6. Challenges
There are several models that needs to be adjusted and tested

7. Desired project outcome
Performance measures of each model

8. Reads
The following links can be useful to better understand the project and
what has been done previously.
https://sencat.lk/ 1

Context: Hello there! I would like to join this project.


Question:
FlaxBigBird for Dutch language
Currently, there are only a very limited amount of long-range sequence models for other languages than English on the hub. The goal of this project is to create a strong Dutch FlaxBigBird model.

Model
A randomly initialized  FlaxBigBird model.

Datasets
CC-100 2
mC4 2

Available training scripts
Scripts to pre-train Flax models are here: transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHub 3
Scripts for fine-tuning and evaluation for FlaxBigBird just got released here: transformers/examples/research_projects/jax-projects/big_bird at master · huggingface/transformers · GitHub 2 Thanks to @vasudevgupta

(Optional) Desired project outcome
The desired project output is a strong Dutch FlaxBigBird model in Dutch. For downstream tasks that require long sequences (e.g. long text classification).

(Optional) Challenges

Context: Agree it would be great to have more long-range sequence models for Dutch! We don’t have Longformer implemented in Jax/Flax - maybe FlaxBigBird could be an option?


Model for Bengali Poetry Generation with GPT-2

Question:
GPT-2 for Bengali Poetry Generation

Description
In this project, I propose to generate Bengali poetry using a GPT-2 model trained from scratch.
As TPUv3-8 will be used, this should not take much time.
After a model has been trained, I will use the model to fine-tune with poems written in Bengali.

Datasets
The model can leverage multiple Datasets:

The Bangla Wikipedia Corpus 6 dataset available on Kaggle
Poems available in the public domain of poets such as Jibananda Dash 4, Rabindranath Tagore, Michael Madhusudan Dutt 1, etc.

Available Training Scripts
I believe this training script 3 can be used to train the model.

Desired project outcome
The goal of the project would be to be able to generate Bengali poetry sensible enough to fool humans into thinking a poet wrote it.

Challenges

As there is no pretrained GPT-2 model that exists, it might take a while longer to train it from scratch, and then fine-tuning it poetry.
The format of the data is different for different websites.
Some authors (Tagore) have much more voluminous contributions to the literature than others (Madhusudan Dutt). Their writing style will disproportionately influence the model.

Suggested Reading

 Bengali Poem Generation Using Deep Learning Approach 3
Poetic Machine: Computational Creativity for Automatic Poetry Generation in Bengali 1
Sequence-to-sequence Bangla Sentence Generation with LSTM Recurrent Neural Networks
 Context-driven Bengali Text Generation using Conditional Language Model
 Training an AI to create poetry (NLP Zero to Hero - Part 6)
 NLP with Bangla: Generating Text and Sentiment Analysis 1

Context: @patrickvonplaten any update on this project being greenlit?


Question:
RoBERTa for Kannada
Currently, there are only two models available for hate speech detection in the hugging face model hub. By pre-training a RoBERTa model, we wish to increase the accessibility to one of the oldest Indic languages.

2. Language
Kannada

3. Model
A randomly Initialized RoBERTa model.

4. Datasets
Here are some of the datasets containing Kannada sentences: preprocessing required.

Automate Text-based Workflows at Scale 1

5. Training scripts
A masked language modeling script for Flax is available here. Probably the same can be used.

6. Desired project outcome
To use this model and fine-tune it for a sentiment analysis task for Kannada text sentences.

Context:
More datasets containing Kannada sentences :

Kannada News Dataset | Kaggle
Kannada Covid-19 Sentiment Analysis Dataset | Kaggle


Question:
PreTrain Wav2Vec2 in Turkish
On the previous community event I pretrained a Turkish Wav2Vec2 model 2 using the fairseq script. This is a subpar model because I hadn’t cleaned the data.
This time I want to do it properly with the freshly merged    FlaxWav2Vec2 + PreTraining script 5

Model
A randomly initialized Wav2Vec2 model

Datasets
One can make use Common Voice the dataset is also available through the datasets library here: common_voice · Datasets at Hugging Face.

Available training scripts
FlaxWav2Vec2 will be merged soon: [Flax] Add wav2vec2 by patrickvonplaten · Pull Request #12271 · huggingface/transformers · GitHub  1 and a pretraining script should be relatively easy to be merged.

(Optional) Desired project outcome
The best Turkish ASR model.

(Optional) Challenges
I have some additional scraped audiobook data. Might need a bit more though.

Context: This is a well-defined topic, let’s define it as well!


Pretrain GPT-2 from scratch in Mongolian

Question:
GPT2 for Mongolian
Goal is to create a strong language generation model for Mongolian
Since initial code and data is pretty much written by @patrickvonplaten and other huggingface members, it should not be so hard to get the first sense.

Model
Randomly initialized GPT2 model

Datasets
We can use OSCAR which is available through datasets

Datasets
A causal language modeling script for Flax is available here 2. It can be used pretty much without any required code changes.
If there is time left, I’d love to try some private crawling and integrate it datasets.

Expected Outcome
Understandable Mongolian text generation model

Challenges
Lack of data → OSCAR Mongolian is just 2.2G. Maybe we need to research ways to acquire more data with this.

Links

Colab Notebook to train CLM 3
OSCAR
flax code 2

Context: I am also so interested in pretraining Mongolian Roberta/DistillBERT on OSCAR and fine-tune it on the translated Mongolian SQuAD2.0.


Generate GIF reply to English text with VQGAN + CLIP

Question:
So this would be a hybrid dialog system combining intent recognition with text-to-image generation. The idea is to train the model to generate visual replies (static or dynamic GIFs, depending on how hard the latter is) to English-language text input. E.g.:
text input: ‘this restaurant is the bomb’
model output: generated picture of exploding restaurant
There are quite a few GIF and text-to-image datasets out there, but we’d need to combine them in a way that the model can be trained. A large part of the work will be to implement VQGAN and CLIP in JAX.

Context:
I’m also interested in this task
I would like to join forces if you are interested. I’m pretty confident in NLP.
I know CLIP, (some of) vqgan model but don’t know Jax.
My github: cceyda 24
Also see this dataset 10


PreTrain RoBERTa/T5 from scratch for Programming Languages

Question:
RoBERTa/T5 for Programming Languages
Currently, there is no MLM-based model (some are available) 15 that was trained from scratch for Various Programming Languages on the hub. For this project, the goal is to create a strong programming language generation model.

2. Language
Primary Languages: C++, Python, Java
Apart from that, we can also consider the following languages based on availability. We have scrapped a large set of programming language codes. However, we are in the process of discussing the matter of licensing issues with the data owner. These are the scrapped languages so far.
lang_dict = {
    "C":["GNU C", "GNU C11"],
    "C++":[ "GNU C++", "GNU C++0x", "GNU C++11", "GNU C++14", "GNU C++17", "GNU C++17 Diagnostics", "MS C++", "MS C++ 2017", "Clang++17 Diagnostics"],
    "C#":[".NET Core C#", "Mono C#", "MS C#"],
    "D":["D"],
    "Delphi": ["Delphi"],
    "FPC":["FPC"],
    "Go":["Go"],
    "Haskell":["Haskell"],
    "Java":["Java 6", "Java 7", "Java 8", "Java 11", "Kotlin"],
    "JavaScript":["JavaScript", "Node.js"],
    "Ocaml":["Ocaml"],
    "PascalABC.NET":["PascalABC.NET"],
    "Perl":["Perl"],
    "PHP":["PHP"],
    "Python 2":["PyPy 2", "Python 2"],
    "Python 3":["PyPy 3", "Python 3"],
    "Ruby":["Ruby"],
    "Rust":["Rust"],
    "Scala":["Scala"]
}

Please note that we cannot share our own dataset of codes without resolving licensing issue. However, there are enough data available online to train an LM.

3. Model
A randomly initialized RoBERTa model.

4. Datasets
We can use dataset from,

GitHub - facebookresearch/TransCoder: Public release of the TransCoder research project https://arxiv.org/pdf/2006.03511.pdf 9
GitHub - github/CodeSearchNet: Datasets, tools, and benchmarks for representation learning of code. 4
If we can resolve the licensing issue, we can share our own datasets via hugging-face dataset pipeline.

5. Training scripts
A masked language modeling script for Flax is available here 10.

6. Challenges

Data pre-processing
Sample preparation for LM

7. Desired project outcome
The desired project output is an MLM/T5 model that is able to generate Programming language.

8. Reads
The most important read would be the following colab:

Google Colaboratory 14

Apart from that we need to check the tokenization procedure from,

GitHub - facebookresearch/TransCoder: Public release of the TransCoder research project https://arxiv.org/pdf/2006.03511.pdf 9
GitHub - github/CodeSearchNet: Datasets, tools, and benchmarks for representation learning of code. 4

UPDATE 1: In addition to some interest from many people, I must add that, there are some pre-trained models already in hugging-face which I was not aware of at the time of writing this proposal.  Model Link: Hugging Face – The AI community building the future. 15
UPDATE 2: We can try training a T5 like model with Programming Languages instead of an MLM-like model.

Context: Interesting project. I am in!


Train T5 model for commit message generation

Question:
T5 for commit message generation
Commit message generation is known to be a difficult task due to the following reasons:

No available clean and large dataset for pre-training and fine-tuning
(Almost) No available pre-trained seq2seq models on both programming and natural languages corpora
Context limitations (sequence length, external knowledge, etc.)

In the recent research [1 3] [2 2] [3 2] it has been shown that pre-trained transformer models (like T5 or BART) can capture code understanding domain and fine-tune for various seq2seq programming downstream tasks (e.g. code summarization, code documentation generation, commit message generation, etc).
Nevertheless, each of the listed downstream tasks deserves more thorough research. I’ve chosen commit message generation to be such a task because it has many practical applications, such as auto version control systems.
To wrap up, these are the main goals of this research project:

Release a publicly available dataset for programming language models pre-training
Release a publicly available dataset for commit message generation fine-tuning
Release a publicly available (pre)-trained T5 model for commit message generation
Release all pre-processing, post-processing, and training scripts for further research

2. Language
We’ll use English and Python languages for training.
This research is limited in time, so it would be great to show a strong baseline for at least one pair of languages. Later we’ll be able to extend the model for more language pairs.

3. Model
We’ll be using a random T5 model similar to t5-base in configuration.

4. Datasets
Possible links to publicly available datasets include:

bigquery-public-data:github_repos 9
C4 3

Please note that the BigQuery GitHub dump needs to be cleaned up before being used for pre-training/fine-tuning. We should also consider using only those repos whose license allows us to do so.

5. Training scripts
We can make use of run_summarization_flax.py 19 to train the model.

6. (Optional) Challenges
The main challenges are to prepare the dataset and set up training tasks in Flax. Besides, further challenges include facing the short context sizes and transferring knowledge across domains. I suppose we could try using sparse attention blocks and techniques like RAG for bringing the domain knowledge.

7. (Optional) Desired project outcome
The desired project outcome is to achieve the listed goals and have fun! A demo would be a simple application (e.g., VSCode extension) that generates commit messages given the changed python files.

8. (Optional) Reads
The following links can be useful to better understand the project and
what has previously been done in the research community.

Contextualized Code Representation Learning for Commit Message
Generation 2
A Transformer-based Approach for Source Code Summarization 3
PYMT5: multi-mode translation of natural language and PYTHON code with transformers 2
Unsupervised Translation of Programming Languages 1

Context:
This idea is definitely quite  interesting
I would love to join in and be a part of this
I have worked on Seq-2-Seq tasks like Machine Translation on low-resource languages, Character Transliteration for code-mixed conversations, Built and Trained Transformer models in both PyTorch as well as TensorFlow before. I have been meaning to find another seq-2-seq task to work on and this is just  ideal .


Question:
Description:
Given a list of ingredients, generate a recipe – similar to what the GPT3 API offers: openAI recipe generator 8. A model exists on the hub that does Recipe NLG 7, but it uses a GPT2 architecture. I’m curious if T5 or Bart will produce better results.
Model:
T5 or Bart.
Dataset:
Recipe NLG (2,231,142 recipe examples) - hugging face link 4 - download site 2
Data example:

{‘NER’: ‘[“oyster crackers”, “salad dressing”, “lemon pepper”, “dill weed”, “garlic powder”, “salad oil”]’,
‘Unnamed: 0’: 1000,
‘directions’: ‘[“Combine salad dressing mix and oil.”, “Add dill weed, garlic powder and lemon pepper.”, “Pour over crackers; stir to coat.”, “Place in warm oven.”, “Use very low temperature for 15 to 20 minutes.”]’,
‘ingredients’: ‘[“12 to 16 oz. plain oyster crackers”, “1 pkg. Hidden Valley Ranch salad dressing mix”, “1/4 tsp. lemon pepper”, “1/2 to 1 tsp. dill weed”, “1/4 tsp. garlic powder”, “3/4 to 1 c. salad oil”]’,
‘link’: ‘www.cookbooks.com/Recipe-Details.aspx?id=648947’,
‘source’: ‘Gathered’,
‘title’: ‘Hidden Valley Ranch Oyster Crackers’}

Script:
Maybe run_summarization_flax.py 2 and just treat it as seq2seq?
Expected result:
Give it a list of ingredients (e.g. sugar, flour, egg, peanut butter) and it spits out a recipe (peanut butter cookies). Useful for when you are trying to use up all those miscellaneous ingredients in your pantry and fridge!
Unfortunately I don’t think I will have the time to be able to work on this, but I thought it might be an interesting idea that someone else could run with.
Optional 1: Have it create a title for the recipe.
Optional 2: Input the amounts of each ingredient so that it has some constraints.

Context: Think this a really cool and creative project idea! Would love to see this project take place! Let’s see if anyone else is interested in joining


Question:
PreTrain Electra/T5 for Korean from scratch
This project will be train the pretrained-model (Electra, T5, …) from Korean corpus called as a 모두의말뭉치.

2. Language
The model will be trained in Korean.

3. Model

Electra
T5

4. Datasets
모두의말뭉치 4

5. Training scripts
We can make use of example 20 to train the model.>
Addtionaly, we will be make the train codes origin from the transformers example

7. Desired project outcome
A pretrained weights. After, fine-tune both text classification and text summarization

Context: I’m interested in pretrained korean model, too! I wanna join in. ol