Question:
Please read the topic category description 17 to understand what this is all about

Create your own search engine
In Chapter 5 of the  Course, you learned how to use FAISS to find documents that are most semantically similar to a given query. The goal of this project is to extend this idea to build a retrieval and reranking system, where the retriever returns possibly relevant results, while the reranker evaluates the how relevant these hits are to the query.
An example of the architecture might looks as follows (taken from the sentence-transformers library):

image945×279 8.53 KB

Model(s)
The sentence-transformers models on the Hub 9 are great for the reranking task.

Datasets
Wikipedia is usually a good corpus to test retrieval systems on and you can find a dump in various languages here:

wikipedia 12

Challenges
Implementing the full retriever-reranking architecture might be a challenge, so a simpler place to start is with a single long document. You can then chunk that document into paragraphs and compute the relevancy scores across each paragraph

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 3 that allows a user to enter a search query about a document (or a whole corpus of documents), and returns the top 5 most relevant paragraphs.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

https://www.sbert.net/examples/applications/retrieve_rerank/README.html#retrieve-re-rank 24

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 9 and:

Follow the instructions on the #join-course channel

Join the neural-search-engine channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: Hi , I will be working on this cool project during the event.


Poster2Plot: Generate Movie/T.V show plot from poster

Question:
Description
Our team is working on building an image captioning model which can generate a movie/t.v show plot from it’s poster.
The goal of this project is to create an image captioning model using a transformer encoder model like Vision Transformer (ViT) and a transformer decoder language model like GPT-2

Model(s)
Any vision based encoder and language model decoder would be a good candidate to train the VisionEncoderDecoderModel for image captioning. We are trying the following models first:

Encoder: google/vit-base-patch16-224-in21k 1

Decoder: gpt2 1

Datasets
We are using publicly available IMDb datasets to train the model.
Some examples:

IMDb movies extensive dataset | Kaggle 1
48K IMDB Movies With Posters | Kaggle 1

Challenges
The main challenge is to create a good dataset of poster and movie plots. Also it will be interesting to see if the model gives good predictions for non-english movies/tv shows.

Desired project outcomes
We will create a Streamlit or Gradio app on  Spaces 3 that can predict a movie/t.v show plot from it’s poster.

Context: Let’s give it a try.


Build a title recommender for scientific articles

Question:
Please read the topic category description 1 to understand what this is all about

Description
If you’ve ever worked in research, then you know that picking a catchy title for your articles is not easy! In this project, we’ll train a summarization model that can convert abstracts into titles.

Model(s)
Several of the Summarization models 3 on the Hub should serve as a good starting point for this project.

Datasets
There are several scientific datasets available on the Hub:

arxiv_dataset 6
pubmed 3

Challenges
Training a model that can summarise across any scientific domain is unlikely to be feasible. We recommend picking one subdomain (e.g. high-energy physics) and focusing your efforts there.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that can generate titles from a given abstract.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

https://cs230.stanford.edu/projects_spring_2020/reports/38954132.pdf 4

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 1 and:

Follow the instructions on the #join-course channel
Join the science-title-generator channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: This looks like a great project and I would love to contribute to this.


Question:
Please read the topic category description to understand what this is all about

Description
Emoticons are often used as a proxy for emotional content in social media posts or instant messaging chats. As a result, emojis are often used as a label to train text classifiers. The goal of this project is to create a Transformer-based implementation of DeepEmoji 6, a research project from MIT that studied this task with LSTMs.

Model(s)
Any BERT-like model would be a good candidate for fine-tuning on an emoji dataset and you can get inspiration from models like these:

cardiffnlp/twitter-roberta-base-emoji

Datasets

tweet_eval 7

Challenges
To get better performance, you may want to perform domain adaptation by fine-tuning the language model on in-domain data. We recommend trying this approach only after building a baseline classifier.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces that can predict the top 5 emojis associated with a piece of text
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

https://huggingface.co/spaces/ml6team/emoji_predictor 6 (a Space for inspiration)

Discord channel
To chat and organise with other people interested in this project, head over to our Discord and:

Follow the instructions on the #join-course channel
Join the learning-from-emojis channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: This project sounds interesting. I would like to work on something like this.


Question:
Hi,
I am following the course. I am now at Fine-tuning Fine-tuning a pretrained model - Hugging Face Course.  When I set up DataCollatorWithPadding as following I got an error while trying to reproduce the course code in Kaggle. This error occurs with either a CPU-only-device or a GPU-device.
Input:
checkpoint = 'bert-base-uncased' tokenizer = AutoTokenizer.from_pretrained(checkpoint) data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
Output:
`---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_42/1563280798.py in
1 checkpoint = ‘bert-base-uncased’
2 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
----> 3 data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors=“pt”)
TypeError: init() got an unexpected keyword argument ‘return_tensors’`
When I call help method, it too confirms that there is no argument return_tensors.
Input:
help(DataCollatorWithPadding.__init__)
Output:
`Help on function init in module transformers.data.data_collator:
init(self, tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, padding: Union[bool, str, transformers.file_utils.PaddingStrategy] = True, max_length: Union[int, NoneType] = None, pad_to_multiple_of: Union[int, NoneType] = None) → None`
But, the source file Data Collator — transformers 4.12.5 documentation says that there is such an argument. By default, it returns Pytorch tensors while I need TF tensors.
Where do I miss?
Please help me.

Context:
Hi all, this issue is due to the older version of transformers library. When I upgraded it, problems are resolved.
# upgrade transformers and datasets to latest versions
!pip install --upgrade transformers
!pip install --upgrade datasets

Best Regards.


Dataset object has no attribute `to_tf_dataset`

Question:
I am following HuggingFace Course. I am at Fine-tuning a model.
Link: Fine-tuning a pretrained model - Hugging Face Course 1
I use tokenize_function and map as mentioned in the course to process data.
# define a tokenize function
def Tokenize_function(example):
    return tokenizer(example['sentence'], truncation=True)
# tokenize entire data
tokenized_data = raw_data.map(Tokenize_function, batched=True)
I get Dataset object at this point. When I try converting this to a TF dataset object as mentioned in the course, it throws the following error.
# convert to TF dataset
train_data = tokenized_data["train"].to_tf_dataset(
  columns = ['attention_mask', 'input_ids', 'token_type_ids'],
   label_cols = ['label'],
   shuffle = True,
   collate_fn = data_collator,
   batch_size = 8
)
Output:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_42/103099799.py in <module>
      1 # convert to TF dataset
----> 2 train_data = tokenized_data["train"].to_tf_dataset( \
      3     columns = ['attention_mask', 'input_ids', 'token_type_ids'], \
      4     label_cols = ['label'], \
      5     shuffle = True, \
AttributeError: 'Dataset' object has no attribute 'to_tf_dataset'
When I look for dir(tokenized_data["train"]), there is no method or attribute in the name of to_tf_dataset.
Why do I get this error? And how to clear this?
Please help me.

Context: Hey @rajkumar I believe the to_tf_dataset() method was only added in a recent version of datasets. Could you try upgrading to the latest version and check if the problem persists?


Question:
Hello everyone !
I’ll collect here any frequently asked questions that arise during the course of the community event on November 15th - 19th. Feel free to comment here if something is unclear

Can I propose my own project?
Yes, this is certainly encouraged! Just create a new topic under the #course:course-event category. You can use one of the existing project ideas as a template for what information to include.

Can I work on more than one project?
Yes, but we recommend completing one first with before starting/joining another.

How do I join a team?
Just comment on one of the project ideas in #course:course-event to indicate that you’d like to contribute. Alternatively, you can propose your own project idea by creating a new topic on the forums. Just tag me (@lewtun), so I know what you’re proposing

How many people can each team have?
Each team can have 1-4 people. We’ve found in previous events that having more than 4 people adds planning complexity. If you want to work by yourself, please note that you’ll have to use your own GPUs / cloud provider as the Amazon SageMaker compute is reserved for teams with 2 or more people.

What should I do after joining a team?
There are three main things you should do after joining a team:

Head over to Discord and join the project’s channel. If a channel does not yet exist, you can request it in the #request-a-channel channel and someone in the  team will create it for you.

Join the Amazon Sagemaker Community organisation 13 on the Hub. Every model trained using the free compute on Amazon SageMaker will need to be uploaded to that organization.

[Optional] Join your team’s organisation on the Hub. The organisation can be used to store your datasets and Streamlit / Gradio demo as a Space 3. If an organisation doesn’t yet exist for your project, just create one yourself and share the link in the project topic.

What are the requirements to obtain the  course certificate?
Your team should have a Streamlit or Gradio demo running on Spaces 3. There should only be one Space per team, and you can deploy it under a single user account or organization.
Someone from the  team will check that the application runs and then everyone in the team will receive their certificate. The deadline is November 24th.

Where should our team deploy our Space?
You can deploy your team’s space in several locations:

Under a single team members username (e.g. here’s a Space 1 created under my account)
Under an organisation that you create for the team (e.g. here’s a Space 4 created under the course org)
Under the Amazon SageMaker Community 13 org (here’s an example)

If you want to transfer a Space from your user account to an organisation, just select the Settings tab of your Space and scroll down to the Rename or transfer this space box:

Screen Shot 2021-11-17 at 19.25.261052×822 74.7 KB

There you can select New owner to change the location of your Space to an organisation.

My team has deployed a Space for our project, what do we do next?

Make sure all the team members (i.e. Hub usernames) are listed on the Space
Post a link to your Space in the corresponding project topic in #course:course-event or project channel in Discord. Feel free to tag @lewtun so we don’t miss it

[Optional] Write a nice model card 4 for the models that you trained for the Space
Celebrate on completing the project  !

How can I access the compute provided by Amazon SageMaker?

Fill out this registration form 11

Request to join the Amazon SageMaker Community 13 organisation on the Hub

Do I have to use Amazon Sagemaker to train my models?
Although we encourage you to use the free compute generously provided by Amazon Sagemaker, you are welcome to use your own GPUs / cloud provider.

What kind of compute resources will be supplied by Amazon Sagemaker?
You can expect to have access to a single T4 GPU (16 GB RAM) between November 17-19. After November 19 you’ll need to use your own compute. Fortunately, there are several free options you can try:

Google Colab
Kaggle notebooks
Paperspace Gradient notebooks 2

Can I share my Amazon SageMaker credentials with another team member?
No, please fill out the registration form 11 and you’ll receive a confirmation email with instructions sometime later.

Context:
What kind of compute resources will be supplied by AWS? Which type, how much RAM, for how long can we run it?
Background: I have some ideas for projects, but they require training a large model which cannot be trained with a normal single google colab pro T4/P100 GPU (for training e.g. a DeBERTa-large model even the ‘high-RAM’ GPUs from colab are not enough).


Question:
Please read the topic category description to understand what this is all about

Description
Many open-source projects on GitHub use Issues to triage feature requests, bugs, and so on. For example, check out the Issues tab of  Transformers and  Datasets to get an idea. The goal of this project is to pick your favourite open-source project and create a bot that can automatically assign a Label (e.g. bug, enhancement etc) to a new GitHub issue.

Model(s)
Any of the pretrained BERT-like models on the Hub should serve as a good basis for this project. Given the domain is about source code, you may find that fine-tuning the language model first on the dataset gives a boost in performance.

Datasets
For this project you’ll have to create your own dataset by downloading and processing the GitHub issues associated with an open-source project. You can do this with GitHub’s REST 1 or GraphQL APIs. You can find an example dataset on the Hub here:

lewtun/github-issues 5

Challenges
This is a multilabel classification task, so you’ll need to do some data exploration to figure out which classes can be feasibly detected.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces that injests new GitHub issues from an open-source project and predicts the Labels of each one.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

Predicting Issues’ Labels with RoBERTa 1
Check out this chapter 1 of the  course for more details

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 1 and:

Follow the instructions on the #join-course channel

Join the #github-issues-classification channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: I would like to proceed with this project if still possible


Question:
Please read the topic category description 8 to understand what this is all about

Description
Twitter classifies trending tweets according to predefined set of topics like “Data science”, “Hip hop”, “Sport” etc. However, their algorithm appears to often get confused by certain keywords in the tweet or the content of the image (see here 14 for some funny examples).
The goal of this project is to explore whether it’s possible to create a better topic extractor, or at least one that is more targeted for a smaller set of domains. There’s several ways to approach to task:

Frame it as a multiclass classification problem
Frame it as an unsupervised clustering problem, and combine this with techniques like UMAP 3 and/or HDBSCAN

Model(s)
Since Tweets are short in length, picking one of the sentence-transformers models on the Hub 4 is likely a good place to start.

Datasets
There are various Twitter datasets on the Hub and here’s a few examples to start with:

tweet_eval 10
sentiment140 8
emotion 8

Challenges
If you take the unsupervised learning approach, be warned that there’s no “correct” answer and you will have to experiment with the dimensionality reduction / cluster algorithms to get meaningful clusters.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that either can automatically classify a Tweet according to a topic, or visualises the 2D projection of the embeddings and colours them by topic.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

https://towardsdatascience.com/how-exactly-umap-works-13e3040e1668 2

https://huggingface.co/spaces/edugp/embedding-lenses 6 (useful for UMAP inspiration)

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 1 and:

Follow the instructions on the #join-course channel

Join the #twitter-topic-extractor channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: I’d like to work on this project!


Project: Create a new zero-shot model with NLI data

Question:
Description
The zero-shot classification pipeline has becomes very popular on Hugging Face. It allows you to classify a text in any category without having to fine-tune a model for the specific classification task you are interested in.
The zero-shot pipeline is based on models trained on Natural Language Inference (NLI). This project will train a new NLI model, which can then be used in the zero-shot classification pipeline.

Model(s)
Any base-model can be used. Since there are already several NLI models on the model hub, I suggest to train a new model based on Microsoft’s DeBERTa-v3 5 model. Version three 2 was only published few weeks ago and can outperform larger models (see an example here 5).
We can probably create a new SOTA NLI model with the new DeBERTa version and enough NLI data.

Datasets
Established NLI datasets include:
MultiNLI 1
SNLI 1
ANLI
Other interesting NLI datasets include:
FEVER-NLI 2
DocNLI 1
LingNLI
More datasets can be included!

Challenges

NLI models can be trained as either 3-class classifiers (entailment/neutral/contradiction) or as 2-class classifiers (entailment/not_entailment). Both setups have different advantages and disadvantages
There is a lot of NLI data (2 mio++ texts in the datasets linked above), which makes training computationally expensive. Optimising the training pipeline is a challenge.
Many different datasets can be translated into NLI-format. Including more datasets can be beneficial, but requires manual transformation of datasets.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces that provides an interface for zero-shot classification with a new NLI model in the backend.

Additional resources
See the links to the datasets above. Also see Joe Davidson’s original blog post 8 on the zeroshot pipeline

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 8 and:

Follow the instructions on the #join-course channel

Join the #zero-shot channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: Hyee! I’d love to contribute in this one. I guess further discussion will take place in Discord?


Question:
Please read the topic category description 12 to understand what this is all about

Description
Many email and word processing applications can now automatically detect and correct common grammatical errors as you write. For example, the sentence “I am doing fine. How is you?” might be corrected to “I am doing fine. How are you?”. The goal of this project is to fine-tune a Transformer model to be able to automatically provide these corrections, similar to how Grammarly 1 does

Model(s)
This task can be viewed as a sequence-to-sequence task, so models like T5 would be a great starting point

Datasets

jfleg 11

Challenges
If you use T5, you’ll need to define a suitable prefix for the text-to-text task. You’ll also need to think about suitable metrics for the evaluation.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces that can automatically provide  suggestions to improve the grammar of some input text. Check out Grammarly for some inspiration on the visualization side.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

https://towardsdatascience.com/fine-tune-a-transformer-model-for-grammar-correction-b5c8ca49cc26 14
https://github.com/PrithivirajDamodaran/Gramformer 2

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 7 and:

Follow the instructions on the #join-course channel
Join the writing-assistant channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Team organization on the Hub
To join this team, make sure you join the following organisation on the Hub:

team-writing-assistant (🤗 Course Team Writing Assistant) 13

Context: Hi, I am Ashish and I am interested in this project and would like to build it.


Question:
Please read the topic category description 11 to understand what this is all about

Description
The Contract Understanding Atticus Dataset (CUAD) is a new dataset for legal contract review. Legal contracts often contain a small number of important clauses that warrant review by lawyers. This is a time-intensive task that requires specialised knowledge, so the goal of this project is to see if Transformer models can be used to extract answers to a predefined set of legal questions.

Model(s)
Many of the Question Answering models 3 on the Hub could serve as a good baseline to get started. Given the specialised domain, you will probably want to try:

Fine-tuning encoder-based models like BERT, RoBERTa, DeBERTa and friends
Performing domain adaptation, by first fine-tuning the language model before tuning the question-answering head

Datasets
CUAD 16 is available on the Hub.

Challenges
This is a highly specialised domain, so a vanilla Transformer may not obtain great results.

Desired project outcomes
Create a Streamlit of Gradio app on  Spaces that allows someone to select a legal contract, one or more questions, and provide the answer.

Additional resources

CUAD announcement 9
CUAD paper 10
CUAD codebase 1

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 2 and:

Follow the instructions on the #join-course channel
Join the #ai-law-assistant channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: Hi @lewtun, my name is Pavle and I would be interested in working on this project.


Create a SentenceTransformer in Dhivehi using ELECTRA

Question:
Description
Dhivehi is a low resource language. Since the available data is less, ELECTRA seems to be a good option as it requires less computing power and training data as compared to others.
Model
electra-small pretrained in dhivehi available here 2

Discord channel
To chat and organise with other people interested in this project, head over to our Discord and:

Follow the instructions on the #join-course channel

Join the #sentence-transformers-dhivehi channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context:
He @ashraq thanks for proposing this interesting project! One question: what do you mean by create a “sentence transformer”? Are you talking about adding a pooling layer to the electra-small model and then training that on a Dhivevi corpus?
Do you also happen to have access / know of a Dhivevi corpus to train on?


Image captioning for low resource Indian Languages

Question:
There are many image captioning systems exist for english language, here in this project we will develop an Image captioning system for an Indian language
If we have time and resource, we can extend this to other languages as well.
Datasets:
Dataset can be created by translating captions of existing Flickr30k or any other image captioning dataset
An example:
https://www.amitavadas.com/Image2Tweet.html 5
Other resources:
Vision encoder Decoder model: Vision Encoder Decoder Models — transformers 4.12.2 documentation 4
Baseline:

      huggingface.co

flax-community/image-captioning at main 7

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 2 and:

Follow the instructions on the #join-course channel
Join the #image-captioning channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: Hey Sean - Looks really interesting.  I am interested.  Joined the Discord Channel.


Question:
Please read the topic category description 2 to understand what this is all about

Description
If you treat music notation as a form of “text”, you can use language modelling to generate new songs! The goal of this project is to explore how well Transformers perform at music modelling.

Model(s)
None that we could find on the Hub, but see here 6 for some pretrained music Transformers

Datasets
None that we could find on the Hugging Face Hub, but see here 4 for some ideas

Challenges
This task probably involves pretraining a Transformer, which can potentially take multiple days using the GPU resources provided by AWS. An interesting alternative would be to see whether one can integrate an existing pretrained model within  Transformers and use that to either fine-tune.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that people can remix famous songs with newly generated ones
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

https://towardsdatascience.com/creating-a-pop-music-generator-with-the-transformer-5867511b382a 3
https://arxiv.org/abs/2002.00212 1
https://openai.com/blog/jukebox/
https://research.google/teams/brain/magenta/ 1

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 1 and:

Follow the instructions on the #join-course channel

Join the #pop-music-transformer channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: This one seems pretty difficult, but too interesting to pass up. I’ll try working on it.


Question:
Please read the topic category description 5 to understand what this is all about

Description
One of the most exciting developments in 2021 was the release of OpenAI’s CLIP model, which was trained on a variety of (text, image) pairs. One of the cool things you can do with this model is use it for text-to-image and image-to-image search (similar to what is possible when you search for images on your phone).
The goal of this project is to experiment with CLIP and learn about multimodal models. Several ideas can be explored, including:

Create a text-to-image search engine that allows users to search for images based on natural language queries. Although CLIP was only trained for English text, you can use techniques like Multilingual Knowledge Distillation 2 to extend the embeddings to new languages
Create an image-to-image search engine that returns similar images, given a “query” image.

Model(s)
The CLIP models 5 can be found on the Hub

Datasets
A common dataset that’s used for image demos is the Unsplash Dataset. You can get access to it here 1

Challenges
This project goes beyond that concepts introduced in Part II of the  Course, so some familiarity with computer vision would be useful. Having said that, the  Transformers API is similar for image tasks, so if you know how the pipeline() function works, then you’ll have no trouble adapting to this new domain.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that allows a user to find images that resemble a natural language query or input image.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

https://www.sbert.net/examples/applications/image-search/README.html 8

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 3 and:

Follow the instructions on the #join-course channel=
Join the #image-search channel (currently full!)
Join the #image-search-group2 channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: Hi @lewtun !  I’d love to work on this project.


[Nov 15th Event] It Ain’t Broke So D̶o̶n̶’t̶ F̶i̶x̶ Let’s Break It

Question:
Use this topic to ask your questions to Jakob Uszkoreit during his talk: It Ain’t Broke So D̶o̶n̶’t̶ F̶i̶x̶ Let’s Break It.
You can watch it on YouTube 4 or on Twitch at 11:45am PST.

Context: What does it mean to “marginalize out” dependencies?


Create a detector of toxicity from political tweets in Spain

Question:
Please read the topic category description 1 to understand what this is all about

Description
The goal of this project is to automatically identify toxic speech emitted by politicians on Twitter. It is focused on Spain which is an interesting multilingual case with several co-official languages which are used interchangeably in politics.

Model(s)
Multilingual models like xlm-roberta-base.

Datasets

tweet_eval  1 is a related resource, but it is English-only.

Challenges

Getting high-quality data in Spanish and/or integrating data in other languages.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces that is able to detect toxicity from tweets.

Discord channel
To chat and organise with other people interested in this project, head over to our Discord and:

Follow the instructions on the #join-course channel

Join the #toxic-tweets-es channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: I’d love to work on this project!


Create an ADR (Adverse drug reaction) extraction model from unstructured text

Question:
Create an ADR (Adverse drug reaction) extraction model for unstructured text:
An adverse drug reaction (ADR) can be defined as an appreciably harmful or unpleasant reaction resulting from an intervention related to the use of a medicinal product. The goal of this project is to extracts the “ADR” term from the unstructured text . Like social media text or any EHR document.
For example : “Rash in hands caused by omeprazole.” here we have ADR: “Rash” which is caused by omeprazole.
Model(s)
The bio_bert model are good to start for the ADR extraction task.
https://huggingface.co/dmis-lab/biobert-base-cased-v1.2 1
https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT 2
https://huggingface.co/gsarti/biobert-nli
Datasets
ade_corpus_v2 3 is usually a good corpus to test the ADR extraction model and you can find other open source ADR corups s well.
Challenges
you are given a social media post or raw text from EHR document and you need to extract “ADR” mention as accurately as possible.
Desired project outcomes
Create a Streamlit or Gradio app on huggingface, Spaces that can Extract “ADR” mention from the raw text.
Don’t forget to push all your models and datasets to the Hub so others can build on them!
Additional resources
https://huggingface.co/datasets/ade_corpus_v2#additional-information 1
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6947008/
https://towardsdatascience.com/automated-adverse-drug-event-ade-detection-from-text-in-spark-nlp-with-biobert-837c700f5d8c 1
Discord channel
To chat and organise with other people interested in this project, head over to our Discord 1 and:
Follow the instructions on the #join-course channel
Join the #adr-extraction channel
Just make sure you comment here to indicate that you’ll be contributing to this project

Context:
Hi,
I am interested in this project


[Nov 16th Event] Philipp Schmid: Managed Training with Amazon SageMaker and  Transformers

Question:
Use this topic to ask your questions to Philipp Schmid during his talk: Managed Training with Amazon SageMaker and  Transformers
You can watch it on YouTube  2 or on Twitch  1 at 11:25am PST
Notebook link 6

Context: Hi, for the course project, can multiple team members launch training at the same time?


[Nov 16th Event] Mathieu Desvé: AWS ML Vision: Making Machine Learning Accessible to all Customers

Question:
Use this topic to ask your questions to Mathieu Desvé during his talk: AWS ML Vision: Making Machine Learning Accessible to all Customers
You can watch it on YouTube  2 or on Twitch  1 at 11am PST

Context:
Is there any tensorboard-like tool to monitor model training on Sagemaker?

This was answered at 1:23:33 in the live stream


[Nov 16th Event] Matthew Carrigan: New TensorFlow Features for  Transformers and Datasets

Question:
Use this topic to ask your questions to Matthew Carrigan during his talk: New TensorFlow Features for  Transformers and  Datasets
You can watch it on YouTube  1 or on Twitch  at 8:30am PST

Context: Could the notebook shown in the video be linked?


[Nov 16th Event] Abubakar Abid: Building Machine Learning Applications Fast

Question:
Use this topic to ask your questions to Abubakar Abid during his talk: Building Machine Learning Applications Fast
You can watch it on YouTube  4 or on Twitch  at 10:35am PST

Context: Will Gradio comes out a friendly desktop interface in future? Sometimes the ML model cannot be used publicly but instead is used internally inside a company.


[Nov 16th Event] Sylvain Gugger: Supercharge your PyTorch training loop with  Accelerate

Question:
Use this topic to ask your questions to Sylvain Gugger during his talk: Supercharge your PyTorch training loop with  Accelerate
You can watch it on YouTube  8 or on Twitch  1 at 9:50am PST

Context: Can Accelarate help with hyperparameter optimization, same way Ray Tune does?


[Nov 16th Event] Lucile Saulnier: Get your own tokenizer with  Transformers & Tokenizers

Question:
Use this topic to ask your questions to Lucile Saulnier during her talk: Get your own tokenizer with  Transformers &  Tokenizers
You can watch it on YouTube  5 or on Twitch  1 at 9:25am PST

Context: And here are the links to the notebook presented: colab link 1 and raw notebook


[Nov 16th Event] Lysandre Debut: The Hugging Face Hub as a means to collaborate on and share Machine Learning projects

Question:
Use this topic to ask your questions to Lysandre Debut during his talk: The Hugging Face Hub as a means to collaborate on and share Machine Learning projects
You can watch it on YouTube  2 or on Twitch  at 9am PST

Context: Can we use git tag in hub for model versioning ?


[Nov 16th Event] Lewis Tunstall: Simple Training with the  Transformers Trainer

Question:
Use this topic to ask your questions to Lewis Tunstall during his talk: Simple Training with the  Transformers Trainer
You can watch it on YouTube  3 or on Twitch  1 at 8am PST
Link to Notebook 9

Context: Is it possible to share the link to the notebook Lewis is working on here?


Use EncoderDecoder models for text summarization

Question:
Please read the topic category description 4 to understand what this is all about

Description
Most of the available Transformer models for text summarization are only available for English documents. At the same time, there are now many pretrained BERT-like models in non-English languages. The goal of this project is to explore whether the [EncoderDecoder architecture](Encoder Decoder Models — transformers 4.12.2 documentation 4) in  Transformers can be used to create summarization models using just the pretrained weights of encoder-based models.
Your task is to pick a pretrained encoder in a non-English language and train it to summarise texts in that language.

Model(s)
See here 7 for example models that people have fine-tuned using this architecture. You task is to create your very own model with this technique!

Datasets
Search for summarization datasets on the Hub 1 to get an appropriate corpus for this task

Challenges
Text summarization is a tricky NLP task, so the performance obtained with these models may not match what is observed for their English couterparts (where much more data is available)

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that can summarize a document in your chosen language
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

Leveraging Pre-trained Checkpoints for Sequence Generation Tasks [PAPER]

Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models 1 [BLOG POST]
Examples of these models on the Hub by @mrm8488: https://twitter.com/mrm8488/status/1458475725565141001?s=20 3

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 4 and:
Follow the instructions on the #join-course channel. Then join one of the following channels:

#encoder-decoder-es channel (Spanish)

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: Interesting project, I am interested in a Spanish summarizer using Encoder-Decoder model. Anyone else interested in this approach?


Question:
Please read the topic category description to understand what this is all about

Description
One of the most exciting developments in 2021 was the release of OpenAI’s CLIP model, which was trained on a variety of (text, image) pairs. One of the cool things you can do with this model is use it to combine text and image embeddings to perform neural style transfer. In neural style transfer, the idea is to provide a prompt like “a starry night painting” and an image, and then get the model to produce a painting of the image in that style.
The goal of this project is to learn whether CLIP can produce good paintings from text prompts.

Model(s)
The CLIP models 2 can be found on the Hub

Datasets
For this project, you probably won’t need an actual dataset to perform neural style transfer. Just a single image should be enough to tune CLIP and an image encoder. Of course, you are free to experiment with larger datsets if you want!

Challenges
This project goes beyond that concepts introduced in Part II of the  Course, so some familiarity with computer vision would be useful. Having said that, the  Transformers API is similar for image tasks, so if you know how the pipeline() function works, then you’ll have no trouble adapting to this new domain.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that allows a user to provide an image and a text prompt, and produces a painting of that image in the desired style

Additional resources
You can Google “neural style transfer” to find plenty of information about this technique. Here one advanced example to give you an idea:

GitHub - orpatashnik/StyleCLIP: Official Implementation for "StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery" (ICCV 2021 Oral) 6

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 2 and:

Follow the instructions on the #join-course channel

Join the #neural-style-transfer channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: Very interesting topic, I am in!


[Nov 15th Event] Thomas Wolf: Transfer Learning and the birth of the Transformers library

Question:
Use this topic to ask your questions to Thomas Wolf during his talk: Transfer Learning and the birth of the Transformers library
You can watch it on YouTube 23 or on Twitch 5 at 8am PST
Slides 10

Context: This is an example of test question. You can test the like button below.


[Nov 15th Event] Matthew Watson - Chen Qian: NLP workflows with Keras

Question:
Use this topic to ask your questions to Matthew Watson and Chen Qian during their talk: NLP workflows with Keras.
You can watch it on YouTube 2 or on Twitch 1 at 10:15am PST
Colab notebooks: part 1 10, part 2 6.

Context: Since a lot of people use both keras and pytorch especially while using HuggingFace. Have you guys considered making the code torch code transferrable to keras (to some extent at least) and vice versa so as to ease it for everyone?


Question:
i didn’t know how google search works well, but when i type text i got related images.
same search case i will try to build using AI transformers model.
it involves finding text and image embeddings and find cosine similarity, better score close to 1 leads to result.
lets have fun building
Sample -

Context: There’s a very similar project here already: Use OpenAI's CLIP for image search 4


[Nov 15th Event] Margaret Mitchell: On Values in ML Development

Question:
Use this topic to ask your questions to Margaret Mitchell during her talk: On Values in ML Development.
You can watch it on YouTube 6 or on Twitch 2 at 9:30am PST

Context: How could we balance the filtering of bias between freedom of speech and offensive/biased content?


[Nov 15th Event] Jay Alammar: A gentle visual intro to Transformers models

Question:
Use this topic to ask your questions to Jay Alammar during his talk: A gentle visual intro to Transformers models.
You can watch it on YouTube 9 or on Twitch at 8:45am PST

Context: Can we get the URL Jay was sharing at the start?


Question:
Hi,
I’m doing the second part of the course now, in particular, the chapter “The  Datasets library”. In Part 1, I was following the tensorflow option but it seems that now only the pytorch one is available (when I select tensorflow, it still shows the pytorch-based tutorial). Are you planning to release the tensorflow tutorial for Part 2 also?
Thanks in advance!

Context: Hi Lenn! All the sections have a TensorFlow version. Chapter 5 is completely framework agnostic, that’s why you don’t see any differences between the two, but if you look at chapter 7, you will see the content is very different.


Question:
Please read the topic category description 4 to understand what this is all about

Description
Applications like GitHub’s CoPilot can automatically generate docstrings from a class or function name. The goal of this project is to fine-tune a Transformer like CodeT5 to do this ourselves!

Model(s)
Generating docstrings from source code can be modelled as a sequence-to-sequence task, so T5 models are a good starting point here:

Salesforce/codet5-base 3

Datasets
A good dataset for this task is code_search_net, but feel free to find alternative datasets if you can’t find your favourite programming language there.

Challenges
Models like CodeT5 are rather large and you’ll need to think about what metrics one should use for this type of task.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces that can automatically generate a docstring from a class of function name in your favourite programming language!
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

https://arxiv.org/abs/2109.00859v1 1
https://blog.einstein.ai/codet5/

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 1 and:

Follow the instructions on the #join-course channel

Join the #docstring-generator channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: I find this task interesting and would like to find out more about how to contribute: for example, would fine-tuning for PL to NL (code comment/docstring) generation for Python be a suitable case for this project?


Create a NER tagger for African languages

Question:
Please read the topic category description 2 to understand what this is all about

Description
Africa has over 2,000 spoken languages, but these languages are massively underrepresented in NLP research and datasets. The goal of this project is to train strong models for the MasakhaNER corpus, which is a high quality dataset for named entity recognition in 10 African languages.

Model(s)
There are a few popular multilingual models that you can start with:

xlm-roberta-base
bert-base-multilingual-uncased

Datasets

masakhaner 3

Challenges
It is unlikely that all ten languages in MasakhaNER are represented in multiingual models like XLM-R or mBERT, so some decisions will be need to be made on which subsets to focus on.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that can take text from one or more of the languages in MasakhaNER and extract the person names (PER), organizations (ORG), locations (LOC) and dates & time (DATE) tags.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

MasakhaNER: Named Entity Recognition for African Languages 1

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 2 and:

Follow the instructions on the #join-course channel
Join the african-ner channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Team organization on the Hub
To join this team, make sure you join the following organisation on the Hub:

team-african-ner (🤗 Course Team African NER) 1

Context: I am interested in this project, don’t know African languages, but would be delighted to create something useful for the community, Count me in!


Part II of the  Course goes live in November!

Question:
Hello everyone
We’re super excited to announce that part II of the   course 26 will be released on November 15th

summary2668×807 234 KB

This part of the course takes a deep dive into the  Datasets and  Tokenizers libraries, so you’ll learn how to process huge datasets and train your very own tokenizers on them  . You’ll also learn about the core NLP tasks that Transformers excel at, and even how to debug your training pipelines effectively.
To celebrate the launch, we’re planning a large community event 19 to which you’re all invited! The event involves two days of talks on Nov 15-16 from experts in the field. To register for the event, please fill out this form   : 🤗 Course Community Event 35
After the talks, you will have the opportunity to collaborate on group projects and demo your work as a Streamlit or Gradio app on Hugging Face Spaces 5
We’re looking forward to seeing you at the live event and feel free to post any questions you might have here!

Context: Thank you all for creating these courses; I can’t wait to read part II. I read the first part a few months ago, and it helped me complete my bachelor final project and ever since then, I have been learning more and more about your excellent work, transformers, and NLP.


Question: After following the first section of the course, you should be able to fine-tune a model on a text classification problem and upload it back to the Hub. Share your creations here and if you build a cool app using your model, please let us know!

Context:
It’s not exactly a project, but I’m super excited to share my first public Kaggle dataset

      kaggle.com

Huggingface Modelhub 14

  Dataset containing information on all the models on HuggingFace modelhub

With the help from good folks at HF, I was able to query the metadata information available on model-hub and upload it as a Kaggle dataset.
It should be helpful to anyone looking to analyze and create EDA/Text-processing notebooks on the metadata of publicly available models. The dataset contains the README modelcard data as well.
Please have a look and provide feedback.


Question: Use this topic for any question about Chapter 2 15 of the course.

Context:
In the Handling multiple sequences page of Chapter 2, there is a bug in the code under Attention masks section.
Page: Using 🤗 Transformers - Hugging Face Course 8
The PyTorch toggle is on, but the code uses Tensorflow’s tf.constant function.

Screenshot from 2021-06-15 16-57-42970×796 70.6 KB

There is a typo on https://huggingface.co/course/chapter2/6?fw=pt 1

Screenshot from 2021-06-15 17-12-37917×347 39.9 KB

Isn’t wordpiece a subword algorithm as well?

image976×517 38.6 KB


Question:
Hello … I am following the course but using a different dataset from load_dataset and slight mods.
When I run this code, the accuracy remains constant. I am expecting in the best scenario the accuracy to improve, if not have some variation. But it remains constant between each epoch. Any idea?
from tqdm.auto import tqdm
progress_bar = tqdm(range(num_epochs*num_steps))

for epoch in range(num_epochs):
    model.train()
    for batch in train_dl:
        model_inputs = {k:v.to(device) for k, v in batch.items()}
        outputs = model(**model_inputs)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()
        progress_bar.update(1)

    model.eval()
    metric = load_metric('accuracy')
    for batch in eval_dl:
        model_inputs = {k:v.to(device) for k, v in batch.items()}
        with torch.no_grad():
            outputs = model(**model_inputs)
        logits = outputs.logits
        predictions = torch.argmax(logits, dim=-1)
        metric.add_batch(predictions=predictions, references=model_inputs['labels'])
    print(metric.compute())

Output is:
{‘accuracy’: 0.2112}
{‘accuracy’: 0.2112}
{‘accuracy’: 0.2112}

Context:
This definitely shows your model is not training. A few things to check are:

maybe the learning rate is too high/too low?
maybe there is some problems in your labels and the model can’t learn?


Question:
I’m getting an error that says all my inputs are scalars.  It would be helpful to see a completed working file, as I’m a bit confused about the order.  Here’s my messy code:
#!/bin/env python3

import torch
from transformers import AdamW, AutoModelForSequenceClassification
from datasets import load_dataset
from transformers import AutoTokenizer
from transformers import logging
from torch.utils.data import DataLoader
from transformers import DataCollatorWithPadding
from transformers import TrainingArguments
from transformers import Trainer

logging.set_verbosity_error()
raw_datasets = load_dataset("glue", "mrpc")
checkpoint = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(
    checkpoint,
    num_labels=2,
)

def tokenize_function(example):
    return tokenizer(
        example["sentence1"],
        example["sentence2"],
        truncation=True,
    )

tokenized_datasets = raw_datasets.map(
    tokenize_function,
    num_proc=4,
    batched=True,
)

tokenized_dataset = tokenized_datasets.rename_column("label", "labels")

data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

# train_dataloader = DataLoader(
#     tokenized_dataset["train"],
#     batch_size=16,
#     shuffle=True,
#     collate_fn=data_collator,
# )

training_args = TrainingArguments(
    "test-trainer",
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=5,
    learning_rate=2e-5,
    weight_decay=0.01,
)

trainer = Trainer(
    model,
    training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_datasets["validation"],
    tokenizer=tokenizer,
)

trainer.train()

# optimizer = AdamW(model.parameters())
# loss = model(**batches).loss
# loss.backward()
# optimizer.step()

Context: It’s hard to know what the problem is if you don’t copy the error message as well. I just tried your code and it runs fine on my side.


Question:
Hi,
im just trying to run the code from  the chapter google collaboratory but i keep getting this error:

image913×484 19.9 KB

I am not sure why this is happening.  I tried importing other AutoModel variations and i still get the same error. Please help
the link to the notebook i was trying to run: Google Colaboratory

Context: it works now! thank you to whoever fixed it!


Question: Use this topic for any question about Chapter 4 18 of the course.

Context:
Can someone take a look at the chapter 4 notebook. The .push_to_hub() method isn’t working, at first the error was about ‘git-lfs’ and even downloading that doesn’t seem to work…
Thanks


Question: I heard the udemy is good place to finds course, but I cant access it will a vpn can help.

Context:
Besides udemy, you can also try Coursera and Udacity

      udacity.com

Explore our Programs and Courses | Udacity Catalog 3

  Get the latest tech skills to advance your career. Browse Nanodegree programs in AI, automated systems & robotics, data science, programming and business.

      Coursera

Coursera | Build Skills with Online Courses from Top Institutions 2

  Join Coursera for free and learn online. Build skills with courses from top universities like Yale, Michigan, Stanford, and leading companies like Google and IBM. Advance your career with degrees, certificates, Specializations, & MOOCs in data...


Question:
To launch the course, the team developing it will be hosting live sessions where you can ask any questions. Those live sessions will be on Twitch at the following times (the two sessions for each chapter will cover the same material):

Chapter 1
Chapter 1 with Lysandre 253: Wednesday, June 16th (8:00-9:00 UTC)
Chapter 1 with Sylvain 118: Thursday, June 17th (18:00-19:00 UTC)

Chapter 2
Chapter 2, with Lewis 29: Wednesday, June 23rd (8:00-9:00 UTC)
Chapter 2, with Sylvain 68: Thursday, June 24th (18:00-19:00 UTC)

Chapter 3
Chapter 3 — PyTorch, with Lewis 39: Wednesday, June 30th (8:00-9:00 UTC)
Chapter 3 — PyTorch, with Sylvain 38: Thursday, July 1st (18:00-19:00 UTC)
Chapter 3 — TensorFlow, with Matt 26: Wednesday, June 30th (17:00-18:00 UTC)
Chapter 3 — TensorFlow, with Matt 23: Thursday, July 1st (10:00-11:00 UTC)

Chapter 4
Chapter 4, with Omar 29: Wednesday, July 7th (8:00-9:00 UTC)
Chapter 4, with Omar 29: Thursday, July 8th (18:00-19:00 UTC)

Context: Hi @sgugger , will this be recorded and reshare on youtube?


Question: Use this topic for any questions related to Chapter 0 10 of the course.

Context: Hi @sgugger, thank you for the course. Is it advisable to use a dockerized container as a coding environment on a windows machine? Or inside a WSL2 layer? Or does something or the other may create issues later? (Asking because I have faced a lot of issues with NCCL failure while trying to run XGboost et al on a RAPIDS installation on WSL2).


Huggingface course study group with a fast.ai bent

Question:
Fill out the interest form here:  https://forms.gle/ZwZ6oVhsq3vSM5xf8 24
We’ll be tracking with the official course topic wise, but looking at how to use fast.ai and blurr to train and deploy models … from DataBlocks to inference and all else in between.  Fill out the form and I’ll be following up with those who do with details soon.

Context:
We’ll be announcing dates end of this week.  Please respond to this poll as to your preferred start time (either Saturday or Sundays):

      twitter.com

Wayde Gilliam 1
@waydegilliam

  Thanks to all who filled out the @fastdotai'sh @huggingface course study group interest form!  We'll be announcing dates (either Saturday or Sundays) at the end of this week with things kicking off early July.

Please respond to this poll as to which time works best:

  11:58 AM - 23 Jun 2021


Question: Use this topic for any question about Chapter 1 36 of the course.

Context:
Small mismatch?
From the widget in roberta-large-mnli · Hugging Face 2 I see the classification is between “CONTRADICTION”, “NEUTRAL”, “ENTAILMENT”

Capture d’écran de 2021-06-16 11-42-001280×584 40.8 KB


Question:
Everyone, first of all the huggingface course notes are very nice. Good luck to everyone.
Together with our friends (@basakbuluz @ayyucekizrak 2 ) we are working voluntarily to increase Turkish open source content in the field of machine learning.
We also compile these resources on our YZAI website 2 for access from one place.
We can contribute primarily to the translation of Hugginface Course contents into Turkish.
As a method for translation, for example, the translation repository of ML notes by Amidi brothers from Stanford 6 is very convenient and provides a nice guide.
Can such an infrastructure be provided and is it possible to contribute to other languages?
Thanks.

Context: Thanks a lot for reaching out! We are in the process of open-sourcing the content of the course, which will make it easier to translate the material as well as the video subtitles! It’s probably going to take a few weeks though, but I’ll ping here when it’s done!


Question: is the course meant to be a summarization and distillation of the tutorials or substantially different? either way i’m really excited about it.

Context:
hey @TheIneffableALIAS the course is complementary to the official tutorials 9 as it assumes less prior knowledge and provides “bite-sized” chunks of information through videos and simple examples.
if you’re brand new to transformers (or NLP in general), my suggestion would be to start with the course and then dive into the official tutorials to see more advanced usage


Question:
I did a quick review of the course. I look forward to see the other chapters being published. Any timeline for release?
Dive into Transformers - HuggingFace Free Course [Overview]

Context: Thanks for sharing the review! The next part of the course is scheduled for the fall.


Question:
Why do we need to apply softmax after getting the logit values? I know it says that it would help to normalise the scores and get a probabilistic interpretation. But is it not that that the utility of logits/softmax scores is to determine which value is bigger and then infer the label.
For example, if you get a logits score of [-4.2095,  4.6053], where -4.2095 refers to label0  and 4.6053 refers to label1. Then as 4.6053 > -4.2095, I would keep label1 as my prediction. If instead I apply softmax to the logits score, I get [1.4850e-04, 9.9985e-01]. With this softmax score I will still infer that the predicted label is label1.

Context: If you just want to get the predicted class, you don’t need the softmax layer as, as you pointed out, you just have to take the index of the maximum logits. The softmax will convert the logits into probabilities, so you should use it when you want the probabilities for each prediction.


[Suggestions and Guidance]Finetuning Bert models for Next word Prediction

Question:
Problem Statement : To produce a next word prediction model on legal text. The aim is to build an autocomplete model which will make use of existing typed text as well as a possible concatenation of vectors from prior clauses/paragraphs.
Current Approach: Because Bert based model are based on masked language, pretrained models such as LegalBert did not produce good accuracy for prediction of next word when the word to be predicted was marked as [MASK]. Here is an example sentence, “use of [MASK]” where “marked” is the next word to be predicted in place of “[MASK]” token. (Note that there would not be words present after the mask token, only before the token).
Currently approaching the problem as a SequenceClassification problem where labels are the token ids of the words that are to be predicted next. Will also attempt to finetune gpt2 on the legal text using run_clm.py from huggingface examples directory
Is there a better way to approach this problem of next word prediction?
Any suggestions and guidance would be welcome.
Thank you in advance

Context:
Hi Sumanth! I believe you are already on the right track by finetuning gpt2. The difference is that GPT was trained using causal/autoregressive attention. It means that GPT is specifically trained to predict the next word without having access to the word to the right of the masked token (unlike BERT).
The different models and their architectures are depicted in this chart:

Capture684×642 56.9 KB

Long story short - you should see better results with GPT2. Let us know how it goes.
Cheers
Heiko


Paper Notes: Deepspeed Mixture of Experts

Question:
Summary
The legends over at DeepSpeed released a paper 9 on scaling Mixture of Experts with a bunch of cool ideas.
Since they will probably release some pytorch code soon I wanted to summarize/discuss the findings so that I learn them better.

I provide 0 background on Mixture of Experts, assume knowledge of Top1 vs Top2 gating, for selfish/lazy reasons. Read the deepspeed blog post 1 for background.
I abstract the term “acc” to encompass all types of metrics: validation perplexity, zero shot accuracy, etc.
I used @srush trick of trying to read critically (to get your brain to think harder about other peoples’ results) but I don’t want to come off as too negative. I really enjoyed this paper and am excited to read the code!

The DeepSpeed team proposes:

(a) (sec 4.1) architectural modifications that reduce the number of experts without hurting acc.
(b) (sec 4.1) Moe 2 Moe distillation, (instead of MoE 2 dense distillation like the FAIR paper (appendix Table 9) and the Switch paper)
(c) (sec 5) Systems Optimizations to make inference fast

Improved Communication Collectives for MoE Inference (hierarchical all2all)

tutel style single-device kernels to make routing tokens to experts fast.
4D parallelism!?

I now cover architecture and distillation, and save systems optimizations for later because I don’t fully understand them yet.

Architecture: Pyramid Residual MoE
This section is really well written. It contains two very nice ablations that motivated the changes:
Phenomenon 1: “Pyramid”

We compare the performance of two different half-MoE architectures. More specifically, we put MoE layers in the first half of the model and leave the second half’s layers identical to the dense model. We switch the MoE layers to the second half and use dense at the first half. The results show that deeper layers benefit more from large number of experts.

This also saves a ton of parameters: 40% reduction at 1.3B dense equivalent size, which will be useful at inference time.
Phenomenon 2: “Residual”

we can achieve the benefit of using two experts per layer but still use one communication.

They frame this as trying to get the benefits of top2 routing without the costs.
But, basically MoeLayers become only half sparse – a dense ffn that process the input as does 1 expert – the results are added.
Compared to top2 where 2 different sparse experts process the input, this is cheaper because there is less communication (you only need to send the input to 1 place instead of 2?)
Note this does not improve acc compared to top2, just speed.
Putting it all together:
FAIR arch (see table 1) (52B Params)

Layers: top2 gating (each token gets routed to 2 experts)
512 experts at each MoE layer

Deepspeed Arch: (31B params)

Layers: each token processed by dense FFN and 1 expert (same FLOPs as top2 gating if same number of experts, I believe).
pyramid: somewhere between 32 and 128 experts at each Moe layer – way fewer params!

In terms of acc, (PIQA is the only overlapping evaluation),
the 31B Deepspeed performs between the FAIR 52B and the FAIR 207B and was probably lower training cost than the 52B, even before all the systems optimizations in section 5. Nice!
With the systems optimizations they say training is 5x faster than dense (to the same acc).  The FAIR paper says “4x faster than dense”, but measures TFLOPS, which make the extra communication required for MoE appear to be free. So all in all this definitely seems like a better architecture.
It would have been cool if Tables 2,4 had training cost and inference cost next to the few shot performances (or 1 big joined table somewhere!).

Staged Knowledge Distillation: Mixture Of Students (MoS)
Caveat before you read this section: in most distillation results, the student model is MUCH smaller than the teacher model, like half as large or so. Here, the student model is only 12.5% smaller than the teacher model. (3 fewer layers, 4B fewer params (31B vs 27B)).
They are able to lose very little performance, which is nice, but they also didn’t really lose that much weight, and it would be interesting to try to replicate what they did with smaller students.
Caveat 2: name deeply misleading. It’s normal KD but they switch to cross entropy loss halfway through that’s it!
Anyways, these are the first published MoE 2 MoE Distillation results.  The switch paper and FAIR paper both distill Moe 2 Dense models (since they are much easier to serve than MoE models, a gap deepspeed claims to eliminate in section 5 – the one I don’t understand yet:( ).
They use the same KD loss as the other papers, but they turn it off halfway through training.
They say this improves acc, but I am most interested in the speed implications. I tried MoE2MoE  distillation but  it was extremely slow (like 10x slower than Dense2Dense) because of teacher inference every step.
If we could only run the teacher forward pass for part of the student training, that would be sweet!

Next
Let me know any inaccuracies, important omissions, what you ate for lunch follow up ideas!
Next week I will try to tackle Section 5 (Systems optimizations) and if I don’t I will burn a 20 dollar bill and record it!

Context: What is 4D parallelism?


Question:
Hello everyone!
I tried to apply the technique of data augmentation, mixup, popularly used on computer vision, but in this case on NLP.
The algorithm developed is in two phases:
The first phase gets the representation for each sentence of the batch, computing the mean of the correspondent hidden states of the last layer. The fragment below shows the corresponding module.
class LanguageModel(nn.Module):

  def __init__(self, pretrained_model_name, device="cuda:0", anonymized_tokens=False):
    super(LanguageModel, self).__init__()
    # Load tokenizer
    self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name)
    # Load model
    self.config = AutoConfig.from_pretrained(pretrained_model_name)
    self.config.output_hidden_states = True
    self.model = AutoModel.from_pretrained(pretrained_model_name, config=self.config).to(device)

  def forward(self, input_ids, attention_mask):
    outputs = self.model(
        input_ids=input_ids,
        attention_mask=attention_mask,
    )
    activations = torch.mean(outputs[0], axis=1)
    return activations

After that, it generates the mixup examples using the function proposed on the original code 3, but  being  the input, the representations computed on the previous step, instead the images, like originally.
One time the mixup examples are generated, the second phase realizes the predictions (the fragment below shows the corresponding module). Finally, is computed the loss, in the same way as in the original work.
class ClassifierLayer(nn.Module):

  def __init__(self, num_classes, dropout_rate=0.1, petrained_size=768, device="cuda:0"):
    super(ClassifierLayer, self).__init__()
    self.layer = nn.Linear(petrained_size, num_classes, bias=True).to(device)
    self.drop = nn.Dropout(dropout_rate)

  def forward(self, z):
    activations = self.layer(self.drop(z))
    return activations

In the fragment of the code below, is shown a summary of the training loop proposed, however the full script used is here 3:
    for idx_epoch in range(0, args.num_train_epochs):
        language_model.train()
        classifier_layer.train()
        accs = 0; ps = 0; rs = 0; f1s = 0; lss = 0
        for (idx_batch, train_batch) in enumerate(train_dataloader):
            # 0: input_ids, 1: attention_mask, 2:token_type_ids, 3: labels
            batch_train = tuple(data_.to(device) for data_ in train_batch)
            labels_train = batch_train[-1]
            inputs = {
                'input_ids': batch_train[0],
                'attention_mask': batch_train[1],
            }
            optimizer.zero_grad()
            # 1st phase: conextual embeddings
            contextual_embeddings = language_model(
                input_ids=inputs['input_ids'],
                attention_mask=inputs['attention_mask'],
            )
            # 2nd phase: mixup
            inputs, targets_a, targets_b, lam = mixup_data(contextual_embeddings, labels_train, args.alpha_mixup, use_cuda)
            inputs, targets_a, targets_b = map(Variable, (inputs, targets_a, targets_b))
            predictions = classifier_layer(inputs)
            loss = mixup_criterion(criterion, predictions, targets_a, targets_b, lam)

            # 2nd phase: standard
            # predictions = classifier_layer(contextual_embeddings)
            # loss = criterion(predictions, labels_train)

            lss += loss
            loss.backward()
            optimizer.step()
            scheduler.step()

Experimenting with this approach, the results obtained are very poor…
Have any of you worked on an approximation similar to this one with successful/good results?
Thanks.

Context:
Hi @franborjavalero!
This is really interesting. I remember @sgugger got a little bump using mixup after embeddings with ULMFiT. Would be really awesome to share this code as implementation for this is not trivial.


Question:
Because I love the mindset within the community of the Wave2vec sprint I’d like to share some ideas about improving the accuracy of asr and making more stable for production.
I would be happy to discuss about.
In some experiments I tested many systems and algorithms, but especially one reached amazing accuracy.
When having the transcribed text from the wave2vec model we go many ways to correct them. Either dictionary search for each word an automatically use the nearest result, or an seq2seq model, but what about an hybrid solution based on two or three parts ?
Part 1: token classification, to recognize which words are wrong in the context. Instead of human names or locations just classify wrong or right.
Part 2: When we have the wrong tokens let’s check an dictionary for similar alternative, either using bm25 (tested) or dpr neural search (untested)
Part 3: When we have some alternatives for each token we can either use the best scored result or let an multiple-choice trained model decide. In my quick tests I decided using the best alternative, but definitely need to check the multiple choice variant.
With this 3 steps

Token classification
Dictionary search using bm25 like algorithms
Replacing false tokens with best scored alternative
I reached amazing results and up to WER of 1.3%

At the moment my code is pretty noisy and I would like to start from zero again to build an clean library based on huggingface models, or maybe just an community notebook, depends on your feedback
I’d like to hear what you think about, maybe you have much better idea ?
Maybe someone is interested in joining this research ?

Context: Amazing idea. I would love this. Do you have any code I can check out?


Question:
Hello.
HF community members
I wonder how do you think about the copying mechanism for transformer.
I can see very few papers/tech reports implementing copying mechanism for transformer.

      aclweb.org

2020.acl-main.125.pdf 71

816.26 KB

      web.stanford.edu

15784595.pdf 53

256.28 KB

Also, I couldn’t find anyone who discusses copying mechanism in this forum.
Personally, I am stuck with computing ‘generating-copying switch’ since transformer does not have explicit ‘context vector’ in RNN.
Do you have any thoughts about the lack of reference/discussion for copying mechanism?
Is it worth implement & contribute to HF community with copying mechanism?

Context:
Hi,
I have tried a copy mechanism in the BART model. I directly utilize the cross-attention as the attention score for the source samples. This idea is from openNMT CopyGenerator 69.
My implementation is like this:
def copy_mechanism_v3(self, logits, cross_attentions, decoder_hidden_states, encoder_input_ids):
    last_hidden_state = decoder_hidden_states[-1]
    last_attention_weight = cross_attentions[-1]
    # context_vector shape: batch_size, decoder_length, hidden_size
    p_copy = torch.sigmoid(self.linear_copy(last_hidden_state))
    previous_word_pro = torch.softmax(logits, dim=-1) * (1 - p_copy)
    encoder_word_attention = p_copy * torch.mean(last_attention_weight, dim=1)

    # did not copy the pad
    mask = torch.where(encoder_input_ids == 1,
                       encoder_word_attention.new_zeros(encoder_input_ids.shape),
                       encoder_word_attention.new_ones(encoder_input_ids.shape))
    encoder_word_attention = encoder_word_attention * mask.unsqueeze(1)

    personal_words = encoder_input_ids.unsqueeze(1).repeat(1, encoder_word_attention.shape[1], 1)
    word_pro = torch.scatter_add(previous_word_pro, 2, personal_words, encoder_word_attention)
    return word_pro


Guide: The best way to calculate the perplexity of fixed-length models

Question:
Hey all. Just thought you might be interested in a page I just added to the research docs on the perplexity of fixed-length models 108.
Perplexity (PPL) is defined as the exponential average of a sequence’s negative log likelihoods. For a t-length sequence X, this is defined,

\text{PPL}(X) = \exp \left\{ -\frac{1}{t} \sum_i^t \log p_\theta (x_i|x_{<i})  \right\}

But with fixed-length models (like most transformers), we can’t always condition on the entire preceding subsequence when predicting each token.
The initial instinct for many in dealing with this problem is to break the whole sequence into segments equal to the model’s max input size and calculate the likelihoods of each segment independently. This not the best approach, however, since it gives the model very little context to use for prediction at the beginning of each segment. I’ll illustrate this with the following gif where we imagine a model with a max input size of 6 adding up the log-likelihoods for the sentence, “Hugging Face is a startup based in New York City and Paris”
ppl_chunked1200×160 352 KB
When the model starts the second segment, it has to try to predict the word “in” without any context, even though we have 5 words before it that the model could be using (since we said the max input size is 6).
A better approach is to instead employ a sliding window strategy, where you continually move the context across the sequence, allowing the model to take advantage of the available context.
ppl_sliding1200×160 373 KB
This is slower to compute, but will typically yield better scores and is actually much closer to the way the sequence probabilities are formally decomposed (e.g. see the the equation above).
In the guide 108, we show how to do this in a strided way with GPT-2. When using the first, naive approach, GPT-2 gets a PPL of 19.64 on WikiText-2. In contrast, when we use a strided sliding window, this score improves dramatically down to 16.53.

Context:
Hi, I have a question about the perplexity calculation from the guide 30.
Why do we divide by i in the example, see ppl = torch.exp(torch.stack(lls).sum() / i)?
If you have a codebase or paper that exemplifies this behaviour could you please share it?
Thanks!


Text similarity not by cosine similarity

Question:
Hi all,
I have a question.
I have a dataset containing questions and answers from a specific domain. My goal is to find the find the X most similar questions to a query.
for example:
user: “What is python?”
dataset questions: [“What is python?”, “What does python means?”, “Is it python?”, “Is it a python snake?”, “Is it a python?”]
I tried encoding the questions to embeddings and calculate the cosine similarity but the problem is it gives me high similarity score for “Is it python?” for the query “What is python?” which is clearly not the same question meaning and for “What does python means?” get very low score compared to “Is it python?”
Any suggestions how i can overcome this problem? maybe new approaches…

Context:
if cosine similarity is not giving you the results you want, you could try a different metric like euclidean / manhattan / minkowski distance or jaccard similarity.
alternatively you could try changing the embedding model to see if that improves the comparisons


Improving performance of Wav2Vec2 fine tuning with word piece vocabulary

Question:
Hello,
I’m fine tuning XLSR-Wav2Vec2 on a 200+ hours of a speech in a language not in the original pertaining.
The training progresses nicely, however when it reaches about 40 WER it starts to overfit (WER doesn’t progress much and train loss decreases while eval loss is going up).
I’ve tried increasing some params of the SpecAugment, but it only helped a bit.
I’ve noticed that using the Speechbrain lib implementation I’m getting a bit better results (on the expense of training stability) and was wondering if it is due to a larger vocabulary they use there. Does anyone tried to use a tokenizer with a vocabulary that contains subwords and words in addition to characters? I could’t find any experiment that uses it with Huggingface transformers W2V2.
I see in the Wav2Vec 2 paper they say that:

We expect performance gains by switching to a seq2seq architecture and a
word piece vocabulary.
https://arxiv.org/pdf/2006.11477.pdf 5

Any suggestions on how to do that with Huggingface Transformers?
P.S. my dataset is noisy and not super clean.
Any help or suggestion will be very helpful.
Samuel

Context: Not sure how I’d switch to a seq2seq architecture, but for word piece, I think you just need to change the vocab passed to the Wav2Vec2CTCTokenizer. Instead of the individual alphabet characters used for the vocab in the XLSR example, you’d need to use the wordpiece/BPE algorithm on your language text data and pass that through.


[Help needed] Extending Trainer for Meta learning

Question:
I want to implement MAML with Glue dataset with transformers. In my case, query and support set will come from the same dataset. I’ve read some work in meta learning from HF team (Wolf et al., 18).
Although I’ve implemented my training loop (with higher) (open for other methods as well), I am still looking for a correct reference implementation of MAML or Reptile to confirm. Currently my code inherits from Trainer. If anyone share a sample snippet that would perform MAML gradient updates, that’d be really helpful ?

Context:
So the MetaDataset 13 wraps any GlueDataset to give a list containing all classes when meta_dataset[0] is called. So this will become, num_of_classes (N) way K shot example.
I’ve written this, which extends Trainer for MAML.
def train(self):

        self.create_optimizer_and_scheduler(
            int(
                len(self.train_dataloader)
                // self.args.gradient_accumulation_steps
                * self.args.num_train_epochs
            )
        )

        logger.info("***** Running training *****")

        self.global_step = 0
        self.epoch = 0

        eval_step = [2 ** i for i in range(1, 20)]
        inner_optimizer = torch.optim.SGD(
            self.model.parameters(), lr=self.args.step_size
        )
        self.model.train()

        tqdm_iterator = tqdm(self.train_dataloader, desc="Batch Index")

        #  n_inner_iter = 5
        self.optimizer.zero_grad()
        query_dataloader = iter(self.train_dataloader)

        for batch_idx, meta_batch in enumerate(tqdm_iterator):
            target_batch = next(query_dataloader)
            outer_loss = 0.0
            # Loop through all classes
            for inputs, target_inputs in zip(meta_batch, target_batch):

                for k, v in inputs.items():
                    inputs[k] = v.to(self.args.device)
                    target_inputs[k] = v.to(self.args.device)

                with higher.innerloop_ctx(
                    self.model, inner_optimizer, copy_initial_weights=False
                ) as (fmodel, diffopt):

                    inner_loss = fmodel(**inputs)[0]
                    diffopt.step(inner_loss)
                    outer_loss += fmodel(**target_inputs)[0]

            self.global_step += 1
            self.optimizer.step()

            outer_loss.backward()

            if (batch_idx + 1) % self.args.gradient_accumulation_steps == 0:
                torch.nn.utils.clip_grad_norm_(
                    self.model.parameters(), self.args.max_grad_norm
                )

            # Run evaluation on task list
            if self.global_step in eval_step:
                output = self.prediction_loop(self.eval_dataloader, description = "Evaluation")
                self.log(output.metrics)

                output_dir = os.path.join(
                    self.args.output_dir, f"{PREFIX_CHECKPOINT_DIR}-{self.global_step}",
                )
                self.save_model(output_dir)


Why are huge batch sizes used for pretraining and small ones for finetuning?

Question:
In most, if not all papers on language models, I find that they often use very large batch sizes for pretraining on a language modeling task. But when they then finetune their model to show its performance on downstream tasks, the batch sizes are suddenly very small.
For instance, the RoBERTa paper shows that its batch size during pretraining was 8k sentences (Table 9 in the appendix), however for finetuning the batches are considerably smaller (Table 10, appendix): 16 (RACE), 48 (SQuAD), 16, 32 (GLUE).
This has puzzled me since forever and I have never discovered the rationale behind this. Is it a matter of scale? Something like: while pretraining you have so much different data, that you just want as much in one go as you can - it does not matter as much that the loss is smoothed out (averaged) over such huge batches. But when finetuning over a smaller dataset you do not want to average the loss over too much of the dataset at once because you then lose peculiarities of samples quickly.
Or is there another reason? All ideas are welcome.

Context:
I don’t think they use the same hardware for pretraining and fine-tuning. E.g. multiple TPU pods or a GPU cluster for pretraining allows a big batch size but that’s maybe something the research team can only do once. Fine-tuning, and something more accessible (just one GPU for instance) then requires a smaller batch size to avoid the OOM.
This is just a guess however.


ELECTRA training reimplementation and discussion

Question:
After months of development and debugging, I finally successfully train a model from scratch and replicate the official results.

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators 115
by Kevin Clark. Minh-Thang Luong. Quoc V. Le. Christopher D. Manning

 Code: electra_pytorch 249

AFAIK, the closest reimplementation to the original one, taking care of many easily overlooked details (described below).

AFAIK, the only one successfully validate itself by replicating the results in the paper.

Comes with jupyter notebooks, which you can explore the code and inspect the processed data.

You don’t need to download and preprocess anything by yourself, all you need is running the training script.

Replicated Results
I pretrain ELECTRA-small from scratch and have successfully replicated the paper’s results on GLUE.

Model
CoLA
SST
MRPC
STS
QQP
MNLI
QNLI
RTE
Avg. of Avg.

ELECTRA-Small-OWT
56.8
88.3
87.4
86.8
88.3
78.9
87.9
68.5
80.36

ELECTRA-Small-OWT (my)
58.72
88.03
86.04
86.16
88.63
80.4
87.45
67.46
80.36

Table 1: Results on GLUE dev set. The official result comes from expected results 3. Scores are the average scores finetuned from the same checkpoint. (See this issue 3) My result comes from pretraining a model from scratch and thens taking average from 10 finetuning runs for each task. Both results are trained on OpenWebText corpus

Model
CoLA
SST
MRPC
STS
QQP
MNLI
QNLI
RTE
Avg.

ELECTRA-Small++
55.6
91.1
84.9
84.6
88.0
81.6
88.3
6.36
79.7

ELECTRA-Small++ (my)
54.8
91.6
84.6
84.2
88.5
82
89
64.7
79.92

Table 2: Results on GLUE test set. My result finetunes the pretrained checkpoint loaded from huggingface.

Official training loss curve
My training loss curve

Table 3: Both are small models trained on OpenWebText. The official one is from here 3. You should take the value of training loss with a grain of salt since it doesn’t reflect the performance of downstream tasks.
More results
How stable is ELECTRA pretraining?

Mean
Std
Max
Min
#models

81.38
0.57
82.23
80.42
14

Tabel 4: Statistics of GLUE devset results for small models. Every model is pretrained from scratch with different seeds and finetuned for 10 random runs for each GLUE task. Score of a model is the average of the best of 10 for each task. (The process is as same as the one described in the paper) As we can see, although ELECTRA is mocking adeversarial training, it has a good training stability.
How stable is ELECTRA finetuing on GLUE ?

Model
CoLA
SST
MRPC
STS
QQP
MNLI
QNLI
RTE

ELECTRA-Small-OWT (my)
1.30
0.49
0.7
0.29
0.1
0.15
0.33
1.93

Table 5: Standard deviation for each task. This is the same model as Table 1, which finetunes 10 runs for each task.
Advanced details  (Skip it if you want)
elow lists the details of the original implementation 8/paper that are easy to be overlooked and I have taken care of. I found these details are indispensable to successfully replicate the results of the paper.
Optimization

Using Adam optimizer without bias correction (bias correction is default for Adam optimizer in Pytorch and fastai)
There is a bug of decaying learning rates through layers in the official implementation , so that when finetuing, lr decays more than the stated in the paper. See _get_layer_lrs 12. Also see this issue 7.
Using clip gradient
using 0 weight decay when finetuning on GLUE
It didn’t do warmup and then do linear decay but do them together, which means the learning rate warmups and decays at the same time during the warming up phase. See here 11

Data processing

For pretraing data preprocessing, it concatenates and truncates setences to fit the max length, and stops concating when it comes to the end of a document.
For pretraing data preprocessing, it by chance splits the text into sentence A and sentence B, and also by chance changes the max length
For finetuning data preprocessing, it follow BERT’s way to truncate the longest one of sentence A and B to fit the max length

Trick

For MRPC and STS tasks, it augments training data by add the same training data but with swapped sentence A and B. This is called “double_unordered” in the official implementation.
It didn’t mask sentence like BERT, within the mask probability (15% or other value) of tokens,  a token has 85% chance to be replaced with [MASK] and 15% remains the same but no chance to be replaced with a random token.

Tying parameter

Input and output word embeddings of generator, and input word embeddings of discriminator. The three are tied together.
It tie not only word/pos/token type embeddings but also layer norm in the embedding layers of both generator and discriminator.

Other

The output layer is initialized by Tensorflow v1’s default initialization (i.e. xavier uniform)
Using gumbel softmax to sample generations from geneartor as input of discriminator
It use a dropout and a linear layer in the output layer for GLUE finetuning, not what ElectraClassificationHead uses.
All public model of ELECTRA checkpoints are actually ++ model. See this issue 8

It downscales generator by hidden_size, number of attention heads, and intermediate size, but not number of layers.

Need your help

Please consider help us on the problems listed below, or tag someone else you think might help.

Haven’t success to replicate results of WNLI trick for ELECTRA-Large described in the paper.

When I finetune on GLUE (using finetune.py), GPU-util is only about 30-40%. I suspect the reason to be small batch and model size (forward pass only takes 1ms) or slow cpu speed ?

About more
The updates of this reimplementation and other tools I created will be tweeted on my Twitter Richard Wang 21.
Also my personal research based on ELECTRA is underway, hope I can share some good results on Twitter then.

Context: This is awesome !


Question:
Are there plans to implement online decoding for the speech recognition models such as wav2vec2 and XLSR? More specifically, to be able to receive audio in short chunks, and output partial transcripts as they become available.
Motivation
Many use cases are covered by the current wav2vec2 model in the library, involving batch recognition of pre-recorded text. However for an online application that wanted to continuously recognize speech on a live input stream, this may not be sufficient.

Context: I would very much like to know whether this is possible too! Have you gotten any further on this, @arkadyark?


Question:
Hi,
I’ve been using the HuggingFace library for quite sometime now. I go by the tutorials, swap the tutorial data with my project data and get very good results. I wanted to dig into a little bit deeper into how the classification happens by BERT and BERT-based models. I’m not able to understand a key significant feature - the [CLS] token which is responsible for the actual classification. I hope smart people here could answer my questions because I’m unable to find them on my own.
When I searched for what the [CLS] token actually represent, most of the results indicate that “it is an aggregate representation of the sequence”. I can understand this part. Basically before BERT, people have used different techniques to represent documents ranging from averaging the word vectors of the document to computing document vectors using doc2vec. I can also understand that stacking a linear classification and feeding in the values for the [CLS] token (768 dim for a bert-base-uncased model), we can end up classifying the sequence.
Here are my questions:

Is my above understanding of the [CLS] token correct?
Why is it always the first token? Why not the second, third or last? Did the authors of the original BERT paper get it to be the first token by trial and error?
How exactly does it “learn” the representation of the sequence? I mean its basically trained in the same way as the other input tokens in the sequence, so what makes it special to represent the entire sequence? I couldn’t find any explanation to this question from either the paper or my search afterwards.
Is it at all possible to get back the original sequence using the [CLS] token (I think not but worth asking)?

I hope I can find some answers to these questions (or at least pointers to resources where I can find them). Please let me know if this is not correct place to post these questions and where I should post them.
Thank you.

Context: I would love to hear from others!


Implementing a custom Attention Transformer

Question: Hello everyone, currently I am trying to implement a custom attention transformer, whose attention is given on Page No. 4 of this link 12. They have used hugging face for the implementation, and I am not sure about how to go for approaching this problem, and how to use hugging face to implement custom attention. Can anybody guide me, about how to go about implementing this? Thanks,

Context:
Hey @iakarshu my best guess is that the authors implemented DocFormer from scratch, so as far as I can tell you can’t do some clever subclassing of an existing model to tweak the attention layers.
Having said that, you could look at the implementation of LayoutLMV2 4 which seems to share a similar approach and you can use this template 4 to get all the basic modeling files.
Do you know if AWS open-sourced the pretrained weights of DocFormer? Without them, you might need a lot of compute to build a useful model.
Hope that helps!


Question:
Hello.
How is it customary to cite / credit a language model from huggingface in an academical paper,
when the model does not have a publication of itself? Any examples?
Thanks!

Context:
Hi @Secret, for now you can use the model’s URL (see How can I use BibTeX to cite a web page? - TeX - LaTeX Stack Exchange 34), and we are working with @lysandre and others on plugging a https://www.doi.org/ 3 system into the platform
Let us know if this helps


The (hidden) meaning behind the embedding of the padding token?

Question:
So noticed that the transformers contain different embeddings for PAD tokens, and I know pad tokens typically are simply ignored for the most part (if at all present). However, as a forward pass using a batch typically contain dozens of padding tokens it would be interesting to see if these in fact hold any meaningful information (as padding tokens do attend to the sequence). Does anyone know of any research which has been conducted on what information might be present here?
One might legitimately ask why this is relevant isn’t padding tokens simply a convenience for efficient processing because we need the same tensor shape? This is naturally correct, but quite a few studies have clustered the sentence embedding and it seems relevant to ask what influence the padding embeddings have on this.
For a short demonstration that they indeed have different embeddings:
import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained(
    "bert-base-uncased")
model = transformers.BertModel.from_pretrained(
    "bert-base-uncased")

input_ = tokenizer(["this is a sample sentence"], return_tensors="pt",
                   # add some padding
                   padding="max_length", max_length=128, truncation=True)
output = model(**input_)

# extract padding token embedding
pad_tok_id = [i for i, t in enumerate(input_["input_ids"][0]) if t == 0]
embedding_pad1 = output[0][0][pad_tok_id[0]]
embedding_pad2 = output[0][0][pad_tok_id[1]]

embedding_pad1.shape #embedding size
embedding_pad1[0:10]
embedding_pad2[0:10]

tensor([-0.5072, -0.4916, -0.1021, -0.1485, -0.4096,  0.0536, -0.1111,  0.0525,
        -0.0748, -0.4794], grad_fn=<SliceBackward>)

tensor([-0.6447, -0.5780, -0.1062, -0.1869, -0.3671,  0.0763, -0.0486,  0.0202,
        -0.1334, -0.5716], grad_fn=<SliceBackward>)

Context:
@KennethEnevoldsen  I have been thinking about the same a while ago.
You have a point with different embeddings for pad tokens. But, to my understanding these never interfere with any part of model’s computation  (like, self attention), since the pad tokens are always masked using the attention masks.
Would you have an example of where the pad token embeddings could make a difference, given the attention mask?


Language model to search an answer in a huge collection of (unrelated) paragraphs

Question:
I want to build a question/answer language model to search a large collection of paragraphs.
Say 10k paragraphs. And find relevant  answers in them.
There are 2 issues I don’t know how to solve.

existing solutions often identify an answer from a short paragraph. I don’t know how to deal with a lot of paragraphs. A naive approach would be going through each paragraph and identify an answer in each of them.

existing solutions will generate an answer even when fed with an unrelated paragraph. they don’t give a confidence number. If I have 10k paragraphs to search an answer from, and only 3 paragraphs have an answer, using existing solutions won’t let me to rule out unrelated paragraphs.

Is there a way to generate a document embedding first (using both a question and a paragraph ), and I can use the embedding to find candidate paragraphs first and then do the actual answer search. And when there is no answer, I’d like to get a confidence number that 's below my answer threshold.
Are there any papers dealing with this problem?

Context:
DPR & RAG may be the references you want.
Regarding your questions and my answers with DPR

      huggingface.co

DPR — transformers 3.5.0 documentation 5

DPR (retriever module) select top-k paragraphs from 20 million of possible wikipedia paragraphs (not just 10k, and you can also make your own corpus) using very fast MIPS (maximum inner product search) implemented by FAISS

DPR (reader module) produce a relevance score for each of the top-k passages so this is a confidence number that you mentioned

Finally, RAG is an improvement of DPR where (1) you can combine different passages directly (both relevance and irrelevance) to produce the final answer by “marginalization” and (2) Final answer is generated in free-form, not necessarily contained in any of the passage .
(Please see the paper for details
https://huggingface.co/transformers/model_doc/rag.html 12 )


Seq2Seq Distillation: Methodology Questions

Question: This thread should be used to ask questions about how examples/seq2seq/distillation.py works, and to ask questions about the associated paper after it gets released.

Context:
What is the reasoning behind choosing alternating layers ?
no teacher distillation scores for XSUM ?
no teacher is working for non seq-2-seq task as well as we saw with MNLI, should we also see if it works other tasks as well ?


Question:
t5 and pegasus don’t really work in fp16 because they create activations that overflow fp16 bits. (they were trained in bfloat 16 which has larger range) Has anyone read/seen/heard anything about finetuning/scaling models so that their activations can fit in fp16. (or generally to encourage smaller magnitude activations?
I tried one experiment on google/pegasus-xsum where I finetune with summarization lm loss and add some additional losses based on the magnitude of hidden states, but I haven’t weighted them (the model instantly forgets how to summarize) so I’m looking around.

Context:
It’s been a long time since this post, but maybe you remember if the problem with fp16 will appear when training the models from scratch (pretraining)?
I’ve seen some NaNs already while training with fp16 on, but after lowering the learning rate, beginning of training looks reasonable.