What can transformers learn without position encoding?

Question: So it obviously makes sense that attention mechanisms don’t have any inherent sense of position without encoding it explicitly, and for sequence prediction this seems critical. But, for example, word2vec via CBOW or skip gram is able to learn word embeddings without explicit position encoding. So my question is basically if we train a BERT model without the position encoding on the Masked LM task (something very similar to word2vec it seems to me), what is BERT capable of learning if anything? Would it be better than word2vec for creating word embeddings?

Context: My intuition would be that the transformers would still have a notion of context. It would still know this word appear in context with those other words, but would lose the notion of order loosely associated with position embeddings. Also, it would still allow word embeddings to change depending on the other words in context. So it would still be better than word2vec, which only has one embedding by word (learned as a combination of several contexts).


Question:
Hi @Mads your project looks very interesting, would you mind adding a description?

      huggingface.co

Mads/wav2vec2-xlsr-large-53-kor-financial-engineering · Hugging Face 4

Context:
Hi Snow, thank you for your interest!
I will update in the coming week as soon as possible!


Question:
Hey everyone, I would like to see any scientific evidence regarding model overfitting available for the
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization model.
If anyone can point me to some resources or provide an answer, I’d greatly appreciate it
Thanks and stay safe

Context: het @theprincedrip i don’t know the answer off the top of my head, but one place to start would be to check out the citations of the pegasus paper, e.g. via Google Scholar 1


Classification Heads in BERT and DistilBERT for Sequence Classification

Question:
Hi,
I have been using BertForSequenceClassification and DistilBertForSequenceClassification recently and I have noticed that they have different classification heads.
BertForSequenceClassification has a dropout layer and a linear layer, whereas DistilBertForSequenceClassification has two linear layers and a dropout layer.
Is there a particular reason for this?
Thanks in advance!

Context:
All in all, they have the same head: BertForSequenceClassification has a dropout layer and a linear layer but uses the pooler output, which went through a linear layer inside the BertModel.
DistilBertModel has no pooler output however, so the first linear layer is there to replicate that.


Collaborative Training Experiment of an Albert Model for Bengali

Question:
Huggingface is launching a collaborative training experiment of an Albert Model for Bengali language with our community. We are actively looking for participants who will help us to train the model.
So what do you need in order to participate-

A Google Colab account
That’s everything you need.
[Although if you want to use the power of your own GPUs, Huggingface will also provide a script for that.]

How you can contribute?

If you are a native Bengali speaker, that would be a great help, we are looking for participants who will check the performance of the tokenizer, sentence splitter, etc.

You might want to help us preprocessing the dataset. We are using the Wikidump and OSCAR Bengali dataset to train the model, if you have some suggestions on preprocessing these feel free to contribute in that part.

Now the main part, distributive training. You have been provided a google colab script in order to start the training and if your kernel crashes just restart the training script. (Non native speakers can participate)

Join our discord community link - https://discord.gg/GD9G4j8fJU 43
[A separate slack channel from Huggingface will be provided where you will get to know more about the distributive training framework and other related things.]
We are aiming to start this collaborative training experiment from - May 7th
Please do participate in this first Huggingface collaborative training experiment specifically the native bengali speakers.

Context: Also I forgot to mention the main thing. Thanks to Yandex for creating this collaborative distributive training strategy. Without them this huge community training event would not be possible.


Question:
This is a WIKI post - so if you feel you can contribute please answer a few questions, improve upon existing answers or add an alternative answer or add new questions:
This thread is to discuss Multi-GPU machine setup for ML.

Basic Recommendations
Q. What are basic recommendations on how to design a multi-GPU machine?
Would be great to factor in price vs performance (so we can know how much we save vs pre-built)?
A. See the links to the guides in the Resources sections below.

Critical decisions to make
Q. What are the smartest decisions to make it future proof (mine is already obsolete)?
A. Computers are black holes that suck everything in and give little out (other than some RGB colors). There is no such thing as future proofing in modern computers, other than mechanical parts like your PC tower.
Q. Can we do it at all or is it necessary to redesign it every 1-2 years?

Ideally you just upgrade parts as they need upgrading, rather than replacing the whole PC. I use a 10-year old tower still.

In-house vs. cloud
Q. Is it worth building a good local machine or should you just learn how to leverage the cloud?
A. Typically, for small set ups - up to several consumer GPUs, it’s almost always worth to have a local setup than cloud, unless you find some upstart cloud provider that for a while underprices their cost-per-hour.
Pros:

Of course, it depends on your usage patterns. If you are going to use it once in a blue moon, cloud it is. If you use it a lot then local will be cheaper. You can calculate your costs to purchase the machine vs. renting it.
Not needing to worry about forgetting to turn the instance off and having the $$ counter running might be another plus.
Heat is good. Heat is bad. In cold countries a home-based ML server is a great adjunct to keeping your working space warm. Not so much if you live in tropics.

Cons:

If you want a lot of large GPUs you might not be able to build it on consumer-level hardware, or the cost might be prohibitively expensive.
Electricity cost is another factor. Some cities have very expensive electricity. Especially if you go over the “normal” usage quota that some electric companies have.
Hardware gets outdated fast, so your needs may quickly become larger than what you have. You may or may not be able to recover some of the investment when trying to sell your old hardware.

Key components
Q .What are the main components to look for?
Q. Sample setups would be great too (and why they are great).
A.

Make sure your CPU has enough PCIe lanes to support all the cards you plan to use
Make sure your MB has enough PCIe slots and they are at the right distance to support modern GPUs that take up 2 slots.
Research your PSU - so that it has enough extra power to handle those power-hungry GPUs
Plan to have a lot of RAM, so ideally buy as large of a single RAM stick as possible. i.e. try not to fill out all RAM slots from the get going unless you buy some 256GB from the get going.
NVMe slot or a few are going to be super-important. Try to have your OS on a different drive (e.g. SSD) - you don’t want to share your data NVMe with your OS operations.
Does the box have enough space for cooling? Be it water cooling or lots of fans.
Definitely don’t buy those pre-packaged PCs by large retailers, you can’t mod those. Buy your own components and plan for expansion.

Puchase Timing
Q. Is it a good time to buy GPU or when to know when there are good deals (seem a bit high right now)?
A. Black Friday in North America gives you by far the best deals. But don’t just buy because it’s BF, do your research, since some companies raise their prices, instead of lowering those.

Resources

 Lecture 6 from Full Stack Deep Learning 6
 A 15000$ Machine Learning Rig: 2x3090 + 1xA6000 Build 5

Blogs focusing on ML Hardware:

 The Best 4-GPU Deep Learning Rig only costs $7000 not $11,000 14
Tim Dettmers’ great posts about choosing GPUs for deep learning 4 and Hardware Guide to Deep Learning 2. The guides do not focus on distributed setup, but there are suggestions on multi-GPU machines and how to select a GPU for your task and budget.

Context: I would recommend to check out Tim Dettmers’ great posts about choosing GPUs for deep learning 15 and Hardware Guide to Deep Learning 6. The guides do not focus on distributed setup, but there are suggestions on multiGPU machines and how to select a GPU for your task and budget.


Question: Is there an implementation of the Psuedo Log Likelihood for bidirectional language models (i.e. Salazar et al. Masked Language Model Scoring 14) in transformers? The github repo in the linked paper uses transformers 3.3 and I’ve been unable to get it to work for 4.5.

Context: what kind of problems are you running into? presumably it’s due to a change in the API, so sharing what steps you’re taking and the error messages will help with the debugging


`nan` training loss but eval loss does improve over time

Question:
I’ve been playing around with the XLSR-53 fine-tuning functionality but I keep getting nan training loss.
Audio files I’m using are:

Down-sampled to 16kHz
Set to one channel only
Vary in length between 4 to 10s

I’ve set the following hyper-params:

attention_dropout=0.1
hidden_dropout=0.1
feat_proj_dropout=0.0
mask_time_prob=0.05
layerdrop=0.1
learning rate:

on a warmup schedule to 3e-4 for 3 epochs
at 5e-4 for 3 epochs
back to 3e-4

Sadly, I’m fine-tuning the model on an unpublished corpus, so I am probably not at liberty to upload it here which might hinder reproducibility efforts greatly.
Here’s what the loss and WER progression looks like:

image497×815 75.2 KB

Anyone know what could be happening here? The model seems to be training just fine and some testing proves that the model performs well on the language I’m training it on. So what’s up with the training loss?
Pinging @patrickvonplaten and @valhalla as this might be relevant to them.

Context:
Hey @jjdv,
I’m sorry without a google colab it will be difficult to debug this for us. Given that your WER seems to decrease nicely - there might just be a problem at displaying the values…let’s see whether other people encounter the same problem


XLSR-53: To group tokens or not to group tokens

Question:
In @patrickvonplaten 's Fine Tuning XLSR-53 notebook, he mention how tokens shall not be grouped when computing metrics, in the case of that notebook, the WER metric. And that does make sense. However, later on in the notebook, he goes on to use the processor to decode the predictions and doesn’t pass the group_tokens=False argument to the method.
Shouldn’t the way we decode to compute metrics and to output predictions be the same? Which way would be the correct one? This is probably a minor issue for languages that don’t duplicate graphemes that often, but I’m curious as it could impact the perceived performance one way or another.
Could someone clarify this for me?

Context:
Hey @jjdv,
Could you check whether this issue answers your question: wav2vec2: `convert_tokens_to_string` contracts legitimately repeated characters · Issue #10619 · huggingface/transformers · GitHub 14?


Question:
Hi everyone,
I am dealing with a binary classification task (non-English language) of relatively long documents (~4k words on average). I have tested a Logistic Regression trained on simplistic BoW features, yielding reasonable performance.
I am now testing the multilingual BERT, with two linear layers on top of it and using the Cross-Entropy loss; however, its performance is quite low. The “annoying” part is that on a given test set, BERT always predicts the majority class. It is worth saying that the dataset (both train and test) is rather imbalanced (80/20).
I have tried the following without any luck:
a) Play around with the learning rate, class weighting, num of linear layers & associated configurations.
b) Select different parts of the document as input to BERT.
c) Generate balanced samples (incl. oversampling the minority class).
I have also tried generating a synthetic toy dataset of 1K examples from one document belonging to one class and another 1K examples from one document belonging belonging to the other class - the performance was perfect, as expected.
Is there something obvious that I am missing in terms of debugging my model? Is the problem the imbalanced nature of the dataset I am working with? Could a Focal loss (or anything else) help on this end?

Context:
Hi @aguarius, my naive guess is that the length of your documents is the source of the low performance since BERT has a maximum context size of 512 tokens which is only a handful of paragraphs.
One somewhat hacky approach to this could be to chunk your document into smaller passages, extract the hidden states per passage and then average them as features for your linear layers.
What language(s) are in your corpus? That might be another source of difficulty since mBERT is not great on all of its languages and perhaps you can work with a better model like XLM-RoBERTa (or even a monolingual one if that’s possible)


How does BERT actually answer questions?

Question:
have been trying to understand how the BERT model works. Specifically, I am trying to understand how it picks up up answers to questions on a given passage. I have tried following this 3blog post and whilst It has given me a nice intuition, I would like to better understand what is happening under the hood.
From my understanding, the question and paragraph are tokenised separately and then go through the transformer model. Then, the dot product between the ‘transformed’ tokens and a START/END token is taken, with the higher result giving you that start and Finnish of the answer.
What I would like to understand, what happens to the tokens in this “transformation” (i.e feedforward through the model) that makes it possible to take a dot product and therefore indicate if a word is a START/END.

Context: Hi @theudster, you can find a detailed tutorial on question-answering with transformers here: https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb 20


Hugging Face Reads - 01/2021 - Sparsity and Pruning

Question:
Hugging Face Reads

January 2021 - Sparsity and Pruning
By Victor Sanh 14, François Lagunas 5, and Yacine Jernite 4
Introduction to the Hugging Face Reads (HFR) series
New year, new Hugging Face reading group ! We are launching the Hugging Face Reads (HFR) series: each month, we will choose a topic to focus on, reading a set of four papers recently published on the subject. We will then write a short blog post summarizing their findings and the common trends between them, questions we had for follow-up work after reading them, and how recent advances in the area have affected our work at HF.
The first topic for January 2021 is Sparsity and Pruning, and we are planning to address Long-Range Attention in Transformers next month. Enjoy, and come join the conversation here!
Introduction
While large-scale pre-trained language models help solve an ever-growing set of natural language processing tasks, the progressive increase in their sizes raises concerns about their wide-scale applicability in practical settings, especially on devices with limited memory and computing power.
Sparse neural network models which only use a fraction of the large parameter sets of their dense equivalents offer a promising avenue to reduce these computational demands. Recent works have proposed various methods to achieve impressive levels of sparsity, whether by gradually choosing which parameters to retain during training or by “pruning” the parameter set after the fact. This post presents an overview of four papers proposing or analyzing such methods. We review the following works: the (Chen et al., NeurIPS 2020) 32 paper investigating the applicability of the Lottery Ticket Hypothesis to BERT-style models, the (Frankle et al., 2020) 19 analysis of currently available methods to find sparsity patterns at initialization before doing any training, the (Li et al., 2020) 19 work on the computational and performance trade-offs between training a large model to prune later vs. training smaller models right away, and the (Hooker et al., 2020) 18 study of the biases introduced by current methods used to compress models (including pruning).
Paper summaries
For each paper, we identify some of the claims and contributions, as well as some follow-up questions.
The Lottery Ticket Hypothesis for Pre-trained BERT Networks 32
By Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, Michael Carbin
The Lottery Ticket Hypothesis 6 (LTH) was initially developed and tested on computer vision systems. It states that given an initialization of a model, it is possible to find a subset of sufficient parameters during training: i.e., such that training only those parameters while setting the others to zero allows the model to reach the same performance as training the full model. Unfortunately, this subset can only be found after some amount of computation 3, and the method requires several iterations of re-training (either from scratch or from an earlier checkpoint, a method known as rewinding) and pruning for full effect. However, the approach can still end up improving training time and outputs a ready-to-use sparse model.
This paper sets out to validate the LTH in NLP (and in particular in BERT-style models). Specifically, it asks whether sparse subnetworks of a model pre-trained with Masked Language Modeling (MLM) are sufficient to solve down-stream tasks. The answer is broadly positive.
Findings

Using a pre-trained initialization, BERT contains sparse subnetworks at non-trivial sparsities that can be fine-tuned in isolation to full performance on a range of downstream tasks.
As opposed to previous work, these subnetworks are found at pre-trained initialization and not at random initialization (which was the case with the original LTH work). Rewinding does not significantly improve accuracy on downstream tasks.
There are universal subnetworks that transfer to all studied downstream tasks. By further fine-tuning on the same task that was used for pre-training (Masked Language Modeling), the method finds a 70% sparse sub-network that can yield good results on all downstream applications.

Follow-up questions

In practice, the computational cost of fine-tuning is already much less than that of pre-training. How would “fine-pruning” (pruning while fine-tuning with methods such as movement pruning) a model on a downstream task compare to using the LTH-sparse model obtained with MLM (or with the downstream task)?
The lack of impact of rewinding is in stark contrast with previous work on networks initialized from scratch and bears closer examination. For example, does this finding hold across fine-tuning learning rates? How much does the value of the selected parameters change over time?

Pruning Neural Networks at Initialization: Why are We Missing the Mark? 19
By Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin
This paper analyzes the performance of several methods to prune networks at initialization, so even before training starts, to save on training time as the network is smaller or sparse (SNIP, GraSP, SynFlow, Magnitude pruning). The methods are allowed to sample the dataset to perform the pruning: this sampling can be considered negligible compared to the computation required for training. They compare the methods to two “upper bounds” representing the performance we can hope to achieve when given access to information that is available after training: Magnitude Pruning and Lottery Ticket Rewinding.
Findings

All proposed methods are better than random pruning, but they are not sensitive to the individual selection weights, only to pruning proportions on each layer. Even worse, selecting the weights with the lowest instead of the highest value of the utility criteria improves performance on some methods (GraSP), which appears to invalidate some of the original works’ claims.
The methods are far from competitive with post-training approaches. Moreover, none of the methods is SOTA in all settings: some methods are better at some sparsity levels than others, but this depends on sparsity.
The methods yield better results if they are applied after a few training steps rather than right away at initialization, but they need a significant amount of training to approach the proposed “upper bounds”.

Follow-up questions

The problem of finding a “good” subnetwork right at initialization seems somewhat under-defined and possibly overly difficult: which task or set of tasks is used to measure success? Is it even possible to find an ideal sub-networks that works on any possible task a priori? Consequently, it is hard to tell whether the mixed results stem from flaws in the methods or from the task’s inherent difficulty. More insights here would be particularly enlightening.
The authors note that the studied methods prune “layers, not weights”, which may explain the surprising results they obtain by inverting the weight selection. In that case, would a dense model with adaptive layer sizes following the same patterns work as well?
An interesting follow-up direction could be something along the lines of “pruning as soon as possible”. Recent “Bertology” work 5 has shown that pre-trained models learn different levels of skill in sequence: we are particularly eager to see follow up work that explores the relationship between the emergence of these skills and the optimal pruning strategy.

Train Large then Compress, Rethinking Model Size for Efficient Training and Inference of Transformers 19
By Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, Joseph E. Gonzalez
This paper explores the landscape of the computational tradeoffs between transformer models’ sizes and the required numbers of hyper-parameter settings and training steps to achieve a good performance. It finds larger sizes can allow for fewer hyper-parameter settings and training steps and offers some practical advice about choosing a larger initial number of parameters that can later be pruned to, counter-intuitively, reduce the overall computational cost of training a mode when compared to just training a smaller model from scratch.
Findings

Large models are faster to train: they reach a given precision faster when measuring optimizing steps/wall clock time/ flops, even when they are stopped before convergence. Absolute size is more important than depth or width alone, but depth can be more important than width in some cases. The faster convergence usually makes up for the faster execution of smaller models.
Large models can be compressed to smaller networks. Training large networks might speed up training but would lead to problems at inference time, as their resource cost is much higher. This work finds that pruning them to networks that end up containing fewer parameters than the original smaller alternatives still yields higher performance. They can be quantized too with less quantization error.
Batch size has an influence on training speed. In practice, this means that gradient accumulation should be used for larger models.

Follow-up questions

The results are impressive, but it can still be difficult to get some intuition for why the larger models converge to a better state faster and are easier to prune. The authors mention previous work hinting that deeper networks “promote movement along directions already taken” as a possible explanation, but we are definitely looking forward to reading further analysis.
The connection to Lottery Ticket Hypothesis is mentioned only in passing. Further work exploring whether the sub-networks selected by the two approaches are similar in any fashion (such as by considering the Jaccard distance between the sets).

Characterizing Bias in Compressed Models 18
By Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, Emily Denton
This paper sheds light on the impact of pruning on neural models for vision and shows that reported top-line accuracies often hide the disproportionate negative impact on certain classes of inputs. The paper connects this phenomenon to bias and fairness considerations.
Findings

While the overall error is largely unchanged when a model is compressed (by pruning and quantization), there is a set of data that bears a disproportionately high portion of the error, with their accuracy falling by up to 50% while the overall performance only decreases by 1%, regardless of what the original accuracy was on the group.
These examples (or at least some of them) can be consistently identified by comparing the predictions from a population of compressed models with the predictions from a population of non-compressed models on the same inputs: the examples where the predictions distributions diverge are called Compressed Identified Examples (CIE).
CIE often correspond to low-frequency patterns in the inputs. Compression cannibalizes performance on low-frequency patterns in order to optimize the performance on higher-frequency patterns and preserve the overall accuracy.
Compression thus amplifies biases of models (amplifying certain errors on certain types of inputs). The authors suggest using CIE as an auditing tool for compressed models: surfacing a tractable subset of the data for further inspection by domain experts to assess this issue.

Follow-up questions

This paper studies are pruning and quantization techniques that are run after training. One question that remains open is whether the models are facing an issue of modeling capacity (i.e., less-biased predictions require more representation power) or whether it is tied to the training procedure. Analyzing methods that reduce model size in the course of training or approaches such as gradual pruning 4 would shed some light on this question.
Would up-weighting the CIE examples in training lead to models that are more robust to compression? Or would we expect to find different CIE groups?
The authors suggest using CIE as a diagnostic tool. What can be done with the diagnostic? Are there other calls to action from these insights? For instance, how could we change existing benchmarks on compression to include robustness metrics (i.e., adding another component to the tradeoff size vs. accuracy on CIE groups)?

Reading Group Discussion
The quantitative results obtained on many of the common benchmark tasks by pruning are impressive. At the same time, they also remind us how much we still have to learn about the training dynamics of neural networks. Common wisdom states that “overparameterization helps with optimization”, but we have little theory available to help us understand the phenomenon further, especially in the deep attention-based models that perform so well in NLP.
Each of the four papers above offers a different view of this question of modeling capacity vs. optimization vs. generalization.
The Lottery Ticket Hypothesis relies on the quality of the initial state of the parameters at least as much as on the evolution of the weight values during optimization. As such, the main purpose of increasing the number of parameters would be to exponentially increase the chances of hitting a good sub-network at initialization.
Other approaches focus more on how and whether the gradient flowing through the possibly redundant parameters help optimize the value of the ones we want to keep in the final pruned network: whether they try to evaluate that impact a priori as in the SynFlow algorithm or are content to simply keep them around for optimization based on their empirically proven efficiency and to prune them at the end of the training.
All of the works outlined above, however, assume that the neural networks are indeed over-parameterized and that they can be pruned without changing their qualitative behavior. The CIE work questions that assumption and finds that pruning does change the behavior of the model in non-trivial ways. This assessment also agrees with some experiments Victor Sanh 14 has run on the task for natural language inference, gradually pruning a model trained on multiNLI and testing it on the HANS dataset. As the sparsity increases, the generalization as measured by the accuracy on the HANS test set decreases and gradually drops to 0 while the performance on the multiNLI test set stays mostly constant. Another experiment along those lines would be to see how much factual knowledge pre-trained language models lose as they are pruned (for example by monitoring closed-book QA accuracy for a model like T5).
The question remains whether this loss of generalization and increased bias is a result of the model losing “expressive capacity” as its number of parameters decreases or whether the fault lies in the compression strategies that aren’t quite flexible enough, but the results certainly suggest that a large number of parameters serves as more than a crutch for optimization.
Another question that is somewhat orthogonal to the one above is that of when to optimally prune weights from the model. Pruning early saves computation, but does not benefit from any signal from the target task. Pruning after training can take advantage of additional information but does not save any computation at training time or allow the parameters to adapt to the new sparsity pattern. Gradually pruning during training seems to provide the best of both worlds, but introduces a new set of hyper-parameters which may make optimization more costly. One should also keep in mind that actual computational gains will depend on the capabilities of current hardware and their ability to take full advantage of shifting sparsity patterns.
We’re very much looking forward to the progress on all of these questions that 2021 is sure to bring!

@HuggingFace : Sparsity and Pruning
We first started investigating ways to make existing models more computationally efficient with DistilBERT 8, a method which was used to train one of our most popular models 2. The follow-up on sequence-to-sequence models yielded DistilBart 3, which also reaches similar performances to their larger counterparts at a lesser cost. Recently, we have also investigated approaches which focus on sparsity more specifically.
Movement Pruning
Most of the works referenced above use magnitude pruning, a widely used strategy for pruning which thresholds weight values and simply sets the smallest ones to zero. In our work on Movement Pruning 14 led by Victor Sanh 14, we argue that this approach is less effective in the context of transfer learning and highlight the importance of considering the changes of weights during fine-tuning as opposed to relying (mostly) on the pre-trained values. Code & hyper-parameters are available here.  11
Block Movement Pruning
The main drawback of unstructured pruning from a practical point of view is that current hardware can make it quite difficult to take full advantage of the sparsity pattern to accelerate the computation of the network. A compromise that can help alleviate this issue is the use of “semi-structured” sparsity patterns. By selecting blocks (typically 32x32) of weights instead of single weights, and running the same kind of optimization methods.
Accelerating block sparse linear algebra is easier, and the pytorch_block_sparse 14 library developed at Hugging Face is our attempt to show that. We are pretty confident more and more solutions for block-sparsity computation will emerge, and we will be working with major actors to enable it. We are already providing some sample networks 7 that take advantage of block sparsity, so you can judge by yourself!
Finally, we also work to combine block sparsity with other accelerated sparsity patterns such as NVidia Ampere, to further decrease the memory, the compute and the energy used by the neural networks that will be everywhere in the near future.

Context: Hi @VictorSanh I noticed that your implementation of movement pruning involves some masked versions of BERT like MaskedBertForSequenceClassification. Do you know whether these classes will become part of the main library at some point in the future?


Question:
Hi everyone,
I am looking for any ideas or advice that you guys may have obtained in similar situations.
I have been working on an NLP task to cluster medical documents for some time, and whilst I am eager to use transformers to get the best results, through all my efforts it seems that TF-IDF has worked best.
I am working with the SIDER side effect dataset, which provides annotated FDA medication labels, an example is here:
http://sideeffects.embl.de/media/pdf/fda/17106s032lbl/annotated.html#C0026961_0 2
I have tried TF-IDF and SciBert through sentence transformers, selecting the most relevant passages, but no amazing results yet. Does anyone have any ideas or previous experience?
Many Thanks,
Chris

Context:
Hi @FL33TW00D, I ran into a similar problem last year with TF-IDF and found the following approach gave better results:

Encode the documents, either with your favourite Transformer or Universal Sentence Encoder (the latter works really well!)
Run UMAP 3 on the embeddings to perform dimensionality reduction
Cluster with HDBSCAN 4

HTH!


Why are embedding / pooler layers excluded from pruning comparisons?

Question:
Hi @VictorSanh,
In your Saving PruneBERT notebook 5 I noticed that you only save the encoder and head when comparing the effects of pruning / quantisation. For example, here you save the original dense model as follows:

# Saving the original (encoder + classifier) in the standard torch.save format

dense_st = {name: param for name, param in model.state_dict().items()
                            if "embedding" not in name and "pooler" not in name}
torch.save(dense_st, 'dbg/dense_squad.pt',)
dense_mb_size = os.path.getsize("dbg/dense_squad.pt")

My question is: why are the embedding and pooled layers excluded from the size comparison between the BERT-base model and its pruned / quantised counterpart?
Naively, I would have thought that if I care about the amount of storage my model requires, then I would include all layers in the size calculation.
Thanks!

Context:
Hey!
The QA model actually only needs the qa-head, the pooler is just decorative (it’s not even trained). Start and end of spans are predicted directly from the sequence of hidden state. This explains why I am not saving the pooler.
As for the embedding, I’m just fine-pruning the encoder, and the embedding modules stay fixed at their pre-trained values. So I am mostly interested in comparing the compression ratio of the encoder (since the rest is fixed).
Hope that makes sense.


Question:
Hi- Thank you again for the awesome library & work.
I have been trying to repurpose the RAG code to train on the KILT dataset. As I understand, during the training phase, document encoder (and the index) is fixed, only the query encoder and the generator are fine-tuned.
As I train multiple epochs, something curios happens where the question encoder ‘collapses’ into emitting identical predictions regardless of the input.  Specifically, out1 and out2 are identical, even though input embeddings are different.
emb2 = torch.randn([1, 512, 768])
emb3 = torch.zeros([1, 512, 768])

# encoder
out1 = model.rag.question_encoder.question_encoder.bert_model.encoder(emb2)
out2 = model.rag.question_encoder.question_encoder.bert_model.encoder(emb3)

The way this behavior manifests itself is that the question encoder starts pulling the same wiki entries regardless of the question.
In fact, the last hidden states are identical for each token in the sequence.
Screenshot from 2021-02-08 21-20-17823×215 26.1 KB
I am curious if this type of behavior rings any bells? One hunch I have is whether mixed-precision training might be the cause. Any direction / feedback will be greatly appreciated, before I take the plunge and dig any further.
Thank you!
Deniz

Context:
Hi ! There’s some discussion about that at Retrieval Collapse when fine-tuning RAG · Issue #9405 · huggingface/transformers · GitHub 12
Apparently it can happen in some setups


Question:
Hi,
It is my understanding that all the pretrained models have a fixed maximum number of tokens (512 for bert-base-uncased). Suppose I have texts, that when tokenized exceed that number (like fictional text running through many paragraphs). I feel that there could be a better way than just using the first 512 tokens of the text. I could increase that limit, but my understanding is that for me to do that I have to train model from scratch and not be able to use the pretrained model. I would like to use the pretrained model.
In order to achieve this I have an idea and need some feedback on that:

Split the text into a list of sentences using a sentence Sentence Boundary Disambiguation tool.
Tokenize each sentence using the model’s corresponding tokenizer.
Create our new text, by keep the first and last n sentences from the list and then taking a random subset of the rest of the sentences, such that all the tokens add up to 512.

This will not restrict the input to only the first 512 tokens and will include random sentences from the middle of the text. Any thoughts on this approach?

Context: Sure, that is an option. You can also first run the text through a summarizer model and use the output as the input for your classifying model. There is no one “right” approach. You can try different things and see what works best for you.


Question:
For this science Tuesday, I read Marge, and wrote up a brief summary, as well as some interesting questions to discuss @joeddav @srush @VictorSanh @thomwolf @clem @julien-c @teven @patrickvonplaten @yjernite (only allowed 10 tags)
Pre-training via Paraphrasing (MARGE)
Paper 31: published June 26 2020
Authors are from Facebook AI Research:
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh,  Armen Aghajanyan, Sida Wang, Luke Zettlemoyer.
Summary
Huge models trained with masked-lm pretraining objective, or similar, memorize lots of facts in their parameters and don’t use an external storage to look up facts they are missing. Human brains have separate systems (it seems 11) for memorizing facts and generating language, and often google things. In this spirit, goal of many transformer+retriever models is to decompose memorization of facts and language understanding. MARGE stands for a Multi-lingual Autoencoder that Retrieves and GEnerates.
The pretraining setup:
reconstruct original document by retrieving related documents (from wiki) and trying to regenerate the original maximize likelihood of original doc conditional on retrieved docs, relevance scores. This implicitly forces the retriever to learn how to generate good relevance scores.
There are some tricks related to not scoring all of wikipedia for every example while keeping relevant articles in each batch.
Every 10k training steps, they remake their batches by computing the cosine similarity of every pair of docs, and then greedily adding source and target docs to batches such that the pairwise sum of cosine similarities increases the most. This obviously seems hacky, but allows them to get away without approximate NN or some other expensive way to find related docs. This, and the fact that a randomly initialized encoder will give docs with lexical overlap higher than random cosine similarity, allows the model to train from random.
The retrieval model, ideally, can focus on getting the transformer all the facts that it needs while the transformer learns to paraphrase, which requires generating fluent language.
For finetuning/inference, you don’t need to use the retrieval part.
Marge performs…:

comparably to XLM-Roberta, with 20% of the pretraining compute.
comparably to mbart on de-en, en-zh translation
SOTA on ml-sum, a cross lingual summarization task

Key contributions:
(1) Most of the related work is not multilingual
(2) most of the related work does not zero-shot well?
(3) this pretraining objective unifies learning to retrieve and learning to generate. Previous work requires two pretraining stages.
Related Work
Realm: “At a high level, the method goes like this: find the most similar text passages in BERT space, add those passages to the input as additional context, and then make a prediction.” -Joe a few weeks ago 8

different because the retriever has to be pretrained separately. Realm also seems to use mostly open domain QA benchmarks.

RAG (Retrieval-Augmented Generation)

Different because mostly focused on knowledge intensive benchmarks. MARGE can also do well on translation.
Starts with bart-large + DPR, whereas MARGE pretrains end-to-end.

Questions somebody could answer:

Does MARGE outperform Bart on english only benchmarks like GLUE/ xsum summarization? Why did they only show multilingual benchmarks?
When will there be code?
How long does a forward pass take?
What are the consequences of not using retrieval during inference. Does the model not “know” anything?

Higher Level:

Is Translation “knowledge intensive”?
How could we measure hallucinations?
Authors suggest that we should use a pre-training that is as close as possible to the dowstream task. Pegasus paper also suggests this. Where else could this idea be applied?

Also these two talks are good:
https://slideslive.com/38929793/beyond-bert 38  (Mike Lewis at ACL)
https://www.youtube.com/watch?v=KTQPWoQ7Ol8 33  (Luke Zettlemoyer at AKCD)

Context:
From Mike Lewis, the 1st author:

We didn’t try very hard, but from what I saw MARGE lags a little behind BART on monolingual English tasks. It’s not too surprising, because I think having to be a good multilingual model just dilutes the capacity a bit. Similarly, XLM-R isn’t quite at RoBERTa level.

code coming soon

they also retrieve from CC-News, not just wikipedia.

“We’re going to look at retrieval during inference, but haven’t run that yet. Qualitatively, I think it’s a bit less prone to hallucination than BART because it (somewhat) knows that it doesn’t know anything. That means we get surprisingly literal zero-shot translations, because it tends not to make too much stuff up.”


Model or Dataset available for classifying a grammatical sentence?

Question:
I want to be able to classify if an input text is a complete sentence or not.

The closest accurate definition of ‘being complete’ is if the sentence is a grammatical sentence.
Also ‘being complete’ sentence, can depend on the context of the sentence but I want to focus on a sentence-like text as input for now.

Example of a complete sentence:

“You can write using one of the following styles”
“You can write”
“He writes code”

Example of an incomplete sentence:

“You can write using”
“You can write using one”
“He writes code for”

I found this package for grammar checking which I am going to try:

      PyPI

language-tool-python 15

Checks grammar using LanguageTool.

I am wondering if there is an ML/DL solution for this problem. Is there a dataset or available model for this that you know?

Context:
Hi @emadg ,
I don’t think language tools is best way to go here, because language tool will just check grammer and grammer does not predict wheather this sentence is complete or not. Here are the rules that are implemented in Language tool, you can check if there are any rules which will help you to classify a sentence as complete or not.
https://community.languagetool.org/rule/list?sort=category&order=asc 19
For ML Approach, I think you can try using a Language model, You can trying looking for things which end a sentence and find probaibily of those words like (Punctuations, conjuction words) in the end. Lesser probability means very less chances that sentence will end there.
PS: I will add If I found a concreate method to solve this.


Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Question:
Interesting new paper from Google improving upon T5.

      arXiv.org

Switch Transformers: Scaling to Trillion Parameter Models with Simple and... 23

In deep learning, models typically reuse the same parameters for all inputs.
Mixture of Experts (MoE) defies this and instead selects different parameters
for each incoming example. The result is a sparsely-activated model -- with
outrageous numbers...

Context: Just to add to the previous post… Google Brain recently unveiled a language model of 1.6 trillion (1.6E+12) parameters with performance equal to or better than the SOTA on several NLP tasks. It surpasses the 175 billion (1.75E+11) parameters of GPT-3. The mastodon was made possible by the development of a new attention-based architecture (switch transform) that divides training data and parameters between a multitude of sub-models or mix of experts connected by trainable gating. Despite its gigantic size, this text-to-text model would have been 7 times faster to train on the C4 (Colossal Clean Crawled Corpus, 750 GB) using the same amount of computation. The original article: https://bit.ly/2LQzsmJ 19, the source code: http://bit.ly/390j0ZY 46


Classification problem difficulty when going from 3 classes to 5 classes?

Question:
This question is conceptual in nature.
Suppose I’m working on a text classification problem where I have 3 labels. To make the problem more concrete, let’s say I’m working on sentiment analysis with ground-truth labels positive, neutral, and negative. I am measuring accuracy and macro-F1.
Now I’d like to make another data set with 5 ground-truth labels: very positive, positive, neutral, negative, and very negative. Intuitively, I would think that the 5-label classification problem is more difficult than the 3-label problem, but the only “proof” I can think of is that a random guess is correct only 1/5 of the time with 5 labels but a random guess is correct 1/3 of the time with 3 labels.
Is there a more formal machine learning argument for why a 5-label problem is more difficult than 3-label? How about an N-label problem to an M-label problem where M > N?
I’m willing to brush up on Vapnik–Chervonenkis theory if that’s needed (hopefully not).

Context: Any help, intuition, hints, pointers, or references would be appreciated.


Question:
Hello, I am trying to understand how T5 sentencepiece impacts custom data set. I know T5 does not use lossless training(mT5 does) but unsure of what impact it may have on any custom tokens in my dataset. Can someone please chime in if you have some insight ?
Thanks

Context: What do you mean by “lossless” training?


Question:
Hi, I would like to try the approach suggested in “Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks” (link 11) for BART. I have my own dataset but there are 2 things that are still unclear to me.

I believe I should start with BartForConditionalGeneration , as that is the LM model. is that right?

Can anyone provide more details on the noising algorithm that was used to train the model? The paper is pretty vague about it, as these are the only details I found

A number of text spans are sampled, with span lengths drawn from a Poisson distribution(λ = 3)

We mask 30% of tokens in each document, and permute all sentences.

Context:
Hi @Erpa

Yes, BartForConditionalGeneration is the LM model.

Currently seq2seq pre-training examples are not available in transformers. FairSeq has the implementation of Bart denoising dataset, so that might help, You can find it here 33


Question:
Hello everyone,
Has anyone experimented with Lamb optimizers in HF? I tried using https://github.com/cybertronai/pytorch-lamb 39 but I was only marginally able to increase batch size and the training loss curve was rather flat. If you’ve used lamb would you please share some tips. How did you initliaze it? I am not sure what to use in optimizer_grouped_parameters list of dictionaries that wrap model parameters. Also, I’ve seen some other people use a different lr scheduler with Lamb.
Thanks in advance.

Context:
Hi vblagoje,
I am new to transformer. I have been playing the hugging face model for several month and I think I am thinking to made a some small changes on the Bert Model and pretrain it from scratch. I saw you discussing on another post several days ago about the pretraining process. I was wondering if you know the pretraining repository made by Nvidia?

      GitHub

NVIDIA/DeepLearningExamples 20

Deep Learning Examples. Contribute to NVIDIA/DeepLearningExamples development by creating an account on GitHub.

I think they implemented the lamb optimizer, NSP objective and wrote code to better utilized multiple gpu during distributed training. I still haven’t use it yet because I have some trouble with installing docker  on the remote machine I am working on. I was just wondering if you already seen this repository or tried it, or if you have any advice on pretraining bert from scratch?


About the encoder and generator used in the RAG model

Question:
Hi, I have questions about the Rag model.
In this paper, the query encoder is DPR and the generator is Bart.
My questions are:

Is the generator a full Bart or just the decoder part of the Bart.
If I implement a Rag with the encoder part of Bart as the query encoder, and decoder part of the Bart as generator. Does that make sense w.r.t the Rag concept? I think this is more intuitive to me. why they use a ‘heterogeneous’ setting?

Thanks.

Context:
Hi,

generator is Bart encoder-decoder. If you have a rag model, you can access it by model.generator

RAG’s question-encoder is not the same as RAG’s generator’s encoder … This really may be confusing, so let me try to explain

question encoder is for encoding “question” to retrieve “documents” (or so-called “contexts”) from  retriever.
Then, retriever will concatenate “contexts” with “question” ; this concatenated texts are the new input.
This new input will be encoded by Bart’s encoder to generate answer via Bart’s decoder

Hope this helps!


Using transformers (BERT, RoBERTa) without embedding layer

Question:
I’m looking to train a RoBERTa model on protein sequences, which is in many ways similar to normal nlp training, but in others quite different.
In the language of proteins, I have 20 characters instead of the normal 26 characters used in english (it is 26 right? :D), so that is rather similar. The big difference is that you don’t really combine the characters in proteins to actual words, but rather just keep each character as a distinct token or class.
Hence essentially my input to the transformer model could just be a list of numbers ranging from 0-19. However that would mean that my input would only have 1 feature if I did that, and I’m not sure a transformer could work with that?
I’m thinking of just doing a onehot encoding of these characters, which would give me 20 input features. However this is of course still very low in comparison to how normal transformers are trained, where d_model is somewhere in the range of 128-512 if I understand correctly.
Does anyone have any experience with anything like this? any good advice for how it is most likely to work?

Context:
Hey,
I’d recommend taking a look at this repo: https://github.com/agemagician/CodeTrans 87 by @agemagician . This repo uses transformer models for protein sequences if I understand it correctly.
Also, taking a look at those models:

      huggingface.co

Rostlab (Rostlab) 25

might help. Not sure if there is a notebook on doing protein sequence LM, maybe @agemagician has a good pointer by chance


What are some recommended pretrained models for extracting semantic feature on single sentence?

Question:
Hi, I am more a CV guy and recently get interested in doing a nlp project.
In this project, one part might involve extracting sentence-level semantic representation from a pretrained model.
In computer vision, one standard way to extract feature of an image or a video snippet could be
using Resnet pretrained on Imagenet or I3D pretrained on Kinetics datasets, respectively.
I want to do the similar thing but in nlp domain. I wonder if there are some recommended models pretrained on specific dataset for me to try?
As far as my limited understanding, models trained on datasets which aim to to tell if two sentences are semantically equal could be a direction (e.g. QQP, STS-B ). But it needs a pair of sentences, my case is just feeding one sentence (or one block of sentences), not in a pair format. Any suggestion? Thanks!

Context: Hi! IMO, Bert could be comparable to ResNet as the baseline. (you can use last_hidden_state variable of BertModel just like the global-pooled features of ResNet) Then, newer models like Roberta and many more could be comparable to EfficientNet etc.


BORT: Optimal Subarchitecture Extraction for BERT

Question:
Hi guys,
Wondering if anyone has read the new paper from the Alexa team regarding BERT size reduction.

      arXiv.org

Optimal Subarchitecture Extraction For BERT 8

We extract an optimal subset of architectural parameters for the BERT
architecture from Devlin et al. (2018) by applying recent breakthroughs in
algorithms for neural architecture search. This optimal subset, which we refer
to as "Bort", is...

      GitHub

alexa/bort 8

Repository for the paper "Optimal Subarchitecture Extraction for BERT" - alexa/bort

If anyone has any thoughts on it or would like to discuss please comment here.
Thanks

Context:
Super interesting, thanks for sharing!! Perhaps @VictorSanh can give us the best comments
Wondering if the same technique can be efficiently used for the giant models like T5-11B and GPT-3


EMNLP Picks from the Hugging Face Science Team

Question:
The Hugging Face  team had a great time attending EMNLP the other week. Virtual conferences are tricky, but I personally have come to enjoy some aspects of it like the pre-recorded presentations and gather.town mingling. And not having to travel is a plus, too
Last week a few of us on the science team tried to each select 4-5 presentations we’d recommend others on the team to check out. I’ve compiled our suggestions and included them here for those of you that are interested in our picks & very brief comments. Included are suggestions from myself, @VictorSanh, @yjernite, and @canwenxu (including a couple repeats).
There was an incredible amount of high-caliber work and we couldn’t share all but a few that we thought our team might be interested in, so free to respond with any suggestions (or comments) of your own!
Victor’s picks (@VictorSanh)
BLEU might be Guilty but References are not Innocent
Paper: https://arxiv.org/abs/2004.06063 22
Presentation: https://slideslive.com/38938647 17
Discuss a new reference generation method for calculating more reliable automatic scores (including BLEU) that correlate better with human judgement. + a dataset of references (included in sacrebleu i believe)
Learning from Task Descriptions
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.105.pdf 37
Presentation: https://slideslive.com/38939344 15
Introduce a new dataset for structured task-oriented evaluation on unseen tasks (0-shot settings) conditioned on a description of the task in natural language. (nice discussion, less convinced by the dataset itself)
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.16/ 21
Presentation: https://slideslive.com/38939219 9
Model can learn to represent linguistic features with little pretraining data, but require orders of magniutde more data to learn to prefer linguistic generalization over surface ones (but it is slow…)
Reformulating Unsupervised Style Transfer as Paraphrase Generation
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.55/ 33
Presentation: https://slideslive.com/38938942 15
Propose simple method based on fine-tuning pretrained language models on automatially generated paraphrase data + discusses weaknesses in automatic metrics of style transfer + release of 15M dataset of style transferthe 5th one: I found the talk of Emmanuel Dupoux at Conll very informative
Yacine’s picks (@yjernite)
ETC: Encoding Long and Structured Inputs in Transformers
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.19 16
Presentation: https://slideslive.com/38938951/etc-encoding-long-and-structured-inputs-in-transformers 4
Has local attention and a one global attention token per sentence which is trained with a contrastive loss similar to ICT.
A* Beam Search
Presentation: https://slideslive.com/38939414/bestfirst-beam-search 21
A* algorithm is not quite as easy to batch as regular beam search, but leads to better and more diverse n-best.
F2-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.737/ 13
Presentation: https://slideslive.com/38938686 6
Pretty simple idea: groups tokens into bins of equal probability mass for a hierarchical softmax so the model can focus on choosing between candidates with the same prior. Leads to a nice improvement on human evaluation and generation diversity metrics.
Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems
Comments: https://www.aclweb.org/anthology/2020.emnlp-main.203 9
Presentation: https://slideslive.com/38938871 5
Pre-trains on BPE and fine-tunes on full character decomposition to get the model to train faster.
Towards Debiasing NLU Models from Unknown Biases
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.613 21
Presentation: https://slideslive.com/38938901 4
Related to @VictorSanh’s recent paper: the “biases” tend to show up in easy-to-learn examples, so the model down-weight examples that are classified correctly early in training.
Canwen’s picks (@canwenxu)
Experience Grounds Language
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.703.pdf 51
Presentation: https://slideslive.com/38938907 25
This may be the paper that defines the future direction of NLP. What should a model learn and what ability should a model have? You can find a good guess from this paper.
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.634.pdf 21
Presentation: https://slideslive.com/38938976 2
Yes we know that fine-tuning a pretrained language model can bring the problem of forgetting. Mixout 10 is a valid solution but this EMNLP paper proposes an easy-to-use optimizer to resolve the problem.
Do sequence-to-sequence VAEs learn global features of sentences?
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.350.pdf 12
Presentation: https://slideslive.com/38939119 5
It’s a little surprising to see this title cuz we all thought of course VAEs do. However, through well-designed experiments, the authors reveal the other side of this claim.
Pre-Training Transformers as Energy-Based Cloze Models
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.20.pdf 14
Presentation: https://slideslive.com/38939095 4
It’s a really cool idea and it makes sense mathematically. Though the results are modest, there’re definitely more to explore.
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.633.pdf 16
Presentation: https://slideslive.com/38938938 5
Self-promoting. It’s a really neat idea that you can compress a model by simply replacing their components. No additional loss function needed.
My picks
Learning from Task Descriptions
Paper : https://www.aclweb.org/anthology/2020.emnlp-main.105.pdf 37
Presentation : https://slideslive.com/38939344 15
@VictorSanh mentioned this one but I want to include it as well. They create a new dataset trying to generalize from one set of tasks to another using only task descriptions w/o training data. It’s an ambitious idea to try to formalize and evaluate but I appreciated the work. I’m actually taking a break from adding their dataset “zest” to Datasets to compile this post, so it should be up very soon.
Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.660 9
Presentation: https://slideslive.com/38939094 6
Another approach to “universal” NLP w/ cross-task generalization. The idea here is to pose various tasks as one task (natural language inference) enabling transferability between tasks. Incidentally, the first author is the same who introduced the NLI-based zero-shot 4 classification approach which is roughly the same as the one we now use in our zero-shot pipeline & API 3.
Text Classification Using Label Names Only: A Language Model Self-Training Approach
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.724 29
Presentation: https://slideslive.com/38938946 20
Similar to the “zero-shot” setup of Schick et al. 3's PET and Yin et al. 3's entailment-based approach (though they refer to it as “weak supervision” here). A nice difference from previous work is that they create groups of synonyms to a class label which can be used as a class representation instead of the class name alone. Another demonstration of self-training with unlabeled data only working well for classification.
Experience Grounds Language
Paper: https://www.aclweb.org/anthology/2020.emnlp-main.703.pdf 51
Presentation: https://slideslive.com/38938907 25
Really nice kinda philosophical paper about computational understanding of language. They lay out different “world scopes” to help think about different levels of understanding/experience. Reminiscent in some ways of Bender & Koller’s ACL paper this year, “Climbing towards NLU” 8 and their superintelligent octopus.

Context: Especially like the linguistic shout-outs in there like Warstad et al.  It’s always nice to see authors go back and see what (generativist) linguist theory has been saying for perhaps over sixty years and find ways to link that with how LMs “learn” grammar. I’ll be having some time off soon, can’t wait to catch up with all these latest developments! Thanks for the distillation, (pardon the pun)!


Adding learnable coefficients for multi-objective losses?

Question:
I am running a multi-objective problem where I compute three losses and then sum them up. For each loss, I want to have a learnable coefficient (alpha, beta, and gamma, respectively) that will be optimized.
optimizer = AdamW(model.parameters(), lr=2e-5, eps=1e-8)

for batch in dl:

    optimizer.zero_grad()

    result = model(batch)

    loss1 = loss_fn_1(result)
    loss2 = loss_fn_2(result)
    loss3 = loss_fn_3(result)

    # How to optimize alpha, beta, and gamma?
    loss = alpha*loss1 + beta*loss2 + gamma*loss3

    loss.backward()
    optimizer.step()

Specific questions:

Should I even have coefficients alpha, beta, and gamma? The optimizer will minimize, so they’ll all go to 0.0, right?

If having those coefficients is a good idea, how can I prevent them from going to 0.0? Someone told me to use regularization, but what does that mean in this case?

How do I declare alpha, beta, and gamma to be learnable by AdamW?

Context:
Yes

Theoretically, we have to make a constraint like alpha+beta+gamma = 1. To change this to unconstrained optimization, we have to use Lagrange multiplier to the constraint equation, and that will be the regularization formula your friend talked about e.g. you put

lambda1*alpha, lambda2*beta and lambda3*gamma
into loss function. I believe it complicates the problem even more since finding optimum values of lambdas are difficult even theoretically.
2.5 Sorry not answer you Q3, but I think the practical way is to treat alpha, beta and gamma as hyperparameters and simply optimize them via grid search.
In this case, simply split some of your training set to validation set, and define the metric on it. The “validation metric” has to be specified to be suitable to your problem (e.g. error, f1, spearman or any others) — you can get some ideas on metrics by finding some Kaggle competitions that is similar to your problem and see their metrics.
Select hyperparaeters that optimize your validation metric.


Is there an easy way to apply layer-wise decaying learning rate in huggingface trainer for RobertaMaskedForLM?

Question:
I am pre-training RobertaMaskedForLM on my own custom dataset. I wanted to implement the layer-wise learning rate decay given in https://github.com/aws-health-ai/multi_domain_lm#learning-rate-control 12 corresponding to the paper - An Empirical Investigation Towards Efficient Multi-Domain Language
Model Pre-training 12. Is there an easy way to incorporate this decay of learning rate with layer depth towards input using transformers.Trainer?

Context: I have the same question


Question:
BERT has been trained on MLM and NSP objective. I wanted to train BERT with/without NSP objective (with NSP in case suggested approach is different). I haven’t performed pre-training in full sense before. Can you please share how to obtain the data (crawl and tokenization details which were used) on which BERT was trained on ?. Since it takes a lot of time, I am looking for well tested code that can yield the BERT with/without NSP in one go. Any suggestions will be helpful.
I know about some projects like these 59, but they won’t integrate well with transformers well I guess which is a must have condition in my case.

Context:
BERT was trained on book corpus and english wikipedia both of which are available in dataset library

      huggingface.co

Hugging Face – On a mission to solve NLP, one commit at a time. 123

      huggingface.co

Hugging Face – On a mission to solve NLP, one commit at a time. 63

Transformers has recently included dataset for for next sent prediction which you could use

      github.com

    huggingface/transformers/blob/master/src/transformers/data/datasets/language_modeling.py#L258 120

            i += 1  # go to next line
        return examples

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i) -> Dict[str, torch.tensor]:
        return self.examples[i]

class TextDatasetForNextSentencePrediction(Dataset):
    """
    This will be superseded by a framework-agnostic approach
    soon.
    """

    def __init__(
        self,
        tokenizer: PreTrainedTokenizer,
        file_path: str,
        block_size: int,

and there’s also NSP head  for BERT

      github.com

    huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L560 29

class BertOnlyMLMHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.predictions = BertLMPredictionHead(config)

    def forward(self, sequence_output):
        prediction_scores = self.predictions(sequence_output)
        return prediction_scores

class BertOnlyNSPHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.seq_relationship = nn.Linear(config.hidden_size, 2)

    def forward(self, pooled_output):
        seq_relationship_score = self.seq_relationship(pooled_output)
        return seq_relationship_score

class BertPreTrainingHeads(nn.Module):

EDIT:
BertForPreTraining class can be used for both MLM and NSP
with the current example/languae-modeling I guess it’s only possible to either use MLM or NSP, you might need to write your own script to combine these.


What are some popular datasets for domain adaptation in NLP

Question:
Hello,
Having some experience in domain adaptation in CV but no NLP.
Can someone recommend some popular datasets in NLP for DA?
and even better for me if there is any in the hugginface datasets.
Thanks!

Context: cc @yjernite maybe here (and Angie which should be also on the forum by the way!)


Question:
Hi folks,
How would you best recommend that I pass gradients through generate? below is a rough code snippet explaining the objective.
I am thinking that I could take the hypo_ids directly from the model output (instead of from generate), but this is no longer natural because teacher-forcing is used to generate these.
Thoughts?
Context from Pytorch Lightning Implementation:

# self.model = BartForConditionalGeneration("facebook/bart-base")

def forward(self, batch, batch_id):
    return self.model(input_ids = batch["x"], decoder_inputs=["decoder_inputs"], decoder_labels = ["decoder_labels"] )

def training_step(self, batch, batch_id)
   """Want two losses, language modelling loss and semantic similarity loss"""

    # language modelling loss
    outputs = self(batch)[0]
    language_modelling_loss = outputs[0]

    # semantic similarity loss
    target_ids = batch["target_ids"]
    hypo_ids = self.model.generate(batch["x"]) # no gradients passed of course
    semsim_loss = 1 - nn.CosineSimilarity(dim=0)(target_ids, hypo_ids)

   return {"loss": language_modelling_loss + semsim_loss}

Context:
EDIT: The only method seems to be to use RL to simulate the sampling that occurs.
see https://papers.nips.cc/paper/8682-training-language-gans-from-scratch.pdf 6


Adding features to a pretrained language model

Question:
I’ve often thought about use cases where you think of word or sentence features that you know must be helpful to the system. Features that you would typically use in an SVM or a shallow network. I would want to know if those features still have the ability to add to the performance of a pretrained language model. So rather than just fine-tuning the language model, what are good ways to integrate custom features into LM without pretraining from-scratch?
My guess is that you can just take the output from an LM and add a custom head on top that also takes in these other features. So basically the output of the LM serves as another set of features. This does not seem ideal though, since the final connections might be too shallow, I imagine that a better approach is possible that still involves finetuning the LM along side training the network that the custom features are part of. Any thoughts or best “tried and true” methods out there?

Context:
Hi Bram,
One of my students studied exactly this phenomenon in a recent submission to SemEval: “UoB at SemEval-2020 Task 12: Boosting BERT with Corpus Level Information.” (https://arxiv.org/abs/2008.08547 69)
Excerpts from the paper:
We hypothesise that deep learning models, especially those that use pre-trained embeddings and so are trained on a small number of epochs, can benefit from corpus level count information. We test this on Sub-Task A using an ensemble of BERT and TF-IDF which outperforms both the individual models.
For sub-task B, we hypothesise that these sentence representations can benefit from having POS information to help identify the presence of a target. To test this hypothesis, we integrate the count of part-of-speech (POS) tags with BERT. While this combination did outperform BERT, we found that a simpler modification to BERT (i.e. cost weighting, Section 3.5) outperforms this combination.
And in terms of how the model was built:
This ensemble model is created by concatenating the sentence representation of BERT to the features generated by the TF-IDF model before then using this combined vector for classification. In practice, this translates into calculating the TF-IDF vector for each sentence and concatenating it to the corresponding BERT output. This vector is then fed to a fully connected classification layer. Both BERT and the TF-IDF weights are updated during training.


Question:
Has anyone finetuned bart-base on xsum or cnn summarization task and willing to report the rouge score they got?
I just got 15.5 for xum which feels low, since bart-large can get to 22 ish.
@colanim @valhalla @VictorSanh ?

Context: @sshleifer, could it be due to the adjust_logits  issue ? Just a guess but as I posted there, after modifying the  adjust_logits_during_generation  BLUE-4 score for my model went from 13.09 to 19.14 for bart-base


Question:
Hey everyone,
I am exploring https://github.com/huggingface/pytorch_block_sparse 2 project. One of the issues that popped up almost immediatelly is loading a saved “sparsified” model. So, let’s say you sparsified Roberta using an example provided . Now that the model has been sparsified (it’s linear layers replaced with BlockSparseLinear nn modules) how can I load previously saved model using HF ecosystem? All I can think of is that I again need to create a Roberta model with uninitialized weights, sparsify it, and the load weights with model.load_state_dict(torch.load(PATH))?
Am I overlooking something obvious?

Context: No mechanism in place for loading as of now, which is ok. I sparsed the model again and loaded the weights manually via model.load_state_dict(torch.load(PATH)).


Resume Training / Finetune a language model and further finetune a classifier

Question:
Hi,
I would like to finetune a powerful classifier based on a pre-trained language model. As we know, the typical approach is to fine-tune a classifier using a pre-trained model. What I am wondering is that, if I fine-tune a pre-trained model based on a fine-tune language model settings using DS1(typical text dataset) (OR resume training from the last checkpoint) and then further fine-tune this newly fine-tuned model using another DS2(typical text dataset) for a classifier purpose, would this be a redundant effort as compared to a pipeline which is to just finetune a pre-trained model using DS2?  I would like to receive your thoughts.
Thank you.

Context: Hi, there are papers indeed indicate that “multi-steps” finetuning is helpful. See this paper 15 for one example .


Question: I was wondering if it were possible to somehow train GPT2 to generate question-answer pairs in a particular domain?

Context:
I’ve tried this with seq2seq models. I have worked on qa pair generation (separately)  using T5 with descent results. You can find it here 78.
One way we can do this with GPT-2 is prepare our input like this
Our context is  42 is the answer to life, the universe and everything , answer is  42  and target question is  What is the answer to life, universe and everything ?
Then
input text:  context: 42  is the answer to life, the universe and everything. question: What is the answer to life, universe and everything ? answer: 42
and prepare the attention mask such that, there will be no attention from  question: ...  part, so model won’t look into future tokens and calculate loss only on the  question: ...  part. And it inference time we will feed only the context part and ask the model to generate the question.
This just one one way I can think of the of my mind. Feel free to correct me if this is wrong.


Transformer for Abstractive Summarization for Chats Based on Performance

Question:
Hi, I’ve some general questions related to Transfer Learning on pretrained models for summarization problem. I’ve been trying to engineer Seq2Seq model for Summarizing Chats between two user agents.
I’ve tried T5 model (Pretrained & Transfer Learning), but the results were not satisfactory. The summarized text missed the context entirely after training on the custom dataset.
Can someone please help me understand which model works better for summarizing chats or any pre-processing task that precedes this.
Thanks in advance.

Context:
Hi @anant0308 ! Happy to discuss possible approaches, but what works best (and whether you can expect good results at all) will depend on what your fine-tuning data looks like: for example, how long are the chats? do you have any gold summaries for your chats? do you have examples of summaries without corresponding chats? how many examples do you have? how are you representing speaker turns?
Keep in mind that summarizing chats is quite a different task from summarizing news text: if the pre-training data lacks any kind of dialogue inputs, then the model will have to learn how to interpret multi-turn structure from scratch, which will probably be your main challenge.


Question:
Hey guys,
I’ve read that Perplexity (PPL) is one of the most common metrics for evaluating autoregressive and causal language models. But what do we use for MLMs like BERT?
I need to evaluate BERT models after pre-training and compare them to existing BERT models without going through downstream task GLUE-like benchmarks.
Best,
Vladimir

Context: I found an interesting project https://github.com/awslabs/mlm-scoring 98 which seems to be the step in the right direction. The authors also published the paper https://arxiv.org/pdf/1910.14659v2.pdf 21


Question: So I want to extract (prune) BERT-large such that I get BERT-base fairly. Initially I performed random pruning (near to 110M param count) on BERT-large but it didn’t seem to work well. L1 pruning seemed to work (nearly 131M param), but it doesn’t seem fair. Pre-training seems like a big hurdle given that there are some ambiguities on how to go about it. Please let me know if you’ve any suggestions on getting BERT-base fairly from BERT-large.

Context:
Have you tried Distilling it?
https://medium.com/huggingface/distilbert-8cf3380435b5 1 .
Why would you expect pruning to work?
(Why do you want to extract bert-base from bert-large?)


How I fine-tune BART for summarization using large texts?

Question:
Good night!
I’m using a pre-trained Bart for summarization and I have my own dataset for fine-tuning (which has a set with the big text and its respective summary). Despite this, my input texts are approximately 2500 characters long and the maximum Bart accepts is 1024. Is there any technique I can use to use all text? I thought of splitting each cell into smaller texts (max 1024) and assigning the same summary to each. Makes sense?
Example:
Before:
ABC: summary1
DEF: summary2
After:
A: summary1
B: summary1
C: summary1
D: summary2
E: summary2
F: summary2
Thanks in advance!

Context:
Hi, there’e already thread for this, you might find it helpful

    Summarization on long documents 🤗Transformers

    You can try extractive summarisation followed by abstractive. In the extractive step you choose top k sentences of which you choose top n allowed till model max length.
Another way is to use successive abstractive summarisation where you summarise in chunk of model max length and then again use it to summarise till the length you want. This method will be super expensive.
You can also combine first + second method.


New seq2seq tool: search hparam space with run_eval.py

Question:
FYI, there is a new tool available to you - you can now search the hparam space with run_eval.py.
It’s called run_eval_search.py
It uses the same arguments as run_eval.py, but allows you to parametrize the hparams, so in addition to the normal args you can pass:
--search="num_beams=8:11:15 length_penalty=0.9:1.0:1.1 early_stopping=true:false"

and it’ll search all the possible combinations and at the end print a table of results sorted by the scores of the task. e.g.:

bleu  | num_beams | length_penalty | early_stopping
----- | --------- | -------------- | --------------
41.35 |        11 |            1.1 |              0
41.33 |        11 |            1.0 |              0
41.33 |        11 |            1.1 |              1
41.32 |        15 |            1.1 |              0
41.29 |        15 |            1.1 |              1
41.28 |        15 |            1.0 |              0
41.25 |         8 |            1.1 |              0
41.24 |        11 |            1.0 |              1
41.23 |        11 |            0.9 |              0
41.20 |        15 |            1.0 |              1
41.18 |         8 |            1.0 |              0

You can have one or more params searched.
Here is an example of a full command:
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval_search.py \
facebook/wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt \
--reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json \
--bs $BS --task translation \
--search="num_beams=1:5 length_penalty=0.9:1.1 early_stopping=true:false"

If you encounter any issues please let me know.
It’s documented here: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md#run_eval-tips-and-tricks 3. @sshleifer and I added some more goodies in run_eval.py - you will find them all documented at that url.
Enjoy.
p.s. edited to remove things that are going to change based on Sam’s comment below.

Context:
Great work!
There are only two possible sets of keys to get from run_eval.py since
score_fn = calculate_bleu_score if "translation" in args.task else calculate_rouge
You shouldn’t hard code the possible tasks any more than that IMO.


Question: is there any way to use encoder part of T5 model for representation learning?

Context:
Hi @banucool
You can initialize the T5Model class and only forward pass through it’s encoder. The first element of the returned tuple is the final hidden states.
model = T5Model.from_pretrained("t5-small")
tok = T5Tokenizer.from_pretrained("t5-small")

enc = tok("some text", return_tensors="pt")

# forward pass through encoder only
output = model.encoder(
    input_ids=enc["input_ids"],
    attention_mask=enc["attention_mask"],
    return_dict=True
)
# get the final hidden states
emb = output.last_hidden_state

The shape of emb will be (batch_size, seq_len, hidden_size)


BART question, it seems that pretraining is not work for a small model?

Question:
What is your question?
My task is to generate keywords from sentences.
I pretrain a text-generation model. I mask the sentences’ tokens and predict the whole sentences’ tokens.
Pretraining batch_size = 8 and step = 1000000
I haven’t observed improvement from pretraining. BLEU score is 10.5 for not pretraining, BLEU score is 9.5 for pretraining.
Code
I take the python code from

      github.com

    google-research/pegasus/blob/master/pegasus/models/transformer.py#L38 1

from pegasus.layers import attention
from pegasus.layers import decoding
from pegasus.layers import embedding
from pegasus.layers import timing
from pegasus.layers import transformer_block
from pegasus.models import base
import tensorflow as tf
from tensorflow.contrib import layers as contrib_layers

class TransformerEncoderDecoderModel(base.BaseModel):
  """Transformer encoder+decoder.

  Notations:
    B: batch_size, I: max_input_len, T: max_target/decode_len, D: hidden_size
    V: vocab_size
  """

  def __init__(self, vocab_size, hidden_size, filter_size, num_heads,
               num_encoder_layers, num_decoder_layers, label_smoothing,
               dropout):

hidden_size = 512
num_encoder_layers = 3
num_decoder_layers = 3
Discussion
The task is to generate keyword from sentences.
The keyword may not appear in the sentences.
So input masked sentences to predict whole sentences, it is not benefit the keywords generation task.
Input masked sentences to predict whole sentences, it do not have relation to the keywords generation task.
Am I right? Is it the reason that pretraining do not improve the BLEU score?
Thank you very much.

Context: With all due respect, you are asking a question on a forum dedicated to a specific library transformers by HuggingFace, but the question does not involve that library. In fact, you are using a completely different library. I am not sure if this is the right place for such questions. @sgugger


Why are segment and position embeddings so large?

Question:
Cross-post from: https://forum.opennmt.net/t/size-of-feature-embeddings/3836 2
These days I am part-time doing work on improving translation models. We are working with regular transformer seq2seq networks using OpenNMT. This question is not about OpenNMT but it was triggered by going through its documentation. In onmt one can add features to each word 1. These features are then used to train their own embedding. For example, if you want to train a lower case model but still want to give importance to casing, you can add a casing feature that indicates whether the word was lower case or not.

i￨C like￨l cookies￨l from￨l new￨C york￨C

This will create two embedding layers under the hood. One for the tokens, and one for the case features.
In their documentation 1, they state that the default size for features is

… set to N^feat_vec_exponent where N is the number of values the feature takes.

where the default feat_vec_exponent  value is 0.7.
However, that means that for two features, they would only get a size of 1 or 2 (1.6). The embeddings (token and casing) are then concatenated. This contrasts sharply with the language models that I know. Take for instance, BERT, which has token (30k values), segment (two values), and position (512 values) which all have 512 dimensions, even the segment embeddings. These embeddings are summed.
My question thus ends up being: I always thought that the number of items in the embedding should more or less dictate the hidden size of that embedding (as onmt suggests), but BERT and siblings do not do this. So what is the best way, and why? How come that only two features in a 512 dimension space make sense?

Context:
It’s actually more a question of projecting in a high-dimensionality dense vector space versus a sparse space rather than the dimensionality it-self.
A lot of the recent developments in NLP are about projecting labels and tabular data in a high-dim vector space (assigning learned vectors to spare categorical features) prior to computation.
One striking demonstration of the efficiency of casting in high-dimension is in the work of John Wieting and Douwe Kiela: https://openreview.net/forum?id=BkgPajAcY7 2 but there is also a much older history of work on random projections and the Johnson-Lindenstrauss lemma: https://scikit-learn.org/stable/modules/random_projection.html A related discussion on the JL lemma you may want to join is here: https://github.com/huggingface/awesome-papers/discussions/7
Note however that there is a limit in the optimal dimension for the input embedding and recent models like ALBERT (https://openreview.net/forum?id=H1eA7AEtvS) or approach like Adaptive inputs (http://arxiv.org/abs/1809.10853) keep the input dimension smaller the models hidden-size to reach more optimal ratio between both of these dimensions.


Understanding what went wrong in attention

Question:
I am working on attention analysis. I want to learn more about where self attention made mistakes while attending to context query. Given two sentences, I am interested in learning more about where self-attention should have paid more attention (and not irrelevant tokens) to provide correct answers. In general, what went wrong in processing a given sample even if fine-tuned transformer is employed.
While there are projects based on visualization like BertViz, ExBERT, I am not sure if it’s straightforward to extract the information I’m looking for.
Do you know of any good projects, or workarounds in Transformers to answer my query ?

Context: Can anyone point me to the method on how to visualize attention in matrix form between query and context sentences ? Is there any other alternative ? Any pointers will be appreciated.


Question:
I had a great time at ACL this week. There are many great papers and I’m still going through them. Here’s a summary of just a few that I wanted to highlight. I’d love to get thoughts and retorts from anyone reading!
“To Test Machine Comprehension, Start by Defining Comprehension” 6
by Jesse Dunietz, Gregory Burnham, Akash Bharadwaj, Owen Rambow, Jennifer Chu-Carroll, and David Ferrucci
Like most great ideas, the framework presented here is simple – seemingly obvious, even. They take a specific look at Machine Reading Comprehension (MRC) and argue that current evaluation metrics don’t really inspire much confidence in the system’s comprehension of the relevant information in the passage to make it trust it in any real-world setting. They argue that rather than making questions harder, we should explicitly defining so-called “Templates of Understanding” to measure the different dimensions of comprehension within a particular context. For example, in the context of a story, they lay out the following ToU:
image2190×1140 238 KB
The authors do a great job thinking with clarity and simplicity about how we should approach evaluating MRC systems.
“Intermediate-Task Transfer Learning with Pretrained Language Models” 8
by Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut,
Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann,
Samuel R. Bowman
Recently the pre-train/fine-tune paradigm has become ubiquitous. This paper explores whether we can take advantage of labeled data during an intermediate training step. The authors do really extensive analysis on what kinds of datasets are useful for intermediate training and what downstream tasks they have a positive (or negative) effect on.
image1200×574 58.1 KB
A really interesting insight for me is that commonsense tasks don’t ever seem to have a negative effect. They either help on the downstream task, or don’t have much of an effect at all. I wonder if this because we do have labeled commonsense data that is used, or if we could build some kind of unsupervised commonsense objective into the pre-training procedure that would work just as well.
“Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data” 12
by Emily M. Bender and Alexander Koller
This paper is not focused around any one method or technique, but rather makes a general and pretty bold argument: meaning cannot be learned from form. In other words, just giving a model access to a whole bunch of text will never be enough to learn meaningfully about the real world.
Whether you buy their argument or not, I found it to be an intellectually stimulating presentation. I suspect the hyperintelligent octopus argument will be one that sticks around for a long time.
image2802×1558 296 KB
I also appreciated their word of caution about the way we use different words when communicating about a model’s capabilities. At the very end of the presentation, Alexander warned,

As a community, let’s remember that we’re scientists and not marketing people. Let’s be a little bit careful when we use terms like understanding, meaning, and comprehension.

Context: Intermediate Task Transfer is a very practical one. It exhaustively provides many results that can help engineers save much time.


Finetuning German BERT for QA on biomedical domain

Question:
Hello there and thank you very much for this wonderful work. I am relatively new to this field, so please bear with my amateur question. I want to perform question-answering on a German Biomedical text. From what I understand up to now, I need to fine-tune German BERT on biomedical QA datasets. Is there any script/pipeline that I should be using for this?
Thank you very much in advance.

Context: There is an example of script finetuning a model on question answering here 14, hope it can help!


Question:
I am interested in implementing the orthogonality portion of Towards Robustifying NLI Models Against Lexical Dataset Biases 4 from ACL 2020 in Pytorch.
The overall idea seems simple. Have a primary model and a sub-model (like BOW) to detect superficial features, and then use HEX projection (Wang et al., 2019a) to project the representation of the original primary model to the orthogonal space of the representation of the BoW model.
In this case, I would use a transformer as the primary model. I’m not sure about the implementation of HEX projection. If someone is familiar with it, it would be really helpful if they can share the snippet responsible for projecting the representation orthogonally.
Additionally, adding a debiasing example in Transformers repo would be a good addition which I’m happy to add, once I implement myself.

Context: I figured out myself from the equations in the paper (Wang et al.). My implementation seems to be working. Will share the link for my repo once I open source the code.


Question:
I am not sure I understand what it means to prime a LM. I came across this concept in several blogposts and papers (sometimes also referred to as exploring the capabilities of meta learning of the model or as in context learning).
From the openai gpt2 paper, section 3.7 Translation:
We test whether GPT-2 has begun to learn how to translate
from one language to another. In order to help it infer that
this is the desired task, we condition the language model
on a context of example pairs of the format english
sentence = french sentence and then after a final prompt of english sentence =

This I believe is an example of priming? Since with transformers there is no concept of hidden state being passed from one step to another, we provide the model with an input sequence of tokens of up to 1024 length and the model will output up to 1024 x vocab size softmax activations where each will encode the probability of the subsequent word (following the word at a given position). So priming would be just constructing the input sequence in a specific manner?
If I am reading this correctly, priming would refer to the act of passing a sequence into the model expecting that the model’s meta learning capability would affect its output?
In this sense, for priming, we are always limited to a sequence of < 1024 tokens (where 1024 need to suffice for the priming sequence and the output)?
Passing the past parameter just saves on compute, it provides the model with the key value pairs calculated at earlier steps of text generation but there is nothing else magical happening there?
And last but not least - are such questions okay to ask? Meaning, this would certainly qualify as a beginner question, but it doesn’t directly relate to the library I suppose. I really appreciate the amazing resource you put out there, the transformer library along with the wonderful documentation, in fact I am blown over by how awesome it is, just would like to make sure I am not bothering you with my questions and am using the forums in a way that they were intended to be used.
Thank you very much!

Context:
If I am reading this correctly, priming would refer to the act of passing a sequence into the model expecting that the model’s meta learning capability would affect its output?

You’ve nailed it on the head. When talking about a left-to-right model like GPT-N, priming is just prepending text that is similar in some way to the text you are predicting which often helps the model to predict it correctly.
Incidentally, this is the thing that GPT-3 seems to be especially good at. There seems to be something about language models that we don’t completely understand that can make priming a surprisingly effective meta-learning technique, especially when the models get really big. See this Twitter thread 54 for some examples.
And yes, this kind of question is perfect for the forums. However, I’d say Research is probably a better category fit since this more about general NLP/research talk and rather than the HF libraries


Question:
This may be not entirely about NLP.
I am working on Image captioning and learning textual representations from CNN features.
Idea is to train CNN using captioning. So, I tried to use GPT-2 tokenizer but I had to create Captioning model from scratch.
Is there any way to attach TF Transformer models to other CV applications for better learning?
My VirTex implementation in Keras 7

Context: Hey @surajp. Sorry, I’m not familiar enough with VirTex to give a concrete response here. But our TF models are compatible with TF2/Keras, so you should be able to include them in your TF graph. If you’re having trouble with this, please post some more specifics and I’ll see if we can be of any more help.


Is it reasonableto pretrain by masking certain dimensions of each vector, rather than the individual token?

Question:
Let’s say I want to adapt Transformers to a non-NLP task, like financial data or a multiplayer online video game. You can imagine that the high-dimensional vector of each input will contain information that pertain to different events. For example, the first 10 dimensions might describe player 1, and the next 10 dimensions might describe player 2.
If I were to extend the pre-training exercise to these non-NLP tasks, I think it could be reasonable to mask the actions of certain players in order to predict back their actions. This would essentially involve masking certain dimensions of a vector rather than masking the entire “input”.
My question is: is this reasonable to do and is this even the right approach?

Context: I don’t know what kind of input embeddings you’d be working with in that case, but the problem you’ll probably run into is that latent embeddings are usually not as nicely disentangled as you’ve described here. We sometimes talk about them as if they were for illustrative purposes, but in reality your description of “player 1” is probably distributed across the entire vector rather than existing entirely in some subset of vector positions.


Print All Tokens Over a Certain Probability Threshold

Question: I am curious to know how I would do this using GPT-2. Thank you for your time!

Context:
Hi there, here is a quick way to do this for the last token on a given sentence in PyTorch:
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
import torch.nn.functional as F

# Load model and tokenizer
model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

# Input example
input_txt = "Hello, my name is Sylvain."
inputs = tokenizer(input_txt, return_tensors='pt')
outputs = model(**inputs)

# If you are not on a source install, replace outputs.logits by outputs[0]
predictions = F.softmax(outputs.logits, dim=-1)

thresh = 1e-2
vocab_size = predictions.shape[-1]

# Predictions has one sentence (index 0) and we look at the last token predicted (-1)
idxs = torch.arange(0, vocab_size)[predictions[0][-1] >= thresh]
print(tokenizer.convert_ids_to_tokens(idxs))


Building a custom Squad 2.0 style dataset, is it worth it?

Question:
Was wondering what the experts think and whether this is a sensible approach. The pre-trained Squad 2.0 models perform well in a custom domain, but can be greatly improved, given the target domain is rather narrow and the vocabulary is different but there is overlap.
Do you think it is worth obtaining a custom dataset, say 1000 observations, using the same methodology as Squad v2.0 but derived from data of the target domain?
Is 1000 observation enough for the fine-tuning?

Context:
Hi @swayson, not an expert here but fine-tuning on your domain should give better results. I can’t comment on if 1000 examples will e enough or not, you’ll probably need to experiment.
Also have look at this question generation models 15. You can try to create synthetic QA corpora using these models. Synthetic QA corpora has shown to improve results for QA.


State of the art technique for initializing Embedding Matrix?

Question: What are your thoughts on the state-of-the-art technique for initializing Embedding Weight matrices? Currently, PyTorch uses normal distribution to initialize these. Does using Kaiming Init make more sense?

Context: From what I remember, Transformer modules should use Xavier init by default. I don’t remember the reason why, though, nor whether Kaiming is a better choice.