Question:
Hello everyone,
Is there any guide / example of how to do pretraining of ALBERT (in Tensorflow) using code only from huggingface?
I would really appreciate any help.
Thanks

Context: @sgugger any help here would be very appreciated


Question:
Is there any way to train a basic Transformer (the transformer architecture presented in the original Attention Is All You Need paper). I’m trying to set up a translation baseline on my custom dataset (previously trained models do not exist for the language). If anyone can point out a ready-made implementation of just the basic Transformer it would be very helpful.
Thanks!

Context: I think OpenNMT is the better framework to do something like that. It allows to train a model that you describe easily. transformers is more directed towards specialized, specific transformer architectures.


Question: Does anyone have a Google Colab notebook they can share? (If possible, one that has been adapted to the new model upload)

Context:
Hi, not exactly what you requested (ie. not the latest Huggingface version),
but it should give the main idea with a bonus on working with TPU:
Kaggle notebook on implementation of MLM finetuning on TPUs with a custom training loop in TensorFlow 2.
https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm 11


Question:
Hi,
For some reason I get KeyError(‘inputs’)  on any request to QA hosted inference like:
https://huggingface.co/deepset/bert-base-cased-squad2 6
Thanks for checking

Context: This is for @mfuntowicz


Difference between calling model() and using Trainer()?

Question:
Hi All,
I was wondering if there is any tangible difference between calling model() and feeding data in manually, or using the Trainer() object?
If I run a program to batch my data and feed it manually, will the training results be the same as using Trainer()?
I only ask because I am getting some errors while applying the language modelling protocol (from: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=YZ9HSQxAAbme) with the Trainer() module but manually seems to work fine.

Context:
This might be related, as it discusses a recently fixed bug with a Colan notebook.

    Using hyperparameter-search in Trainer 🤗Transformers

    Hi @sgugger, in case you’re not aware of it, it seems the latest commit on master broke the Colab notebook you shared on Twitter
Trying to run that notebook, I hit the following error when trying to run
best_run = trainer.hyperparameter_search(n_trials=10, direction="maximize")

with the optuna backend.
Stack trace:
/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:900: RuntimeWarning:

invalid value encountered in double_scalars

[W 2020-10-22 14:58:41,815] Trial 0 f…

But to answer your question concerning the results: the results when using the Trainer or your own trainer should be the same as long as you use the same loss function and hyperparameters.


Question:
Hello All. I apologize for the basic question but for some reason I am having difficulty using image features as input to a huggingface model.
My data comes in the form of an image feature numpy array extracted by a 2D CNN but all the models seem to be built for text based input.
If anybody could point me in the right direction of an example or a code snippet I would greatly appreciate it!

Context:
Hi CMPE-PL,
the transformers models are designed for Natural Language Processing, ie text.   What makes you think they would be good for image features?
I expect you could bypass the tokenizers and input numbers directly, but I’m not sure it would do anything useful.   If you did want to do that, you would need to ensure that your numbers were in the right format.  For example, a BERT model expects input that is a vector of 768 real numbers for each word-token, or rather a matrix of 768 x N real numbers, where N is number of word-tokens in a text.
What size is your image feature array?
The main trick of transformers is to use Attention mechanisms.  It is certainly possible to use Attention in image recognition models, but without using transformers. See this article for an example https://arxiv.org/abs/2004.13621 3


Question:
Hi everybody!
I’m just started on nlp and i’m working on my degree thesis, which involves experimenting with some dataset. I found the funcom dataset 5 which is made of pieces of java code and their javadocs. My question is, does anybody has ever tested sota models for summarization on this dataset? Would it give good results? Or the pretraining on such models does not give any knowledge on source code?
Thanks in advance

Context:
Hi airnicco8,
I’m not an expert, but that looks a bit tricky.   What would you intend to do with the funcom data?  Would you be trying to build a seq-2-seq model that could translate from java code to comment string?
If you are supposed to be doing NLP, then java code might not be appropriate, as java is not a Natural Language.
A big advantage of the huggingface library is that it includes many pre-trained models, that you can fine-tune to your own data.  I don’t think there are any models pre-trained on java code.  See this page for the list of models available in huggingface https://huggingface.co/transformers/pretrained_models.html 3
I suggest you start with something simpler.


TransformerXL run_clm.py grad can be implicitly created only for scalar outputs

Question:
Hello,
I am trying to execute run_clm.py for TransformerXL (transfo-xl-wt103) but get the following error:

0% 0/10170 [00:00<?, ?it/s]/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py:445: UserWarning: This overload of nonzero is deprecated:
nonzero()
Consider using one of the following signatures instead:
nonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
indices_i = mask_i.nonzero().squeeze()
Traceback (most recent call last):
File “language-modeling/run_clm.py”, line 352, in
main()
File “language-modeling/run_clm.py”, line 321, in main
model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
File “/usr/local/lib/python3.6/dist-packages/transformers/trainer.py”, line 775, in train
tr_loss += self.training_step(model, inputs)
File “/usr/local/lib/python3.6/dist-packages/transformers/trainer.py”, line 1126, in training_step
loss.backward()
File “/usr/local/lib/python3.6/dist-packages/torch/tensor.py”, line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/usr/local/lib/python3.6/dist-packages/torch/autograd/init.py”, line 126, in backward
grad_tensors_ = make_grads(tensors, grad_tensors)
File “/usr/local/lib/python3.6/dist-packages/torch/autograd/init.py”, line 50, in _make_grads
raise RuntimeError(“grad can be implicitly created only for scalar outputs”)
RuntimeError: grad can be implicitly created only for scalar outputs
0% 0/10170 [00:00<?, ?it/s]

I haven’t changed the original (run_clm.py) code. I am  using:

!python language-modeling/run_clm.py
–output_dir=’/content/drive/My Drive/XL-result’
–model_type=transfo-xl-wt103
–model_name_or_path=transfo-xl-wt103
–save_total_limit=2
–num_train_epochs=2
–do_train
–train_file=’/content/drive/My Drive/train.txt’
–do_eval
–validation_file=’/content/drive/My Drive/test.txt’
–per_device_train_batch_size=4
–per_device_train_batch_size=4
–learning_rate 5e-5
–seed 42
–overwrite_output_dir
–block_size 125

Any help would be much appreciated!

Context:
Have you read this?

      PyTorch Forums – 11 Jan 18

Loss.backward() raises error 'grad can be implicitly created only for scalar... 49

Hi,  loss.backward() do not go through while training and throws an error when on multiple GPUs using torch.nn.DataParallel  grad can be implicitly created only for scalar outputs  But, the same thing trains fine when I give only deviced_ids=[0] to...

  Reading time: 1 mins 🕑
    Likes: 48 ❤

If you haven’t changed the run_clm code, something else must be different. What versions of python, pytorch and huggingface_transformers are you using?
Are you using CPU/GPU/TPU ?  Are you using DataParallel (whatever that is)?


Multiple-Token Input for Text Generations and PPLM?

Question: Hello. I am trying to integrate the results of a LDA topic model and controlled-text generation, which is usually a set of keywords, to generate readable semantics/sentences. I have read some relevant papers and tried the codes at ‘transformers/examples/text-generation/pplm’ and ‘run_geneartion’, but still struggling to understand how to input “a list of strings” as input instead of a single string as the demo presents. Thank you!

Context:
This may help:
https://towardsdatascience.com/data-to-text-generation-with-t5-building-a-simple-yet-advanced-nlg-model-b5cce5a6df45 40


The reason `prepare_seq2seq_batch` for ProphetNet is not existed

Question:
Hi,
I tried to use ProphetNet with Seq2SeqTrainer, but it failed.
The error message tell me: This is because the collator I implemented uses prepare_seq2seq_batch() in _encode(), but prepare_seq2seq_batch() is not implemented for ProphetNet Tokenizer.
Is there any reason ProphetNet cannot have prepare_seq2seq_batch() in its tokenizer?
My understanding may be insufficient, but it seems that a function that assigns special tokens in a unique way is implemented for the tokenizer. Is that the cause?
If it is implemented like other Seq2SeqLM, will ProphetNet’s original performance not be exhibited?
Thank you in advance.
yusukemori

Context:
I don’t know about prophetnet special tokens, but it would be useful to implement that method!
cc @patrickvonplaten @valhalla


How to mask multiple tokens in BartForConditionalGeneration?

Question:
Hi!
How can I use BartForConditionalGeneration to predict multiple tokens instead of one? Or is the text filling task not (yet) supported?

Context:
I cant remember their name, but someone wrote this code allowing one to mask several tokens. If you replace Roberta with Bart, I expect it’ll work.
from transformers import RobertaTokenizer, RobertaForMaskedLM

import torch

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')

model = RobertaForMaskedLM.from_pretrained('roberta-base')

sentence = """My name is <mask> and I enjoy <mask>."""

token_ids = tokenizer.encode(sentence, return_tensors='pt')

# print(token_ids)

token_ids_tk = tokenizer.tokenize(sentence, return_tensors='pt')

print(token_ids_tk)

masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()

masked_pos = [mask.item() for mask in masked_position ]

print (masked_pos)

with torch.no_grad():

    output = model(token_ids)

last_hidden_state = output[0].squeeze()

print ("\n\n")

print ("sentence : ",sentence)

print ("\n")

list_of_list =[]

for mask_index in masked_pos:

    mask_hidden_state = last_hidden_state[mask_index]

    idx = torch.topk(mask_hidden_state, k=100, dim=0)[1]

    words = [tokenizer.decode(i.item()).strip() for i in idx]

    list_of_list.append(words)

    print (words)

best_guess = ""

for j in list_of_list:

    best_guess = best_guess+" "+j[0]


Question:
Hi,
Because of some dastardly security block, I’m unable to download a model (specifically distilbert-base-uncased) through my IDE. Specifically, I’m using simpletransformers (built on top of huggingface, or at least uses its models). I tried the from_pretrained method when using huggingface directly, also, but the error is the same:
OSError: Can’t load weights for ‘distilbert-base-uncased’
From where can I download this pretrained model so that I can load it locally?
Thanks very much,
Mark

Context:
You can now (since the switch to git models explained here: [Announcement] Model Versioning: Upcoming changes to the model hub 74) just git clone a model to your laptop and make from_pretrained point to it:
# In a google colab install git-lfs
!sudo apt-get install git-lfs
!git lfs install

# Then
!git clone https://huggingface.co/ORGANIZATION_OR_USER/MODEL_NAME

from transformers import AutoModel

model = AutoModel.from_pretrained('./MODEL_NAME')

For instance:
# In a google colab install git-lfs
!sudo apt-get install git-lfs
!git lfs install

# Then
!git clone https://huggingface.co/facebook/bart-base

from transformers import AutoModel

model = AutoModel.from_pretrained('./bart-base')

cc @julien-c for confirmation


Message “Some layers from the model were not used”

Question:
Hi,
I have a local Python 3.8 conda environment with tensorflow and transformers installed with pip (because conda does not install transformers with Python 3.8)
But I keep getting warning messages like “Some layers from the model checkpoint at (model-name) were not used when initializing (…)”
Even running the first simple example from the quick tour page generates 2 of these warning (although slightly different) as shown below
Code:
from transformers import pipeline
classifier = pipeline('sentiment-analysis')

Output:
Downloading: 100% 629/629 [00:11<00:00, 52.5B/s]
Downloading: 100% 268M/268M [00:11<00:00, 23.9MB/s]

Some layers from the model checkpoint at distilbert-base-uncased-finetuned-sst-2-english were not used when initializing TFDistilBertModel: ['pre_classifier', 'classifier', 'dropout_19']
- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased-finetuned-sst-2-english.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.

Downloading: 100% 232k/232k [00:02<00:00, 111kB/s]
Downloading: 100% 230/230 [00:01<00:00, 226B/s]

Some layers from the model checkpoint at distilbert-base-uncased-finetuned-sst-2-english were not used when initializing TFDistilBertForSequenceClassification: ['dropout_19']
- This IS expected if you are initializing TFDistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFDistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some layers of TFDistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased-finetuned-sst-2-english and are newly initialized: ['dropout_38']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

My configuration (‘transformers-cli env’ output):
2020-11-10 21:32:33.799767: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-11-10 21:32:33.804571: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
WARNING:tensorflow:From c:\users\basvdw\miniconda3\envs\lm38\lib\site-packages\transformers\commands\env.py:36: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2020-11-10 21:32:37.029143: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-10 21:32:37.049021: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x154dca447d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-10 21:32:37.055558: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-11-10 21:32:37.061622: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2020-11-10 21:32:37.065536: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2020-11-10 21:32:37.074543: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: S825
2020-11-10 21:32:37.080321: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: S825

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 3.5.0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.8.6
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: No:
- Using distributed or parallel set-up in script?: No

Does anyone know what causes this messages and how I could fix this ? I do not really understand the warning, because I thought I was using a pre-trained model which doesn’t need any more training…
Any help would be appreciated !

Context: Tagging @jplu so he’s aware, this looks like a bug (but you can proceed safely, the bug is that there is a warning when there should be nothing).


Question:
Hello.
I am playing around with the minimal number of tokens in the generated output by “google/pegasus-cnn_dailymail”. Basically I follow the documentation so my code looks like this
    batch = tokPeg.prepare_seq2seq_batch(src_texts=[s]).to(torch_device)
    gen = modelPeg.generate(**batch,
                        num_beams=int(8), min_lenght=100)
    summary: List[str] = tokPeg.batch_decode(gen, skip_special_tokens=True)

However, when I count number of tokens in output text by len(tokPeg.tokenize(summary[0])) the output text produces fewer tokens than is specified in min_length. Is there anything I am missing?

Context: this might be a red herring, but your code snippet shows “min_lenght” where it should be “min_length”


Restarting gpt-2 finetuning after power failure

Question:
Hi All,
I was several days in on finetuning the medium model when my machine shut down due to a power failure. How do I restart from the last saved checkpoint? I was using the following command:
python run_language_modeling.py --output_dir=output --model_type=gpt2 --model_name_or_path=D:\Development\models\gpt2-medium --do_train --train_data_file=./twitter_train.txt --do_eval --eval_data_file=./twitter_test.txt --per_device_train_batch_size=1 --overwrite_output_dir
Thanks!

Context:
[I am assuming that gpt-2 saving works in the same way as BERT saving.  I am not an expert.]
Hi @pgfeldman
did you save the optimizer state-dictionary?
In order to restart a previous training run, you need to have both the saved model state and the state of the optimizer’s parameters.  (These take up a surprisingly large amount of memory - about half the size of the model).
If you haven’t got the optimizer state-dict, then you can still load the saved model from the model checkpoint, but you will need to start a new training run .  You will probably need to estimate how far along the first run was, and what Learning Rate it might have got up to.
This thread might help:

    Loading finetuned model to generate text 🤗Transformers

    This is a basic question. Suppose I finetune a pretrained GPT2 model on my particular corpus. How do I go about loading that new model and generating some text from that new model?


Looking for ways to extract custom tokens from text

Question:
Hello community,
I am working on a project that requires extraction of a specific value from a text. Here is an example:
“This job offers a salary of $60000 and additional benefits like equity, health insurance and a private apartment”.
I want to be able to train a model that is able to recognize that $60000 is the salary of the job, but also be able to get the additional information that is related to the benefits like the equity and health insurance.
I have already solved this with a large corpus of regular expressions and manual text extraction, but as you are aware, there is always this one example that breaks the system. Therefore, I am hoping that I can use something to train my model model to recognize these “tokens”.
So in my internal language, the “$60000” is a token of the type “salary_value” and “equity”, “health insurance” and “private apartment” are tokens of the type “benefits”. There are a couple of other token types, but for the example let’s stay with this.
I have a lot of training data where these are annotated, so the text area that hast the token and what token is expected.
Can I use any of the hugging face libraries to build something similar? I have looked at the existing models, but they focus a lot on NER like “location”, “name”, “company”, etc.
I guess a good summary is that I am looking for some guidance on what to use best here.
Thanks!
Alex

Context:
it really looks like something called knowledge graph extraction, i remembered that I’ve seen something similar there, LSTM and convnet maybe transformers perform well there:

      Medium – 25 Oct 19

Knowledge extraction from unstructured texts 6

There is an unreasonable amount of information that can be extracted from what people publicly say on the internet. Learn how to do it.

  Reading time: 11 min read

you should really search about this
another links:
https://towardsdatascience.com/auto-generated-knowledge-graphs-92ca99a81121 1

      Programmer Backpack – 1 Feb 20

Python NLP Tutorial: Building A Knowledge Graph using Python and SpaCy 6

NLP tutorial for Information Extraction and building a Knowledge Graph in Python and spaCy.

      Mining Knowledge Graphs from Text

Mining Knowledge Graphs from Text 4

A Tutorial


How can I go about building Grammarly for my local language?

Question:
Hello Hugging Face forum members,
my name is Mislav, I’m a recent computer science graduate from Croatia. I had the idea of building Grammarly for Croatian Google Chrome plugin. I want to pursue that idea, but before I do that, I want to have a roadmap of what I would have to do in order to build it.
I am completely new to NLP and don’t have any experience with NLP, but I do have some basic experience with machine learning.
Here are the features that I want to build (in order). There will probably be others later on:

Spell checker
Tone detector - tells the user how does the text sound - confident, formal, informal etc.
Improvements suggesstor - suggests how the user can improve his/her text so that it resembles a certain tone better (i.e. how to make his/her message seem more formal)

I don’t know how Grammarly works - what does it use as a technology to detect grammar errors? What does it use to predict sentiment? What does it use to measure how engaging is the text? How does it judge the delivery of the message intended by the text? Essentially, if you had to reverse-engineer Grammarly and use that knowledge to help me build it, I’d be grateful.
I’m doing this project to both learn about NLP and ship the product, so ideally I could have a balance of learning & speed while working on this project. Maybe I could reuse some code from the Hugging Face repo? I intend to make the product a freemium product, so I’m not sure if that’s aligned with the code license.
Any suggestions as to how to go about building this would be appreciated. Again, I am new to this, so a  detailed description would go a long way.
Best,
Mislav

Context:
Hi Mislav
Although I respect your dedication, I urge you to be realistic. The road from “knowing nothing about NLP and just the basics of ML” to “building and shipping a whole NLP and deep learning focused product” is long. On top of that, this “Beginners” category is not a good fit for this kind of question. “Research” might be better suited.
I can’t give you a detailed description because “multiple roads lead to Rome”, in other words there are different ways to go about this. I’ll list some things that I think are important for you to get familiar with before even starting with this project. Don’t try to do too much at once. What you want to do takes years of expertise to get even the background right.

Read papers about readability. This is probably the most important field that you need to get a theoretical understanding of if you want to do things right. Grammarly does not only do a “right or wrong” prediction, but it also suggests improvements. Those improvements increase readability, which is quite a big field in (psycho)linguistics. This paper 4 is a good (recent) starting point, code available here 5. However, I’d also suggest to read up on the theoretical work that has been done way back concerning readability formulas to get a better understanding of the problem.
For spell checking you need to look into (grammatical) error correction. A lot of work has been done and lately it has been influenced by many neighbouring fields such as machine translation and quality estimation. From a programming perspective, here you may want to look into seq2seq and denoising auto-encoders, although I have to admit that I do not know what the SOTA is these days.

Sentiment analysis will be another large chunk of what you want to do, as its principles can be applied to style as well. You will find a lot information about this. The difficult part is not necessarily detecting which style a text has, but highlighting in that text which spans are confidently the points of interest that contribute to this sentiment. The problem, as so often with neural systems these days, is ensuring that there is some linguistic validation there and that the predicted spans are actually meaningful. (For instance, you may want to exclude prepositions and proper nouns from the result.)
Perhaps the hardest part is the “improvement suggestor” that you mention. You may want to look into the task of paraphrasing which also has gained some attention over the last few years. Style transfer also seems to be useful here.

As a first step, you should read up on all subfields above. Second, you can start learning how to implement these kinds of things, and lastly you can try to bring them all together into a single system with an interface.
As is hopefully clear to you now, this is not a one day, not even a one year, job. If it interests you, go for it! But be aware that if you want to do this right and actually understand what you are doing, that it will take time.
Good luck!
Bram


Question:
Am I reading this thread (Training using multiple GPUs 50) correctly? I interpret that to mean:

Training a model with batch size 16 on one GPU is equivalent to running a model with batch size 4 on 4 GPUs

Is that correct? And does it differ between DataParallel and DistributedDataParallel modes?

Context:
It is correct. The difference between DataParallel  and  DistributedDataParallel is (in your current example):

in DataParallel mode you have to set the batch size to 16 for your data loaders.
in DistributedDataParallel you have to set the batch size to 4 for your data loaders.


Question:
Hi all,
I would like to train a T5 model (t5-base version) without loading the pretrained weights, if I write the following:
from transformers import T5Config, T5Model
config = T5Config.from_pretrained(‘t5-base’)
model = T5Model(config)
It will produce the t5-base sized T5 model without loading the checkpoint weights?
Thanks

Context:
sarapapi:

from transformers import T5Config, T5Model
config = T5Config.from_pretrained(‘t5-base’)
model = T5Model(config)

Yes, that’s correct. You will definitely see the “pretrained weights” download progress bar in the other case


Question:
Hi! I am pretty new to Hugging Face and I am struggling with next sentence prediction model. I would like it to use a GPU device inside a Colab Notebook but I am not able to do it. This is my proposal:
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased', return_dict=True)
model.to("cuda:0")
prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
next_sentence = "The sky is blue due to the shorter wavelength of blue light."
tokenizer_output = tokenizer(prompt, next_sentence, return_tensors='pt')
tokens_tensor = tokenizer_output['input_ids'].to('cuda:0')
token_type_ids = tokenizer_output['token_type_ids'].to('cuda:0')
attention_mask = tokenizer_output['attention_mask'].to('cuda:0')
encoding = {'input_ids' : tokens_tensor,
        'token_type_ids' : token_type_ids,
        'attention_mask' : attention_mask}
outputs = model(**encoding)
logits = outputs.logits
print(logits) # next sentence was random

However, it does not work. The GPU device is properly set, since I used the following command to check it:
import torch; print(torch.cuda.get_device_name(0))

I would appreciate any help

Context: What do you mean by “doesn’t work”. What errors do you get or how do you know the GPU isn’t used.


Question:
Hi, I’m running run_clm.py to fine-tune gpt-2 form the huggingface library, following the language_modeling example:
!python run_clm.py \
    --model_name_or_path gpt2 \
    --train_file train.txt \
    --validation_file test.txt \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-clm

This is the output, the process seemed to be started but there was the ^C appeared to stop the process:
The following columns in the training set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
The following columns in the evaluation set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
***** Running training *****
  Num examples = 2318
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 870
  0% 0/870 [00:00<?, ?it/s]^C

Here’s my environment info:

transformers version: 3.4.0
Platform: Linux-4.19.112±x86_64-with-Ubuntu-18.04-bionic
Python version: 3.6.9
Tensorflow version: 1.14
Using GPU in script?: yes

What would be the possible triggers of the early stopping?

Context: Hi, if you are running Tensorflow, don’t you need TFGPT2LMHeadModel rather than GPT2LMHeadModel ?


Customize tokenizer in model card’s widget

Question:
I trained a Chinese Roberta model. In the model card, the widget uses a tokenizer defined in config.json( RobertaTokenizer ). But my model uses  BertTokenizer . Can I customize the tokenizer in the widget of the model card just like I can choose any combination of model and tokenizer in a pipeline?
I tried to use  BertModel  instead of  RobertaModel  (copy weights from Roberta to Bert). But the position embedding is different. And the outputs are different… So I have to use this combination of  RobertaModel  and  BertTokenizer . Is that mean I can’t use the inference widget?

Context: We are working on adding this indeed (as I understand it’s the same as https://github.com/huggingface/transformers/issues/8125 12)


Estimating Training Time for Fine Tuning

Question:
I have a dataset with about 15,000 documents, containing about 7GB of uncompressed text, which I’m hoping to use for fine-tuning the Megatron-11b model 6, but before I get started, I’d like to get a rough estimate of the cost.
If I train in the cloud, I’ll probably use AWS P3 2 or G4 1 instances. Roughly how many instance-hours would I expect to need, to complete the fine-tuning?
At a certain price point, I might just decide to buy my own hardware, and do the training on-prem… What the minimum specs of hardware I’d need to be able to perform this task, without running out of memory? The pre-trained Megatron_11b model is 19GB of data… So does that mean I could perform model fine-tuning with only a single 24GB graphics card? Or would I need more than that?

Context: maybe @teven?


Bert LM pretraining: training loss goes to 0 at masking probability of 0.999

Question:
Hi, I am using the Trainer class to perform masked language modeling with a pretrained Bert checkpoint (fine-tune on own domain). I’m using the official run_language_modeling.py script from
https://github.com/huggingface/transformers/tree/master/examples/language-modeling 8. I merely changed it to use a different dataset implementation, as the default ones load the full data into memory. But __getitem__ works the same, returning a token-id converted sequence at a time, so this should not matter.
Even when setting the mlm probability to something ridiculous like 0.999, this is what I observe:
image2400×745 46.6 KB
The loss becomes zero rather quickly (full dataset would be 40k training steps)
This is the command I’m using, working on google cloud.
python3 xla_spawn.py --num_cores=8 train_mlm.py \
--output_dir=real_runs/6 \
--model_type=bert \
--model_name_or_path=Rostlab/prot_bert \
--do_train \
--train_data_file=/home/preprocessed_allorgs_alllengths.txt \
--mlm \
--line_by_line \
--block_size 512 \
--max_steps 30000 \
--out_of_core  \
--logging_steps 20 \
--learning_rate 0.00001 \
--per_device_train_batch_size 20 \
--lazy \
--run_name high_mlm_prob \
--save_steps 2000 \
--warmup_steps 2666 \
--weight_decay 0.01 \
--mlm_probability=0.9999

(train_mlm.py is run_language_modeling.py with a custom dataset class, --lazy is an arg for that dataset. The rest is default, didn’t change anything about it.)
Am I missing something with regards to the setup? This just seems wrong in general, and not an issue with the dataset.

Context:
Maybe the model is almost perfect already (?)
What is mLm probability, and is it supposed to have a capital L?
Is your data very similar to the data originally used to pre-train BERT?
What value of loss do you get if you try to fine-tune using the data originally used to pre-train BERT?
Maybe there are a few differences, but BERT has encountered enough examples of each difference to learn the patterns within the first 150 steps.


Correct way to structure BERT for genetic segmentation?

Question:
I’m doing research on gene segmentation, and I’m currently trying to implement a BERT network.
I have a database of 350 GB database (~300M proteins) of unlabelled proteins, where each protein is a sequence of letters with various lengths, and each letter correspond to an amino acid. There are only about 20 different amino acids, so they are easy to tokenize. And should be easy to use for a BERT model to predict.
(here is an example of 3 such proteins)
DGQPEIPAGRGEHPQGIPEDTSPNDIMSEVDLQMEFATRIAMESQLGDTLKSRLRISNAQTTDTGNYTCQPTTASSASVLVHVINGE
AGQLWLSIGLISGDDSLDTREGVDLVLKCRFTEHYDSTDFTFYWARWTCCPTLFENVAIGDVQLNSNYRLDFRPSRGIYDLQIKNTSYNRDNGRFECRIKAKGTGADVHQEFYNLTVLTAPHPPMVTPGNLAVATEEKPLELTCSSIGGSPDPMITWYREGSTVPLQSYALKGGSKNHYTNATLQIVPRRADDGAKYKCVVWNRAMPEGHMLETSVTLNVNYYPRVEVGPQNPLKVERDHVAKLDCRVDAKPMVSNVRWSRNGQYVSATPTHTIYRVNRHHAGKYTCSADNGLGKTGEKDIVLDVLYPPIVFIESKTHEAEEGETVLIRCNVTANPSPINVEWLKEGAPDFRYTGELLTLGSVRAEHAGNYICRSVNIMQPFSSKRVEGVGNSTVALLVRHRPGQAYITPNKPVVHVGNGVTLTCSANPPGWPVPQYRWFRDMDGDIGNTQKILAQGPQYSIPKAHLGSEGKYHCHAVNELGIGKIATIILEVHQPPQFLAKLQQHMTRRVGDVDYAVTCSAKGKPTPQIRWIKDGTEILPTRKMFDIRTTPTDAGGGVVAVQSILRFRGKARPNGNQLLPNDRGLYTCLYENDVNSANSSMHLRIEHEPIVIHQYNKVAYDLRESAEVVCRVQAYPKPEFQWQYGNNPSPLTMSSDGHYEISTRMENNDVYTSILRIAHLQHSDYGEYICRAVNPLDSIRAPIRLQPKGSPEKPTNLKILEVGHNYAVLNWTPGFNGGFMSTKYLVSYRRVATPREQTLSDCSGNGYIPSYQISSSSSNSNHEWIEFNCFKENPCKLAPLDQHQSYMFKVYALNSKGTSGYSNEILATTKVSKIPPPLHVSYDPNSHVLGINVAATCLSLIAVVESLVTRDATVPMWEIVETLTLLPSGSETTFKEAIINHVSRPAHYTTATTSGRSLGVGGGSHLGEDRTMALAETAGPGPVVRVKLCLRSNHEHCGAYANAEIGKSYMPHKSSMTTSALVAIIIASLSFVVFLGLLYAFCHCRRKHAAKKESSSVGGGVGGGNANATANPGSTGAKEYDLDLDASRRPSLSQDPQQSQQQPPPPPPYYPTGTLDSKDIGNGNGGMELTLTALHDPDEQLNMQQQQHHSNHGQYQQPKAILGIYGGVAGSGGNNSGGQHPHSNGYGYHVTSAIGVDSDSYQVLPSVANSAAGSHGHGSGHGHGLGAGEXPLEATPPTCNISGGSSSNSGINPMQQQHSARANLTNQPTIATASSTNNYNNHLNNTNIAHTTNNTNNCTTLKRGHLGNRERERERCQVTAATAATTTALATTITTTSRNAKAATTTTTLAITGSSSNSNENNYSNARDLSQEALWRLQMATAQSQQIYVERPPSAFSGLVDYSGYSPHIPTVTSSLSQQSFSPTQQLAPHEMLQAAQRYGTLRKSGKQPPLPPQRKDMQQQAKPPQQMADIIQDLAN
MKQINAASALCGQLKQHENRAGPSNLGNVISQILLCKQFTPDFNEEELCSITKDSQDIAVLLAEMQEYMPQHEAYLERNAALDTTGPWQAKRRQNYICKNMSLLCCVS

Furthermore, I have a few small databases with 800-25000 proteins of labelled proteins, meaning for each amino acid in each protein it contains 1 of three labels (0 = postive, 1 = negative, 2 = unknown)
So in this case my data would look like (2 small proteins shown here):
DGQPEIPAGRGEHPQGI, 22222111111222222
YLERNAALDTTGPWQA, 00002222222222222

So far I have made a standard BERT model, with a linear layer on top (see code below) that I can use in a cross entropy loss function to predict the 15% masked amino acids in the proteins. It should be noted here that I do not start each input sequence with [cls] and end with [sep] since each sequence is only one protein I figured that I could do without this (Though maybe this is wrong and should still be included?)
class BERTseq(nn.Module):
    def __init__(self, bert: BERT, vocab_size):

        super().__init__()
        self.bert = bert
        self.linear = nn.Linear(self.bert.hidden, vocab_size)

    def forward(self, x, segment_label):
        x = self.bert(x, segment_label)
        return self.linear(x)

Now this part seems to work rather well. After training for a few hours I get 96% accuracy on testing data when predicting these proteins.

So now I want to try and use this pretrained network for the segmentation task, but I must admit I’m uncertain how exactly I should do this, my initial idea was to remove the last linear layer from before and insert a new linear layer with size nn.Linear(self.bert.hidden, 2), since I have 2 segmentation classes I want out.
However in this case I don’t know whether I should still use the masked approach from before? or should I just use the full label segmentation as input?, Also should I keep the old linear layer at the end and add another layer on top of it, or replace it with the new linear layer?
So far I have been trying to replace it and use the segmentation labels directly in BERT, but that doesn’t seem to work.

Context:
Is the sequence IPAGRG always going to be Negative, or does it depend on what is around it?
Is IPAGRG a commonly-occurring subset?
When IPAGRG occur together, will they always have the same positivity.   That is, will IPAGRG always be either 000000, 111111 or 222222, or could it be 012201 etc?
I think you somehow need to have a single “answer” for each sequence that you give to BERT, so that it can learn.
(I’m a bit worried that if you give it a label such as 22222111111222222 it will treat that as a single numerical value, which is not what you want).
Could you treat IPAGRG as a single token, and DGQPE as another, and EHPQGI as another, so that BERT could try to learn to predict the value (positive/negative/unknown) of the central sequence in each input of 3 sequences?
From your Masked prediction success, it seems that there must be some common sequences.   If you don’t have any a priori idea of what these might be, you could maybe try an n-gram analysis.
Completely different idea:
Could you use 60 tokens instead of 20, where each token includes information about the amino acid AND the positivity?
So,
token A0 could represent alanine with positive,
token  A1 could represent alanine with negative,
token A2 could represent alanine with unknown.
By the way, it’s not relevant to the BERT implementation, but what do the positive/negative labels actually signify?


Question:
I am using the transformers library to get embeddings for sentences and tokens. More  specifically I use the first token embedding [CLS] for the embedding that represents the sentence and I compare sentences using cosine similarity. This  approach is naive and completely unsupervised.
Now I would like to gain some experience in fine tuning  the model: For example how to fine tune BERT for NER and how to use BERT for sentence pairs. How can I study how  to fine tune BERT?

Context:
nice tutorial on tuning using transformers with pytorch by Chris McCormick here https://mccormickml.com/2019/07/22/BERT-fine-tuning/#4-train-our-classification-model 183.  See also his youtube videos and other posts.
If you want to make a custom model, try this by abhimishra https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb 128


How to load pre-trained model parameters only in specific layers?

Question: Hey , I want to know how to load pre-trained model parameters only in specific layers ? For example, I use EncoderDecoderModel class (bert-base-uncased2bert-base-uncased model) .  And I only want to load parameters in  specific layers like 2 or 10 of the pretrained model . How can I do this ?

Context:
guoziyuan:

or 10 of the pretrained

This is usually what I did:
original_model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
my_model = BartForConditionalGeneration.from_pretrained('facebook/bart-base', encoder_layers=2)
## for example, I want my model's last encoder layer's weight = original model's last encoder's weight
model.model.encoder.layers[1].load_state_dict(original_model.model.encoder.layers[5].state_dict())


Unexpected keyword argument ‘return_dict’ in BertForSequenceClassification

Question:
My transformers version is 3.4.0. When my code is here:
from transformers import BertTokenizer, BertForSequenceClassification
import torch
bert_config = BertConfig.from_json_file('torch_bert_chinese/config.json')
model = BertForSequenceClassification.from_pretrained('torch_bert_chinese/', from_tf=False, config=bert_config, local_files_only=True, return_dict=True)

The exception is here:
TypeError                                 Traceback (most recent call last)
<ipython-input-33-08c5f3693153> in <module>
      5 bert_config = BertConfig.from_json_file('torch_bert_chinese/config.json')
      6 model = BertForSequenceClassification.from_pretrained('torch_bert_chinese/', from_tf=False, config=bert_config, local_files_only=True,
----> 7                                                      return_dict=True)
      8

/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    945
    946         # Instantiate model.
--> 947         model = cls(config, *model_args, **model_kwargs)
    948
    949         if state_dict is None and not from_tf:

TypeError: __init__() got an unexpected keyword argument 'return_dict'

Context: If you are instantiating your model with a config, you need to pass this return_dict=True to the config create (so the line above in your case).


What does EvalPrediction.predictions contain exactly?

Question:
I want to implement a function for computing metrics and pass it to the Trainer. In the doc, EvalPrediction has 2 attributes: predictions and label_ids. It is written that both of them are of type ndarray but this is not the case for me.
The label_ids have is correct. It is ndarray and has shape (4, seqlen) where 4 is the number of samples in my validation. However, the attribute predictions are a tuple?
At index 0, I have an array of sizes (3, 4, 56, 32104). 4 again is the number of samples 56 is the sequence length and 32104 is the vocabulary size but what is the 3 then?
At index 1 I have first a tuple/list of tuples with size 4,6 and then an array of 4, 8, 56, 64
And at index 2 I have an array of size 4, 78, 512.
What are all these arrays actually? I think this should be clarified in the documentation.
Thanks for your help!

Context: The Trainer will put in predictions everything your model returns (apart from the loss). So if you get multiple arrays, it’s likely because your model returns multiple things. No one can help you determine what they are without seeing your model (which is why you should always post the code you’re using when asking for help  )


“IndexError: index out of range in self” for bert LM example on https://huggingface.co/transformers/quickstart.html

Question:
Hi!
I was trying to use my own data for the language model example (BERT) mentioned here:

      huggingface.co

Quickstart — transformers 2.11.0 documentation 8

However, I get an IndexError: index out of range in self when I use my own data. At first I thought that it is related to the sequence length but I also get the error for sequences smaller than <512.
The code is:
tokenized_text = ['[CLS]', '#', '#', 'steps', '[SEP]', '1', '.', 'if', 'the', 'area', 'is', 'hot', 'or', 'in', '##fl', '##ame', '##d', 'after', 'your', 'laser', 'tattoo', 'removal', 'session', 'you', 'can', 'apply', 'an', 'ice', 'pack', 'wrapped', 'in', 'a', 'damp', 'cloth', '.', '[SEP]', '2', '.', 'over', 'the', 'counter', 'pain', 'relief', 'such', 'as', 'para', '##ce', '##tam', '##ol', 'can', 'help', 'by', 'reducing', 'any', 'temporary', 'pain', '.', '[SEP]', '3', '.', 'el', '##eva', '##te', 'the', 'area', 'is', 'its', 'an', 'ex', '##tre', '##mity', 'such', 'as', 'a', 'wrist', 'or', 'ankle', 'to', 'reduce', 'swelling', '.', '[SEP]', 'keep', 'the', 'tattoo', 'site', 'clean', 'and', 'dry', 'and', 'avoid', 'soaking', '[MASK]', 'in', 'the', 'first', 'week', 'or', 'two', 'during', 'the', 'healing', 'stage', '.', '[SEP]']
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
index_of_masked_token = tokenized_text.index('[MASK]')
# make the segments_ids
counter = 0
segments_ids = []
for token in tokenized_text:
      segments_ids.append(counter)
      if token == '[SEP]':
         counter +=1
# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
print("size of tokens in tensor {0}".format(tokens_tensor.shape))
print("size of segment tokens in tensor {0}".format(segments_tensors.shape))

# Load pre-trained model (weights)
model = BertModel.from_pretrained('bert-base-uncased')
model.eval()

# Predict hidden states features for each layer
with torch.no_grad():
     outputs = model(tokens_tensor, token_type_ids=segments_tensors)
     encoded_layers = outputs[0]
assert tuple(encoded_layers.shape) == (1, len(indexed_tokens), model.config.hidden_size)

# Load pre-trained model (weights)
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()

# Predict all tokens
with torch.no_grad():
       # error is caused by the line below.
        outputs = model(tokens_tensor, token_type_ids=segments_tensors)
        predictions = outputs[0]

The error is:
      File "/Users/talita/Documents/PhD/corpora/rulebook_diffs/2019-09-23/boardgame_scripts/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
      File "/Users/talita/Documents/PhD/corpora/rulebook_diffs/2019-09-23/boardgame_scripts/venv/lib/python3.8/site-packages/transformers/modeling_bert.py", line 752, in forward
    embedding_output = self.embeddings(
      File "/Users/talita/Documents/PhD/corpora/rulebook_diffs/2019-09-23/boardgame_scripts/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
      File "/Users/talita/Documents/PhD/corpora/rulebook_diffs/2019-09-23/boardgame_scripts/venv/lib/python3.8/site-packages/transformers/modeling_bert.py", line 180, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
      File "/Users/talita/Documents/PhD/corpora/rulebook_diffs/2019-09-23/boardgame_scripts/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
      File "/Users/talita/Documents/PhD/corpora/rulebook_diffs/2019-09-23/boardgame_scripts/venv/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 124, in forward
    return F.embedding(
      File "/Users/talita/Documents/PhD/corpora/rulebook_diffs/2019-09-23/boardgame_scripts/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 1852, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
    IndexError: index out of range in self

Does anyone know why I get this IndexError?

Context:
You are training to process multiple sentences separated by SEP tokens. That does not make sense, but it is not clear what you are trying to do.
The index error is raised because BERT was pretrained with only two segment IDs, 0 and 1. Those were needed for the NSP objective. It therefore does not make sense to add more segments/segment IDs - and that simply won’t work as you found out. Perhaps you should start by explaining what you are trying to do.
I highly recommend you to read the original BERT paper to better understand this.


Question:
System Setup

Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 3.0.2
Python: 3.7.6

Background Code
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch
import time
import functools

def time_gpt2_gen():
    prompt1 = 'We present an update on the results of the Double Chooz experiment. Double Chooz searches for the neutrino mixing angle, ÃÂ¸13, in the three-neutrino mixing matrix via the disappearance of produced by the dual 4.27 GW/th Chooz B Reactors. Here we discuss updated oscillation fit results using both the rate and the shape of the anti-neutrino energy spectrum. In the most recent oscillation analysis we included data with neutron captures on Gadolinium and Hydrogen along with the reactor off data that we collected. This is an important step in our multi-year program to establish the value of ÃÂ¸13.'
    prompt2 = 'The paper covers detailed discussion on novel control system developed for adaptive fluid-based shock-absorbers serving for mitigation of unknown impact excitations. In order to provide complete independence of the control system from the loading conditions, the Hybrid Prediction Control (HPC) was elaborated. The proposed method is an extension of previously introduced kinematic feedback control which ensures optimal path finding, tracking and path update in case of high disturbance or sudden change of loading conditions. Implementation of the presented control system allows to obtain self-adaptive fluid-based absorbers providing robust impact mitigation. In contrast to previously developed methods of Adaptive Impact Absorption, the proposed control strategy does not require prior knowledge of impact excitation or its preliminary identification. The independence of applied control system from parameters of impact loading results in the capability of automatic path correction in the case of disturbance occurrence and re-adaptation to a number of subsequent impacts. The successful operation of the self-adaptive system is investigated with the use of numerical examples involving double-chamber pneumatic shock-absorber equipped with controllable valve. Efficiency of the HPC is proved by comparison with passive absorber as well as device equipped with adaptive and optimal control modules.'
    prompt3 = 'This study aimed to produce biosurfactant from Pseudozyma tsukubaensis using cassava wastewater and an inoculum (biomass) for galactooligosaccharides synthesis from lactose as an integrated system. First, the use of cassava wastewater as a low cost culture medium by P. tsukubaensis to produce biomass and biosurfactant was evaluated and optimized. Then, the microbial cells (biomass) obtained from the optimized process were used to produce galactooligosaccharides from lactose. The optimum conditions for biosurfactant and biomass synthesis were found to be 80% (v/v) of cassava wastewater at 30ÃÂ°C and 200rpm for 48h. The highest concentration of biosurfactant, that is, minimum surface tension value and maximum biomass concentration predicted were experimentally confirmed as 26.87mN/m and 10.5g/L, respectively. The biosurfactant obtained showed good thermal (121ÃÂ°C/1h), pH (2Ã¢ÂÂ11) and ionic strength (0Ã¢ÂÂ25% NaCl) stability. Excellent emulsifier activity was also verified, suggesting a potential application in enhanced oil recovery. Galactooligosaccharides synthesized by the Kluyveromyces genus have been extensively investigated, however, few studies have reported transgalactosylation ability by other yeast genera. The transgalactosylation activity of the yeast biomass at optimized conditions from 40% (w/w) lactose resulted in galactooligosaccharides production of 73.12g/L and a yield of 18.28% (w/w) at pH 8.0 and 30ÃÂ°C in 24h. This research showed the technical feasibility of an integrated process: biosurfactant and GOS production from P. tsukubaensis, which takes advantage of the remarkable metabolism of this microorganism. To the best of our knowledge, this is the first study reporting the potential of P. tsukubaensis to produce two economical biotechnological products of increase interest as an integrated process.'
    prompt4 = 'Advantages of a fuzzy predictive control algorithm are discussed in the paper. The fuzzy predictive algorithm is a combination of a DMC (Dynamic Matrix Control) algorithm and TakagiÃ¢ÂÂSugeno fuzzy modeling, thus it inherits advantages of both techniques. The algorithm is numerically effective. It is in fact generalization of the standard DMC algorithm widely used in the industry, thus the existing implementations of the DMC algorithm can be extended using the presented fuzzy approach. A simple and easy to apply method of fuzzy predictive control algorithms synthesis is presented in the paper. It can be easy applied also in the case of Multiple Input Multiple Output (MIMO) control plants. Moreover, information about measured disturbance can be included in the algorithms in an easy way. The advantages of the fuzzy predictive control algorithm are demonstrated in the example control systems of two nonlinear chemical reactors: the first oneÃ¢ÂÂwith inverse response and the second oneÃ¢ÂÂa MIMO plant with time delay.'
    batch = [prompt1, prompt2, prompt3, prompt4]

    tokenizer = run_func(GPT2Tokenizer.from_pretrained, 'gpt2', print_str='Initialize GPT2Tokenizer', padding_side='right')
    tokenizer.pad_token = tokenizer.eos_token
    encoded_results = run_func(tokenizer, batch, print_str='Calculate initial encodings', padding=True, truncation=True, return_tensors='pt', return_attention_mask=True)

    gpt2 = run_func(GPT2LMHeadModel.from_pretrained, 'gpt2', print_str='Initialize GPT2LMHeadModel')

    temperature = 0.92
    tmp_input_ids = encoded_results['input_ids']
    tmp_attention_mask = encoded_results['attention_mask']
    max_gen_length = 30
    counter = 0
    gen_dict = {'a1': '', 'a2': '', 'a3': '', 'a4': ''}
    while_tic = time.perf_counter()
    while counter < max_gen_length:
        print('\ncounter = {}'.format(counter))

        outputs = run_func(gpt2, print_str='   Calculate GPT2 outputs', input_ids=tmp_input_ids,
                           attention_mask=tmp_attention_mask
                           )

        # (batch_size, sequence_length, vocab_size)
        lm_logits_w_temp = outputs[0] / temperature

        # (batch_size, vocab_size)
        last_tokens = lm_logits_w_temp[:, -1, :]
        last_token_softmaxes = run_func(torch.softmax, last_tokens, print_str='   Last token softmax', dim=-1).squeeze()

        next_tokens = run_func(torch.multinomial, last_token_softmaxes, print_str='   Generate next token', num_samples=1)

        list_comp_tic = time.perf_counter()
        next_strs = [tokenizer.decode(next_token).strip() for next_token in next_tokens]
        prev_input_strs = [tokenizer.decode(id_tensor, skip_special_tokens=True) for id_tensor in tmp_input_ids]
        prev_split_list = [prev_input_str.split() for prev_input_str in prev_input_strs]
        list_comp_toc = time.perf_counter()
        print('   List comprehension calcs elapsed time: {} seconds'.format(list_comp_toc-list_comp_tic))

        gen_dict['a1'] += next_strs[0] + ' '
        gen_dict['a2'] += next_strs[1] + ' '
        gen_dict['a3'] += next_strs[2] + ' '
        gen_dict['a4'] += next_strs[3] + ' '
        str_list_to_join = []

        next_strs_tic = time.perf_counter()
        for ii, prev_split2 in enumerate(prev_split_list):
            next_str = next_strs[ii]
            tmp_prev = prev_split2
            tmp_prev.append(next_str)
            str_list_to_join.append(tmp_prev)
        next_inputs = [' '.join(str_to_join) for str_to_join in str_list_to_join]
        next_strs_toc = time.perf_counter()
        print('   Add generated tokens onto previous full strings elapsed time: {} seconds'.format(next_strs_toc-next_strs_tic))

        if counter == max_gen_length - 1:
            final_str_batch = next_inputs
        else:
            new_encoded_results = run_func(tokenizer, next_inputs, print_str='   Tokenizing next full strings', padding=True, truncation=True, return_tensors='pt',
                                            return_attention_mask=True)
            tmp_input_ids = new_encoded_results['input_ids']
            tmp_attention_mask = new_encoded_results['attention_mask']

        counter += 1

    while_toc = time.perf_counter()
    print('Time to complete while loop for {} passes: {} seconds'.format(max_gen_length, while_toc-while_tic))
    print('------------------------------------------------------------------------------------')

run_func(time_gpt2_gen, print_str='Total time')

Question
I was wondering what I could do to speed up the generation of GPT2? I feel like this is a pretty naive implementation so I’d love to here feedback on some speed optimization strategies. Here is the output for generating 30 new tokens on a batch of 4 scientific abstracts. It takes about 4.8 minutes.
Initialize GPT2Tokenizer elapsed time: 1.4727641880017472 seconds
Calculate initial encodings elapsed time: 0.03799170200363733 seconds
Initialize GPT2LMHeadModel elapsed time: 18.23160586800077 seconds

counter = 0
   Calculate GPT2 outputs elapsed time: 9.893233462003991 seconds
   Last token softmax elapsed time: 0.0007064949968480505 seconds
   Generate next token elapsed time: 0.0007856059964979067 seconds
   List comprehension calcs elapsed time: 0.13995413600059692 seconds
   Add generated tokens onto previous full strings elapsed time: 3.628500417107716e-05 seconds
   Tokenizing next full strings elapsed time: 0.012479234996135347 seconds

counter = 1
   Calculate GPT2 outputs elapsed time: 8.576971583999693 seconds
   Last token softmax elapsed time: 0.0006583049980690703 seconds
   Generate next token elapsed time: 0.0007301439982256852 seconds
   List comprehension calcs elapsed time: 0.12174680699536111 seconds
   Add generated tokens onto previous full strings elapsed time: 4.426100349519402e-05 seconds
   Tokenizing next full strings elapsed time: 0.011608965003688354 seconds

counter = 2
   Calculate GPT2 outputs elapsed time: 8.446395003004 seconds
   Last token softmax elapsed time: 0.0006648650014540181 seconds
   Generate next token elapsed time: 0.0007324170001083985 seconds
   List comprehension calcs elapsed time: 0.14154231199790956 seconds
   Add generated tokens onto previous full strings elapsed time: 5.113699444336817e-05 seconds
   Tokenizing next full strings elapsed time: 0.014487013999314513 seconds

counter = 3
   Calculate GPT2 outputs elapsed time: 9.05919142899802 seconds
   Last token softmax elapsed time: 0.0007119319998309948 seconds
   Generate next token elapsed time: 0.0007321929952013306 seconds
   List comprehension calcs elapsed time: 0.1303824819988222 seconds
   Add generated tokens onto previous full strings elapsed time: 4.862200148636475e-05 seconds
   Tokenizing next full strings elapsed time: 0.012177805001556408 seconds

counter = 4
   Calculate GPT2 outputs elapsed time: 8.18601783199847 seconds
   Last token softmax elapsed time: 0.02430849100346677 seconds
   Generate next token elapsed time: 0.0007163990012486465 seconds
   List comprehension calcs elapsed time: 0.15182566899602534 seconds
   Add generated tokens onto previous full strings elapsed time: 4.859599721385166e-05 seconds
   Tokenizing next full strings elapsed time: 0.014472196999122389 seconds

counter = 5
   Calculate GPT2 outputs elapsed time: 8.919605226001295 seconds
   Last token softmax elapsed time: 0.0006643329979851842 seconds
   Generate next token elapsed time: 0.0007834899952285923 seconds
   List comprehension calcs elapsed time: 0.12023409699759213 seconds
   Add generated tokens onto previous full strings elapsed time: 5.939300172030926e-05 seconds
   Tokenizing next full strings elapsed time: 0.011963372999161948 seconds

counter = 6
   Calculate GPT2 outputs elapsed time: 7.393029450999165 seconds
   Last token softmax elapsed time: 0.0007156190040404908 seconds
   Generate next token elapsed time: 0.0007048089973977767 seconds
   List comprehension calcs elapsed time: 0.13225654000416398 seconds
   Add generated tokens onto previous full strings elapsed time: 4.454200097825378e-05 seconds
   Tokenizing next full strings elapsed time: 0.012416008998116013 seconds

counter = 7
   Calculate GPT2 outputs elapsed time: 7.838971406003111 seconds
   Last token softmax elapsed time: 0.0023600620042998344 seconds
   Generate next token elapsed time: 0.0012576709996210411 seconds
   List comprehension calcs elapsed time: 0.1481078790020547 seconds
   Add generated tokens onto previous full strings elapsed time: 5.014600174035877e-05 seconds
   Tokenizing next full strings elapsed time: 0.017490088001068216 seconds

counter = 8
   Calculate GPT2 outputs elapsed time: 9.155262512998888 seconds
   Last token softmax elapsed time: 0.0010118979989783838 seconds
   Generate next token elapsed time: 0.000767368997912854 seconds
   List comprehension calcs elapsed time: 0.1483391650035628 seconds
   Add generated tokens onto previous full strings elapsed time: 9.162300557363778e-05 seconds
   Tokenizing next full strings elapsed time: 0.012755158000800293 seconds

counter = 9
   Calculate GPT2 outputs elapsed time: 10.77434083299886 seconds
   Last token softmax elapsed time: 0.010747615997388493 seconds
   Generate next token elapsed time: 0.0007793960030539893 seconds
   List comprehension calcs elapsed time: 0.14875024900538847 seconds
   Add generated tokens onto previous full strings elapsed time: 5.4737000027671456e-05 seconds
   Tokenizing next full strings elapsed time: 0.01669158499862533 seconds

counter = 10
   Calculate GPT2 outputs elapsed time: 10.110196126996016 seconds
   Last token softmax elapsed time: 0.02143118399544619 seconds
   Generate next token elapsed time: 0.0010516420006752014 seconds
   List comprehension calcs elapsed time: 0.1679224549952778 seconds
   Add generated tokens onto previous full strings elapsed time: 8.338599582202733e-05 seconds
   Tokenizing next full strings elapsed time: 0.014697657999931835 seconds

counter = 11
   Calculate GPT2 outputs elapsed time: 9.811458320000384 seconds
   Last token softmax elapsed time: 0.000684321996232029 seconds
   Generate next token elapsed time: 0.0007668550024391152 seconds
   List comprehension calcs elapsed time: 0.1525734469978488 seconds
   Add generated tokens onto previous full strings elapsed time: 8.502999844495207e-05 seconds
   Tokenizing next full strings elapsed time: 0.015574641001876444 seconds

counter = 12
   Calculate GPT2 outputs elapsed time: 10.353367308998713 seconds
   Last token softmax elapsed time: 0.0010184349957853556 seconds
   Generate next token elapsed time: 0.0007333970061154105 seconds
   List comprehension calcs elapsed time: 0.13797010699636303 seconds
   Add generated tokens onto previous full strings elapsed time: 5.920600233366713e-05 seconds
   Tokenizing next full strings elapsed time: 0.01412803399580298 seconds

counter = 13
   Calculate GPT2 outputs elapsed time: 10.826486637000926 seconds
   Last token softmax elapsed time: 0.0030568980000680313 seconds
   Generate next token elapsed time: 0.0016750930008129217 seconds
   List comprehension calcs elapsed time: 0.13814785299473442 seconds
   Add generated tokens onto previous full strings elapsed time: 5.037700611865148e-05 seconds
   Tokenizing next full strings elapsed time: 0.013047558997641318 seconds

counter = 14
   Calculate GPT2 outputs elapsed time: 7.55167671199888 seconds
   Last token softmax elapsed time: 0.0005718500033253804 seconds
   Generate next token elapsed time: 0.0007639390023541637 seconds
   List comprehension calcs elapsed time: 0.13264076500490773 seconds
   Add generated tokens onto previous full strings elapsed time: 5.1554998208303005e-05 seconds
   Tokenizing next full strings elapsed time: 0.013588694004283752 seconds

counter = 15
   Calculate GPT2 outputs elapsed time: 7.818915773998015 seconds
   Last token softmax elapsed time: 0.0005817519995616749 seconds
   Generate next token elapsed time: 0.0009528160007903352 seconds
   List comprehension calcs elapsed time: 0.13134208500559907 seconds
   Add generated tokens onto previous full strings elapsed time: 5.866299761692062e-05 seconds
   Tokenizing next full strings elapsed time: 0.01613930299936328 seconds

counter = 16
   Calculate GPT2 outputs elapsed time: 8.578778709998005 seconds
   Last token softmax elapsed time: 0.006186169004649855 seconds
   Generate next token elapsed time: 0.0007223930006148294 seconds
   List comprehension calcs elapsed time: 0.148166503997345 seconds
   Add generated tokens onto previous full strings elapsed time: 5.607099592452869e-05 seconds
   Tokenizing next full strings elapsed time: 0.015603724998072721 seconds

counter = 17
   Calculate GPT2 outputs elapsed time: 8.770265252001991 seconds
   Last token softmax elapsed time: 0.0005352449952624738 seconds
   Generate next token elapsed time: 0.0006981940023251809 seconds
   List comprehension calcs elapsed time: 0.13802178599871695 seconds
   Add generated tokens onto previous full strings elapsed time: 4.810000245925039e-05 seconds
   Tokenizing next full strings elapsed time: 0.013090499996906146 seconds

counter = 18
   Calculate GPT2 outputs elapsed time: 9.192157427001803 seconds
   Last token softmax elapsed time: 0.0013120759977027774 seconds
   Generate next token elapsed time: 0.0006921610038261861 seconds
   List comprehension calcs elapsed time: 0.1327957250032341 seconds
   Add generated tokens onto previous full strings elapsed time: 4.9251000746153295e-05 seconds
   Tokenizing next full strings elapsed time: 0.013492570993548725 seconds

counter = 19
   Calculate GPT2 outputs elapsed time: 8.60694089000026 seconds
   Last token softmax elapsed time: 0.003325685000163503 seconds
   Generate next token elapsed time: 0.000762072995712515 seconds
   List comprehension calcs elapsed time: 0.1455982879997464 seconds
   Add generated tokens onto previous full strings elapsed time: 5.135399987921119e-05 seconds
   Tokenizing next full strings elapsed time: 0.01226243400014937 seconds

counter = 20
   Calculate GPT2 outputs elapsed time: 7.204778202001762 seconds
   Last token softmax elapsed time: 0.000613894997513853 seconds
   Generate next token elapsed time: 0.0007320740041905083 seconds
   List comprehension calcs elapsed time: 0.13817419400584185 seconds
   Add generated tokens onto previous full strings elapsed time: 5.026099825045094e-05 seconds
   Tokenizing next full strings elapsed time: 0.011972155996772926 seconds

counter = 21
   Calculate GPT2 outputs elapsed time: 9.892961628000194 seconds
   Last token softmax elapsed time: 0.0006187749968376011 seconds
   Generate next token elapsed time: 0.0007513390009989962 seconds
   List comprehension calcs elapsed time: 0.12703227200108813 seconds
   Add generated tokens onto previous full strings elapsed time: 5.13460036017932e-05 seconds
   Tokenizing next full strings elapsed time: 0.012355157996353228 seconds

counter = 22
   Calculate GPT2 outputs elapsed time: 8.470063239998126 seconds
   Last token softmax elapsed time: 0.0005823390019941144 seconds
   Generate next token elapsed time: 0.0008517509995726869 seconds
   List comprehension calcs elapsed time: 0.15461847899860004 seconds
   Add generated tokens onto previous full strings elapsed time: 5.156599945621565e-05 seconds
   Tokenizing next full strings elapsed time: 0.013039129000389948 seconds

counter = 23
   Calculate GPT2 outputs elapsed time: 7.189594238996506 seconds
   Last token softmax elapsed time: 0.0006244809992494993 seconds
   Generate next token elapsed time: 0.0008493330024066381 seconds
   List comprehension calcs elapsed time: 0.13466514900210313 seconds
   Add generated tokens onto previous full strings elapsed time: 5.236799916019663e-05 seconds
   Tokenizing next full strings elapsed time: 0.012745897998684086 seconds

counter = 24
   Calculate GPT2 outputs elapsed time: 8.126932711005793 seconds
   Last token softmax elapsed time: 0.0006478180002886802 seconds
   Generate next token elapsed time: 0.0011431679959059693 seconds
   List comprehension calcs elapsed time: 0.14799942199897487 seconds
   Add generated tokens onto previous full strings elapsed time: 6.185499660205096e-05 seconds
   Tokenizing next full strings elapsed time: 0.01267693200497888 seconds

counter = 25
   Calculate GPT2 outputs elapsed time: 7.3953299740023795 seconds
   Last token softmax elapsed time: 0.0005992939986754209 seconds
   Generate next token elapsed time: 0.0007862490019761026 seconds
   List comprehension calcs elapsed time: 0.14582869299920276 seconds
   Add generated tokens onto previous full strings elapsed time: 5.401299858931452e-05 seconds
   Tokenizing next full strings elapsed time: 0.013731771003222093 seconds

counter = 26
   Calculate GPT2 outputs elapsed time: 8.018481942002836 seconds
   Last token softmax elapsed time: 0.04562465199705912 seconds
   Generate next token elapsed time: 0.0008113009971566498 seconds
   List comprehension calcs elapsed time: 0.13469937299669255 seconds
   Add generated tokens onto previous full strings elapsed time: 5.5969001550693065e-05 seconds
   Tokenizing next full strings elapsed time: 0.013372118999541271 seconds

counter = 27
   Calculate GPT2 outputs elapsed time: 8.428962316000252 seconds
   Last token softmax elapsed time: 0.0005464290006784722 seconds
   Generate next token elapsed time: 0.0007245989982038736 seconds
   List comprehension calcs elapsed time: 0.13549309900554363 seconds
   Add generated tokens onto previous full strings elapsed time: 4.968699795426801e-05 seconds
   Tokenizing next full strings elapsed time: 0.013636056995892432 seconds

counter = 28
   Calculate GPT2 outputs elapsed time: 7.557854796999891 seconds
   Last token softmax elapsed time: 0.0006319280000752769 seconds
   Generate next token elapsed time: 0.0007067559999995865 seconds
   List comprehension calcs elapsed time: 0.14673257899994496 seconds
   Add generated tokens onto previous full strings elapsed time: 5.2330004109535366e-05 seconds
   Tokenizing next full strings elapsed time: 0.012924063004902564 seconds

counter = 29
   Calculate GPT2 outputs elapsed time: 7.523497490998125 seconds
   Last token softmax elapsed time: 0.0005815810000058264 seconds
   Generate next token elapsed time: 0.0008171070003299974 seconds
   List comprehension calcs elapsed time: 0.14285911399929319 seconds
   Add generated tokens onto previous full strings elapsed time: 5.161000444786623e-05 seconds

Time to complete while loop for 30 passes: 269.7878909170031 seconds
------------------------------------------------------------------------------------
Total time elapsed time: 289.6417858999994 seconds

Thanks in advance for your help!

Context:
decoding is slow for auto-regressive decoder because the tokens are generated one at a time. I can think of only two things to improve speed for generation.

put the model in fp16 (not sure if GPT-2 works with fp6, haven’t tried myself).
See if you can use onnx to speed up inference. Maybe this 63 will help.


Fine-Tune BART using “Fine-Tuning Custom Datasets” doc

Question:
I am trying to fine-tune BART for a summarization task using the code on the “Fine Tuning with Custom Dataset” page (https://huggingface.co/transformers/custom_datasets.html 210). The data is a subset of the CNN/Daily Mail data.
I am encountering two different errors. The first comes when I implement the exact code on the page:  "TypeError: new(): invalid data type ‘str’
I assume this is caused by the fact that the labels are not encoded/tokenized, instead they are strings.  I see this is the case in the sample code on the page, the labels are not tokenized.
If I tokenize the labels and run the code I receive a different error:  “Indexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers”
I am not sure how to interpret that.  I get the same errors whether I am using the Trainer class or using native PyTorch.  Any suggestions?  Thanks!

Context: It’s a bit hard to help you without seeing the code you run.


How to know if gpu memory is enough before starting training?

Question: Are there any available code to be used directly?

Context: There is this: https://huggingface.co/transformers/benchmarks.html 16, which you can use to test memory usage amongst other things.


Question:
Hi,
I just saw that I have still embeddings of padding tokens in my sentence. I assumes that the BERT output would be a 768 dim 0 vector.
So if I feed sentences with max length of 20 into TFBert-Model I get for every of the 20 tokens an embedding different from 0 despite if the sentences is just of length 10 and the rest is padded.
How can I get output of BERT which ignores the padded output?
I thought that is why I feed the attention masks such paddings are ignored?
Maybe I misunderstand something?
If I feed the sequence of hidden states of the output of BERT to a GlobalAveragePooling-Layer, do I need the masking tensor to avoid using padding embeddings?

Context:
Hey,
I think I should use the attention_masking from the tokenizer for use in GlobalAveragePooling1d(x, mask=mask) , right?
Otherwise, the average of all token (padded tokens included) is built, right?


How can we test Transformer Models after converting it to TFLite format

Question:
I tried converting the Distilbert Model of Huggingface to TFLite format using this script Reference Script 44
I was able to convert it. After that, I wanted to test it in Python itself using tf.lite.Interpreter but I was not able to figure out the correct input dimension required for the model.  Can anyone help me with this?
Colab Notebook 23

Context: Pinging @jplu,  as he’s working on TF


Mutli GPU freezes on Roberta Pretraining

Question: I’m getting annoying crashes when I try to train a roberta model with two Titan X GPUs. I see in the documentation that the model should train on mutli gpu automatically and I see that with nvidia-smi that the gpus are in use. But I don’t see any progress and the session freezes. Any suggestions would be most helpful.

Context:
What do you mean by freeze? What do you see in the terminal?
Also show us which command you used to train the model.


Question:
Hey, I want to load the  cnn-dailymail dataset for fine-tune.
I write the code like this
from datasets import load_dataset
test_dataset = load_dataset(“cnn_dailymail”, “3.0.0”, split=“train”)
And I got the following errors.
Traceback (most recent call last):
File “test.py”, line 7, in
test_dataset = load_dataset(“cnn_dailymail”, “3.0.0”, split=“test”)
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\load.py”, line 589, in load_dataset
module_path, hash = prepare_module(
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\load.py”, line 268, in prepare_module
local_path = cached_path(file_path, download_config=download_config)
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\utils\file_utils.py”, line 300, in cached_path
output_path = get_from_cache(
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\utils\file_utils.py”, line 475, in get_from_cache
raise ConnectionError(“Couldn’t reach {}”.format(url))
ConnectionError: Couldn’t reach https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/cnn_dailymail/cnn_dailymail.py 8
How can I fix this ?

Context:
I can browse https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/cnn_dailymail/cnn_dailymail.py 47
through Google Chrome


The difference between Seq2SeqDataset.collate_fn and Seq2SeqDataCollator._encode

Question:
Hello,
I’m yusukemori, who asked about “Seq2SeqTrainer” yesterday ( How to use Seq2seq Trainer with my original "[MASK]" 1 ).
Thanks to the helpful comment, I’m now trying implementing my customized version of Seq2seqTrainer.
Now I have a question about Seq2SeqDataset and Seq2SeqDataCollator in examples/seq2seq/utils.py 3 (the part of it is shown below).
It seems Seq2SeqDataset has its collate_fn as a method.
However, Seq2SeqDataCollator doesn’t use Seq2SeqDataset.collate_fn, but has its own method _encode.
Could you please tell me what’s the difference between these two methods, and when to use each of them?
Or, should I use both of them to run Seq2SeqTrainer?
Thank you in advance.
yusukemori
class Seq2SeqDataset(AbstractSeq2SeqDataset):
    """A dataset that calls prepare_seq2seq_batch."""

    def __getitem__(self, index) -> Dict[str, str]:
        index = index + 1  # linecache starts at 1
        source_line = self.prefix + linecache.getline(str(self.src_file), index).rstrip("\n")
        tgt_line = linecache.getline(str(self.tgt_file), index).rstrip("\n")
        assert source_line, f"empty source line for index {index}"
        assert tgt_line, f"empty tgt line for index {index}"
        return {"tgt_texts": tgt_line, "src_texts": source_line, "id": index - 1}

    def collate_fn(self, batch) -> Dict[str, torch.Tensor]:
        """Call prepare_seq2seq_batch."""
        batch_encoding: Dict[str, torch.Tensor] = self.tokenizer.prepare_seq2seq_batch(
            [x["src_texts"] for x in batch],
            tgt_texts=[x["tgt_texts"] for x in batch],
            max_length=self.max_source_length,
            max_target_length=self.max_target_length,
            return_tensors="pt",
            **self.dataset_kwargs,
        ).data
        batch_encoding["ids"] = torch.tensor([x["id"] for x in batch])
        return batch_encoding

class Seq2SeqDataCollator:
    def __init__(self, tokenizer, data_args, tpu_num_cores=None):
        self.tokenizer = tokenizer
        self.pad_token_id = tokenizer.pad_token_id
        assert (
            self.pad_token_id is not None
        ), f"pad_token_id is not defined for ({self.tokenizer.__class__.__name__}), it must be defined."
        self.data_args = data_args
        self.tpu_num_cores = tpu_num_cores
        self.dataset_kwargs = {"add_prefix_space": isinstance(tokenizer, BartTokenizer)}
        if data_args.src_lang is not None:
            self.dataset_kwargs["src_lang"] = data_args.src_lang
        if data_args.tgt_lang is not None:
            self.dataset_kwargs["tgt_lang"] = data_args.tgt_lang

    def __call__(self, batch) -> Dict[str, torch.Tensor]:
        if hasattr(self.tokenizer, "prepare_seq2seq_batch"):
            batch = self._encode(batch)
            input_ids, attention_mask, labels = (
                batch["input_ids"],
                batch["attention_mask"],
                batch["labels"],
            )
        else:
            input_ids = torch.stack([x["input_ids"] for x in batch])
            attention_mask = torch.stack([x["attention_mask"] for x in batch])
            labels = torch.stack([x["labels"] for x in batch])

            labels = trim_batch(labels, self.pad_token_id)
            input_ids, attention_mask = trim_batch(input_ids, self.pad_token_id, attention_mask=attention_mask)

        if isinstance(self.tokenizer, T5Tokenizer):
            decoder_input_ids = self._shift_right_t5(labels)
        else:
            decoder_input_ids = shift_tokens_right(labels, self.pad_token_id)

        batch = {
            "input_ids": input_ids,
            "attention_mask": attention_mask,
            "decoder_input_ids": decoder_input_ids,
            "labels": labels,
        }
        return batch

    def _shift_right_t5(self, input_ids):
        # shift inputs to the right
        shifted_input_ids = input_ids.new_zeros(input_ids.shape)
        shifted_input_ids[..., 1:] = input_ids[..., :-1].clone()
        shifted_input_ids[..., 0] = self.pad_token_id
        return shifted_input_ids

    def _encode(self, batch) -> Dict[str, torch.Tensor]:
        batch_encoding = self.tokenizer.prepare_seq2seq_batch(
            [x["src_texts"] for x in batch],
            tgt_texts=[x["tgt_texts"] for x in batch],
            max_length=self.data_args.max_source_length,
            max_target_length=self.data_args.max_target_length,
            padding="max_length" if self.tpu_num_cores is not None else "longest",  # TPU hack
            return_tensors="pt",
            **self.dataset_kwargs,
        )
        return batch_encoding.data

Context:
Hi @yusukemori
The _encode method does the same work as collate_fn.
The difference is that Seq2SeqTrainer also supports TPU and for that the padding needs to handled differently. It also prepares the correct labels and decoder_input_ids rather than doing this inside the trainer.
If you are using Seq2SeqTrainer , use Seq2SeqDataCollator.


How to use Seq2seq Trainer with my original “[MASK]”

Question:
Hello,
My account name is yusukemori.
(I’m who asked about Seq2seq Trainer at https://github.com/huggingface/transformers/issues/7740 8 .
Thank you for giving me a quick, kind, and detailed reply then!)
This is my first time posting on this forum and I apologize if I’m being rude.
I’m now trying to find how to use my original “[mask]” for pre-training/fine-tuning the Seq2seq model.
I’m wondering how to randomly assign [mask] at training time (per epoch) and how to get Seq2seq Trainer to load it.
As far as BERT is concerned, there seems to be a BERTProcessor in huggingface/tokenizers, which is not intended to be used with the Seq2seq Trainer, if I understand correctly.
Is there any processer, such as “BARTProcessor”, for the Seq2seq Trainer?
Please allow me to ask one more question, is there a code for pre-training (from scratch) BART with the Seq2seq Trainer? (For fine-tuning, thanks to the clear example!)
I’m afraid this is a beginner’s, rude question.
Thank you for your help.
Sincerely,
yusukemori

Context:
Hi @yusukemori
The Seq2SeqTrainer examples doesn’t (yet!) support pre-training, however you could use Seq2SeqTrainer with your custom data processing logic(masking etc) as it’s a generic s2s trainer.
But you’ll need to implement masking yourself.


Question:
Hello, can anyone please explain to me how is the truncation done for long sequences in Distilbart ?
Thank you very much ! This is a very important matter.

Context: max seq len for BART is 1024 tokens, the tokenizer truncates all the tokens beyond that limit


Question:
I have trained a BERT model using ktrain (wrapper that uses  huggingface transformers library) to recognize emotion on text, it works but it suffers from really slow inference. That makes my model not suitable for a production environment. I have done some research and it seems pruning could help.
The problem is that pruning is not a not a widely used technique and I can not find a simple enough example on Kaggle or Stack that could help me to understand how to use it. Can someone help?
I provide my working code below for reference. My question can be also found on stack https://stackoverflow.com/questions/64445784/how-to-apply-pruning-on-a-bert-model 3
import pandas as pd
import numpy as np
import preprocessor as p
import emoji
import re
import ktrain
from ktrain import text
from unidecode import unidecode
import nltk

#text preprocessing class
class TextPreprocessing:
    def __init__(self):
        p.set_options(p.OPT.MENTION, p.OPT.URL)

    def _punctuation(self,val):
        val = re.sub(r'[^\w\s]',' ',val)
        val = re.sub('_', ' ',val)
        return val

    def _whitespace(self,val):
        return " ".join(val.split())

    def _removenumbers(self,val):
        val = re.sub('[0-9]+', '', val)
        return val

    def _remove_unicode(self, text):
        text = unidecode(text).encode("ascii")
        text = str(text, "ascii")
        return text

    def _split_to_sentences(self, body_text):
        sentences = re.split(r"(?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?)\s", body_text)
        return sentences

    def _clean_text(self,val):
        val = val.lower()
        val = self._removenumbers(val)
        val = p.clean(val)
        val = ' '.join(self._punctuation(emoji.demojize(val)).split())
        val = self._remove_unicode(val)
        val = self._whitespace(val)
        return val

    def text_preprocessor(self, body_text):

        body_text_df = pd.DataFrame({"body_text": body_text},index=[1])

        sentence_split_df = body_text_df.copy()

        sentence_split_df["body_text"] = sentence_split_df["body_text"].apply(
            self._split_to_sentences)

        lst_col = "body_text"
        sentence_split_df = pd.DataFrame(
            {
                col: np.repeat(
                    sentence_split_df[col].values, sentence_split_df[lst_col].str.len(
                    )
                )
                for col in sentence_split_df.columns.drop(lst_col)
            }
        ).assign(**{lst_col: np.concatenate(sentence_split_df[lst_col].values)})[
            sentence_split_df.columns
        ]

        body_text_df["body_text"] = body_text_df["body_text"].apply(self._clean_text)

        final_df = (
            pd.concat([sentence_split_df, body_text_df])
            .reset_index()
            .drop(columns=["index"])
        )

        return final_df["body_text"]

#instantiate data preprocessing object
text1 = TextPreprocessing()

#import data
data_train = pd.read_csv('data_train_v5.csv', encoding='utf8', engine='python')
data_test = pd.read_csv('data_test_v5.csv', encoding='utf8', engine='python')

#clean the data
data_train['Text'] = data_train['Text'].apply(text1._clean_text)
data_test['Text'] = data_test['Text'].apply(text1._clean_text)

X_train = data_train.Text.tolist()
X_test = data_test.Text.tolist()

y_train = data_train.Emotion.tolist()
y_test = data_test.Emotion.tolist()

data = data_train.append(data_test, ignore_index=True)

class_names = ['joy','sadness','fear','anger','neutral']

encoding = {
    'joy': 0,
    'sadness': 1,
    'fear': 2,
    'anger': 3,
    'neutral': 4
}

# Integer values for each class
y_train = [encoding[x] for x in y_train]
y_test = [encoding[x] for x in y_test]

trn, val, preproc = text.texts_from_array(x_train=X_train, y_train=y_train,
                                                                       x_test=X_test, y_test=y_test,
                                                                       class_names=class_names,
                                                                       preprocess_mode='distilbert',
                                                                       maxlen=350)

model = text.text_classifier('distilbert', train_data=trn, preproc=preproc)

learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=6)

predictor = ktrain.get_predictor(learner.model, preproc)

#save the model on a file for later use
predictor.save("models/bert_model")

message = "This is a happy message"

#cleaning - takes 5ms to run
clean = text1._clean_text(message)

#prediction - takes 325 ms to run
predictor.predict_proba(clean)

Context:
I don’t know how to perform pruning.   The idea is simple enough - cut out some of the attention heads that are not apparently doing anything useful - but the implementation will be tricky.
Have you considered fine-tuning a DistilBERT or ALBERT model instead?


Question:
I am looking to train a Transformer model on biological sequence data where a “sentence” may be represented as follows: [“A”, “LLGR”, “V”, “GD” …]
In order to do so, I need a word-level tokenizer that doesn’t split up words, so I’ve opted for TransformerXL.
The train() function from ByteLevelBPETokenizer() (as described in this blog: https://huggingface.co/blog/how-to-train) is not available for TransfoXLTokenizer()
Is there a way to train TransfoXLTokenizer() on my custom “language”? Or do I simply train the TransfoXLModel and it’ll take care of custom tokenization? (I can’t see this being the case as its a completely unseen “language”)

Context:
No need to train.TransfoXLTokenizer() doesn’t merge words like ByteLevelBPETokenizer().
Simply create your vocab file as a txt file with each word on a new line, and feed it’s path as a parameter to TransfoXLTokenizer():
tokenizer = TransfoXLTokenizer(vocab_file=vocab_path)


Model giving same output for eval function but trains

Question:
Hi as title says my model is giving the exact same output for different inputs when evaluating but is training and giving varied outputs when training.
I can’t workout why I know it’s not that it’s the same input.
I am also ending up with holes in the outputs when there are no such holes within the input data.
both errors can be seen in the screenshout of my terminal output here:
My training loss is: 121,
whereas my RMS of my eval function is 23,000??
image708×360 18.8 KB
Data_set and datacollator:
**class smiles_dataset(Dataset):
    def __init__(self, smiles, targets):
        self.smiles = smiles
        self.targets = targets
        #self.tokenizer = tokenizer
        #self.max_len = max_len

    def __len__(self):
        return len(self.smiles)

    def __getitem__(self, index):
        smiles = str(self.smiles[index])
        targets = float(self.targets[index])

        return {
            'SMILES': smiles,
            'targets': targets,
        }

class SMILESDataCollator():
    def __init__(self, tokenizer):
        self.tokenizer = tokenizer

    def __call__(self, data):
        return self.__collate__(default_collate(data))

    def __collate__(self, data):
        # Not sure why padding it twice works here but it gives the desired output so can't complain
        input_id_sequence = []
        for d in data['SMILES']:
            input_ids = self.tokenizer(d)[0]
            input_id_sequence.append(input_ids)
        input_ids_padded = pad_sequence(input_id_sequence, batch_first=True, padding_value=0)
        # input_ids_padded = [pad_sequence((self.tokenizer(d)), padding_value=0) for d in data['SMILES']]
        # input_ids_padded = pad_sequence(input_ids_padded, padding_value=0)
        # for all the d in data
        target_list = []
        for d in data['targets']:
            target_list.append(d)
        atten_mask_list  = []
        for input_ids in input_ids_padded:
            atten_mask = torch.where(input_ids.eq(torch.zeros(1)), torch.tensor([0]), torch.tensor([1]))
            atten_mask_list.append(atten_mask)
        return {
            "input_ids": input_ids_padded,
            "attention_masks": atten_mask_list,
            "targets": target_list
        }

Eval function:
def eval_fn(data_loader, model, device, optimizer):
    model.eval()
    fin_targets = []
    fin_outputs = []
    with torch.no_grad():
        for batch_index, data_set in tqdm(enumerate(data_loader), total=len(data_loader)):
            ids = data_set['input_ids']
            mask = torch.stack(data_set['attention_masks'])
            targets = torch.stack(data_set['targets'])
            #token_type_ids = data_set['token_type_ids']

            ids = ids.to(device)
            mask = mask.to(device)
            targets = targets.to(device)
            #token_type_ids = token_type_ids.to(device, dtype=torch.long)

            optimizer.zero_grad()
            outputs = model(
                ids = ids,
                mask=mask,
                token_type_ids=None
            )
            # print(outputs)
            # print(targets)
            fin_outputs.extend(outputs.cpu().detach().numpy().tolist())
            fin_targets.extend(targets.cpu().detach().numpy().tolist())
    return fin_outputs, fin_targets

Model definition
config = BertConfig(
    vocab_size=5000,
    max_positional_embeddings=224,
    num_attention_heads=12,
    num_hidden_layers=46,
    type_vocab_size=1,
    num_labels=1,
    hidden_dropout_prob=0.1
)

loss = nn.L1Loss()

class regression_model(nn.Module):
    def __init__(self):
        super(regression_model, self).__init__()
        self.bert = BertForSequenceClassification(config=config)
        self.drop = nn.Dropout(p=0.3)
        #self.out = nn.Linear(self.bert.config.hidden_size, 1)

    def forward(self, ids, mask, token_type_ids):
        out = self.bert(input_ids=ids, attention_mask=mask, token_type_ids=token_type_ids)[0]
        out = self.drop(out)
        return out

let me know if you need to see anymore code. The model is using a tokenizer specifically made for the dataset and so I had to write the attention_mask part into the collator. It is being trained from scratch on this data as it’s very different to the standard language data.
Thanks
A

Context:
Hi, when you say “holes” do you mean zeros?   You could try increasing the precision of your values (maybe the numbers get so small pytorch thinks they are zero).
Are you sure you need 46 hidden layers?   Seems rather a lot.
What are you passing to your eval_fn?   How are you splitting your train/eval data?
Could you be overfitting to your training data?


Question:
Hi there,
I am not a native english speaker so please don´t blame me for the question. I am currently trying to figure out how I can fine-tune distilBART on some Financial Data (like finBERT). In the examples/seq2seq README it states:
For the CNN/DailyMail dataset, (relatively longer, more extractive summaries), we found a simple technique that works: you just copy alternating layers from bart-large-cnn and finetune more on the same data.
As far as I understand this sentence, I can Only finetune a distilBART student with the same data the teacher was trained with (CCN/DM) or can I use my own dataset that is completely different to the one that the BART Teacher was trained on?
Thanks in advance
Chris

Context:
You can finetune distilbart on any data you want, the question is how well different approaches will perform.
Without knowing much more about the data and assuming you want to be able to train in <24h, I would probably start from sshleifer/distilbart-cnn-12-3.


Question:
Hey,
if I fine tune a BERT model is the tokneizer somehow affected?
If I save my finetuned model like:
bert_model.save_pretrained(’./Fine_tune_BERT/’)
is the tokenizer saved too? is the tokenizer modified? Do I need to save it too?
Bercause loading the tokenizer like:
tokenizer = BertTokenizer.from_pretrained(‘Fine_tune_BERT/’)
is giving error.
>
> OSError: Model name ‘Fine_tune_BERT/’ was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1, wietsedv/bert-base-dutch-cased). We assumed ‘Fine_tune_BERT/’ was a path, a model identifier, or url to a directory containing vocabulary files named [‘vocab.txt’] but couldn’t find such vocabulary files at this path or url.
SO I assume I can load the tokenizer in the normal way?

Context:
The model is independent from your tokenizer, so you need to also do:
tokenizer.save_pretrained(’./Fine_tune_BERT/’)

to be able to load it back with from_pretrained.


How to extract the “student” model after distillation?

Question:
Hey there!
I am using your distillation script 9 (thanks for sharing it!) and based on the dumped checkpoints I see, it seems that they contain both the teacher and the student.
Assuming that my observation is correct, how can I dump only the student sub-model?
@sshleifer wondering if you have any thoughts.

Context:
Great Q!
the saved best_tfmr directory has only student the student saved (in huggingface format).
There is also a pytorch lightning weights_only checkpoint you could pass to ModelCheckpoint here. Be aware that this might break --do_predict/trainer.test, which you can overcome by running eval as a second step, roughly:
# Define useful aliases
run_distributed_eval () {
	proc=$1
	m=$2
	dd=$3
	sd=$4
	shift
	shift
	shift
	shift
	python -m torch.distributed.launch --nproc_per_node=$proc  run_distributed_eval.py \
		--model_name $m  --save_dir $sd --data_dir $dd $@
}
eval_best () {
	proc=$1
	m=$2
	dd=$3
	shift
	shift
	shift
	run_distributed_eval $proc $m/best_tfmr $dd $m/ $@
}

Finally, run:
eval_best 1 output_dir

(if you have more gpus, change the first arg)


What is the proper way to do inference using fine-tuned model?

Question:
Hello.
Once the model is fine-tuned, what is the proper way to do inference?
E.g. I would send list of words to API (e.g. Flask) and predict labels for each word using fine-tuned model.
Should I just run model using do_pretict = True, and do_eval, do_test = False?
Or there is a better way?
Thanks!

Context: this summary of tasks 11 doc shows how you can do inference for various tasks.