noamaanMulla-03/transformers-issues huggingface dataset:
Failure to load checkpoints saved during distributed training
Author: glnmario
Pull request: no
Labels: name: wontfix

name: Ex: LM (Finetuning)

name: Ex: LM (Pretraining)

name: Core: Modeling

# 🐛 Bug

## Information

Model I am using: Bert

Language I am using the model on: English, German, Swedish

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py) using distributed training: e.g. 
`python3 -m torch.distributed.launch --nproc_per_node=4 --nnodes=1 --node_rank=0 run_language_modeling.py` --output_dir output_dir [--args...]
2. When the training is over, try to load the final model: BertModel.from_pretrained('output_dir'): this works.
3. Then, try to load a checkpoint: e.g., `BertModel.from_pretrained('output_dir/checkpoint-1000')`: this gives a runtime error:
`
Traceback (most recent call last):
  File "/cluster/shared/nlpl/software/modules/in5550/202002/lib/python3.7/site-packages/transformers/modeling_utils.py", line 470, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location="cpu")
  File "/cluster/shared/nlpl/software/modules/pytorch/1.4.0/lib/python3.7/site-packages/torch/serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/cluster/shared/nlpl/software/modules/pytorch/1.4.0/lib/python3.7/site-packages/torch/serialization.py", line 709, in _legacy_load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: storage has wrong size: expected 4434893008627221919 got 2359296
`

## Expected behavior
We should be able to load from the checkpoints also when the training is distributed.

## Suggested solution
Check `if args.local_rank == -1 or torch.distributed.get_rank() == 0` on line 370 (just like on line 736).

## Environment info
- `transformers` version: 2.5.0
- Platform: UNIX
- Python version: Python 3.5.3
- PyTorch version (GPU?): 1.4.0 
- Tensorflow version (GPU?): No
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Distributed (1 node, 4 GPUs)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
minor website fix
Author: nbroad1881
Pull request: no
Labels: name: wontfix

name: Should Fix

On this page [https://huggingface.co/transformers/notebooks.html](https://huggingface.co/transformers/notebooks.html)

The first link is fine. The others give 404. Just thought you would like to know

Comments:
Indeed, those notebooks are not up-to-date and should be deprecated.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
how to finetune with PreTrainedEncoderDecoder
Author: vanh17
Pull request: no
Labels: name: wontfix

name: Core: Encoder-Decoder

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue --> I am trying to run seq2seq within the same language (say it is English). I was trying to use PreTrainedEncoderDecoder (BERT, BERT), was trying to use BERT and GPT2 however, looks like it does not support this combination yet. 
I am trying to understand what the forward function does in the class PreTrainedEncoderDecoder, and how can we use it training my dataset.
Also how can we use it at prediction time since forward need to have both decode_input_ids and encode_input_ids. I do not think we will have decode_input_ids at prediction time. Thank you

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
very slow performance on transformer 2.5.0 versus 2.3.0
Author: yes1234man
Pull request: no
Labels: name: wontfix

Hi
I am running the run_glue version with last version of transformer 2.5.0 python=3.5 and this is at least 10 times slower than running the same code with transformer 2.3.0 with python 3.6.9. I use the BERT model. The difference between speed of these versions is extremely high, could you have a look and check why the performance of latest version of the transformer is very low. thank you. 

Comments:
I met the same issue when running the run_squad.py using the 2.5.0 version. 

Which PyTorch version are you using for your benchmarks? Is it the same for both, or is it different?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
What is the most effective way to use BERT , ROBERTA , GPT-2 architectures as frozen feature extractors ?
Author: shamanez
Pull request: no
Labels: name: Discussion

name: wontfix

We use pretrained self-supervised learning (SSL) models for NLP as feature extractors for downstream tasks like sentiment analysis. In most of such cases, we add a simple new classification layer and **fine-tune the whole model**.  With the SSL models getting bigger and the amount of unsupervised training data is huge it would be nice if we can use the problem agnostic behavior of SSL embeddings. In other words if we use them as **Frozen Feature extractors**, we can save lot of time and computational cost. 

**Have anyone seen a good review on using SSL networks as frozen feature extractors?**  

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
TextClassificationPipeline does not work with pretrained BERT model
Author: ogencoglu
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): 'nlptown/bert-base-multilingual-uncased-sentiment'

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
model = BertModel.from_pretrained(pretrained_model_name_or_path='nlptown/bert-base-multilingual-uncased-sentiment')
tokenizer = BertTokenizer.from_pretrained(pretrained_model_name_or_path='nlptown/bert-base-multilingual-uncased-sentiment')
sentiment_analyzer = TextClassificationPipeline(model=model, tokenizer=tokenizer)
sentiment_analyzer('This is awesome!')

/usr/local/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    504     def __call__(self, *args, **kwargs):
    505         outputs = super().__call__(*args, **kwargs)
--> 506         scores = np.exp(outputs) / np.exp(outputs).sum(-1)
    507         return [{"label": self.model.config.id2label[item.argmax()], "score": item.max()} for item in scores]
    508 

ValueError: operands could not be broadcast together with shapes (1,8,768) (1,8) 
```

## Expected behavior

A sentiment score.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Ubuntu 16.04
- Python version: 3.7
- PyTorch version (GPU?): 1.4.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
That would be because you're using a `BertModel` instead of a `BertModelForSequenceClassification`.

What model should we be using, and where can we download it from?
Assert message error in Reformer chunking
Author: erickrf
Pull request: no
Labels: name: reformer

# 🐛 Bug

In the function `apply_chunking_to_forward`, an assertion checking the input tensor shape is trying to print the contents of the tensor itself instead of its shape:

https://github.com/huggingface/transformers/blob/b5015a2a0f4ea63035a877f5626cb0c3ce97e25d/src/transformers/modeling_utils.py#L2195

I'm pretty sure that line should be `input_tensors[0].shape[chunk_dim], chunk_size`

Comments:
Albert language model fine tuning not running run_lm_finetuning.py
Author: abdallah197
Pull request: no
Labels: name: wontfix

name: Ex: LM (Finetuning)

name: PyTorch

name: Usage

# ❓ Questions & Help

## Information

Model I am using (Albert(all types)):

Language I am using the model on (English):

The problem arises when using:
* [ ] the official example scripts: (give details below)
the code returns memory allocation problems when run with any version from albert. i tried to reduce the sequence length and batch size to a minum setting, but the issue still arises. my setting and the minimized setting both run normally with bert or roberta, the issue arises only when i change the model to Albert.
an example:
`tcmalloc: large alloc 1951195136 bytes == 0x7f750f664000 @  0x7f76efbf8887 0x7f764c2a1b79 0x7f764c29fb0f 0x7f764c29fc33 0x7f764c26a155 0x7f764c26837e 0x7f764c26bbb1 0x7f764c2606df 0x50a8af 0x50c5b9 0x509d48 0x50aa7d 0x50c5b9 0x508245 0x509642 0x595311 0x5a067e 0x50d966 0x58efc9 0x4c9546 0x5886f4 0x58892e 0x551b81 0x5aa6ec 0x50abb3 0x50c5b9 0x508245 0x50a080 0x50aa7d 0x50c5b9 0x508245`

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)
language model finetuning for albert
## To reproduce

Steps to reproduce the behavior:

1. in run_lm_finetuning add:
` from transformers import (AlbertConfig,
    AlbertForMaskedLM,
    AlbertTokenizer,
)`
2.add to MODEL_CLASSES dictionary:
`    "albert": (AlbertConfig, AlbertForMaskedLM, AlbertTokenizer),`
3. add file text.txt, a similar txt file to the wiki dataset that's mentioned in the docs. 
4.run the finetuning script:
`python transformers/examples/run_lm_finetuning.py \
    --output_dir=output \
    --model_type=albert \
    --model_name_or_path=albert-base-v1 \
    --do_train \
    --train_data_file test.txt \
    --block_size 50 \
    --per_gpu_train_batch_size 2 \
    --max_steps 520000 \
    --weight_decay 0.01 \
    --logging_steps 5000 \
    --mlm`



## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment

* OS: Google colab
* Python version: 3.7
* PyTorch version: 1.3.1
* `transformers` version (or branch): latest
* Using GPU ? yes
* Distributed or parallel setup ? no
* Any other relevant information:


Comments:
@thomwolf can you give any insights regarding this?

how much lines in `test.txt`?

1,041,130 line

I have a similar issue finetuning the language model with bert. In the end, I had to scale down my training to ~200,000 lines to make it work, which is a very small proportion of my original dataset.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
KeyError when loading a trained EncoderDecoder model
Author: gustavscholin
Pull request: no

# 🐛 Bug

## Information

Error when loading a trained EncoderDecoder model. When loading the config the in `configuration_auto.py` the `model_type` is expected on the form `encoder-decoder` but in   `configuration_encoder_decoder.py` `model_type` is on the form `encoder_decoder` which raises a KeyError.

The hyphen version seems to be convention in the other model configuration files.

I guess this is something for @patrickvonplaten 


Comments:
Hi @gustavscholin, 

Thanks for you issue!
Could you please provide a code example so that I can reproduce the error? 

Hi, @patrickvonplaten @gustavscholin 
For me, setting "base_model_prefix" in modeling_encoder_decoder.py fixed this problem, as finding params is based on self.base_model_prefix. 

Is it fundamental solution? or just short-sighted? 

@patrickvonplaten, here's a colab notebook to reproduce the error:

https://colab.research.google.com/drive/102U7pJJcyw__Yq0PERxAKvKPSx3bvNSi?usp=sharing

> Hi, @patrickvonplaten @gustavscholin
> For me, setting "base_model_prefix" in modeling_encoder_decoder.py fixed this problem, as finding params is based on self.base_model_prefix.
> 
> Is it fundamental solution? or just short-sighted?

No that was the right solution :-) I did exactly the same in this fix:  #4680

> @patrickvonplaten, here's a colab notebook to reproduce the error:
> 
> https://colab.research.google.com/drive/102U7pJJcyw__Yq0PERxAKvKPSx3bvNSi?usp=sharing

A saved encoder-decoder model will always only be saved in a single folder. A single folder can always be loaded with `.from_pretrained()`. So to make your notebook work, you simply have to replace this line:

```python 
saved_model = EncoderDecoderModel.from_encoder_decoder_pretrained('test_run', 'test_run')
```

by 

```python 
saved_model = EncoderDecoderModel.from_pretrained('test_run')
```
Distilgpt2 finetuning and text generation
Author: tripathiaakash
Pull request: yes

This ipynb notebook contains a finetuning and text generation tutorial for distilgpt2. The tutorial also have used code from run_generation.py file to make generation faster than using the original file for every iteration.

Comments:
Hi @BlackJack01 - thanks so much for this great contribution! We are still going through the notebook and discussing how to add it :-) Will let you know soon! 

Hi @BlackJack01, sorry for the late answer. 
We now have community notebooks here: https://github.com/huggingface/transformers/tree/master/notebooks#community-notebooks
Feel free to open a PR to add it there :-)
training GPT2 from scratch : implement causal attention mask?
Author: CNelias
Pull request: no

# ❓ Questions & Help

I am trying to train a ```GPT2``` model from scratch but I noticed, by looking into the code here https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py that there doesn’t seem to be an implementation for a causal mask. Maybe it is in another repo and I missed it, I also couldn't find ressources on this in the docs.
I could write an ugly for loop and feed each of my sequences one token at a time to the network which would be super unefficient. I could also chop up each of my examples token by token, pad them and feed it like a batch, which is probably faster but doesn’t feel super satisfacting.

Do you know if there is a standard implementation of casal mask that I missed, or another way to do what I am describing ?

PS : I have already read huggingface’s blogpost on training from scratch, but unfortunately it doesn't say much about the implementation of said training :/.

Comments:
You'd want to look at the `run_language_modeling.py` script which implements causal language modeling. (do not pass the `--mlm` flag)

I'm thinking some edit to run_language_modeling.py script maybe would make it work. I don't think just to not pass the --mlm flag you solve the problem @julien-c. Have you found any solution @johncwok? I'm searching the same thing.

@johncwok GPT2 always uses a causal mask. It's quite hidden in the code. This line https://github.com/huggingface/transformers/blob/0a4b1068e1d6c46525082b91a4ba00a09c9270ac/src/transformers/modeling_gpt2.py#L145 creates the causal mask that is then applied to the weights. The naming can definitely be improved here! So no matter what mask you insert it will only be applied in combination with the causal mask.
Also take a look at this line that creates the mask:
https://github.com/huggingface/transformers/blob/0a4b1068e1d6c46525082b91a4ba00a09c9270ac/src/transformers/modeling_gpt2.py#L107


After https://github.com/huggingface/transformers/pull/2715/files is merged, I will do some renaming in the code - seems like a lot of people look for the causal mask in GPT2, CTRL and GPT
Text Generation with XLNet is very Slow
Author: urlocal12
Pull request: no
Labels: name: Ex: Generation

Using the run_generation script to generate text with XLNet is currently extremely slow compared to GPT-2 as mentioned in [this issue](https://github.com/huggingface/transformers/issues/789)

> To generate 100 tokens, XLNet takes **3m22s** while GPT-2 takes **14s**. And it grows exponentially : for 500 tokens, XLNet takes **51m46s** while GPT-2 takes **2m52s**.

More information on why this might be happening is included in the issue. However, it was closed before it could be resolved. If anyone could look into this and maybe reopen the original issue, that would be greatly appreciated. 
Thanks! 

Comments:
Reopened the previous issue: https://github.com/huggingface/transformers/issues/789 and will take a look next week :-) 

Closing this as well. Reasons are explained in #789.
Loading pipeline("summarization") failed
Author: asymmetric-supernova
Pull request: no

Hi guys, when I try to load pipeline("summarization") I get the following error:

`Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "PATH", line 11, in <module>
    summarizer = pipeline("summarization")
  File "PATH\venv\lib\site-packages\transformers\pipelines.py", line 1626, in pipeline
    return task_class(model=model, tokenizer=tokenizer, modelcard=modelcard, framework=framework, task=task, **kwargs,)
  File "PATH\venv\lib\site-packages\transformers\pipelines.py", line 367, in __init__
    task_specific_params = self.model.config.task_specific_params
AttributeError: 'NoneType' object has no attribute 'config'
`

**Setup:** 
Python: 3.7.6
transformers==2.8.0
tensorboard==2.0.2
tensorflow==2.0.0
tensorflow-estimator==2.0.1
tensorflow-hub==0.7.0

```
from transformers import pipeline
from transformers import TFAutoModelWithLMHead, AutoTokenizer

summarizer = pipeline("summarization")
```

Comments:
The default summarization pipeline doesn't have support for TF unfortunately, but we should probably add an explicit error message @sshleifer 

+1

You can use `T5's` TF summarization though, like:
`pipeline("summarization", model="t5-base", framework="tf")`

I think the error has nothing to do with "summarization" or "Bart". I think the problem is that you were calling a pytorch pipeline without having pytorch installed. If this happens a weird error message like the one above is thrown out. We should probably add a better error message here:
https://github.com/huggingface/transformers/blob/700ccf6e35616fcbee59de81edd60cec9e14fb6b/src/transformers/pipelines.py#L1567
Something along the lines: `if modelclass in None: <good_error_message>`

To run pipelines in TF, the argument `framework="tf"` should be added to `pipeline()`

I don't think that's right, @patrickvonplaten.

Pipelines use TF automatically if that's what you have instead of PyTorch: ie it does `framework = "pt" if is_torch_available() else "tf"`

However, as I was saying, the **default** (bart-based) summarization pipeline doesn't have a TF model, see line 1447:
```python
"default": {
            "model": {"pt": "bart-large-cnn", "tf": None},
}
```



Sorry, you are 100 % right @julien-c! 

I overlooked this line:
https://github.com/huggingface/transformers/blob/700ccf6e35616fcbee59de81edd60cec9e14fb6b/src/transformers/pipelines.py#L1564

So, we do need a better error message for pipelines that have a `None` as a default model.

@julien-c works fine with pytorch, thanks

Thanks, works for me as well!
As mentioned before, the improved error message would help a lot.

Do you want to take a stab at a better error message for this @patrickvonplaten?
Faster mask computation 
Author: rasoolims
Pull request: no

# 🚀 Feature request

Currently, the masking is done using the full prediction matrix which is both memory and computation inefficient. One example is in [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_albert.py#L675). I think with Pytorch indexing, first and foremost, we don't need to do full mask matrix construction. Second, it can be much faster. Fairseq does a similar thing in [here](https://github.com/pytorch/fairseq/blob/cce6dcb1cca85955a82879ea5064fe8202e8f412/fairseq/models/roberta/model.py#L217)

## Motivation

If input is [n,m], currently the code creates a clone of [n,m] mask where non-masked inputs are -100. Then it does a full output projection: if input in [n, m] and hidden representation is [n, m, h], the final output will be a huge [n, m, v] if v is the vocabulary size. Instead we can think of mask indices of size k<< n* m, and thus the we can extract a [k, h] submatrix from [n, m, h], then can have a smaller output result [k, v].

## Your contribution
This a sample code (I extracted from a bigger code of mine; sorry if some variables are not defined in the snippet):
```py
mask_prob = 0.15
mask = torch.empty(input_ids.size()).uniform_(0, 1) < mask_prob
mask[pads] = False  # We should not mask pads.
masked_ids = input_ids[mask]
replacements = masked_ids.clone()
for i in range(len(replacements)):
    r = random.random()
    if r < 0.8:
        replacements[i] = mask_id
    elif r < 0.9:
        # Replace with another random word.
        random_index = random.randint(0, vocab_size - 1)
        replacements[i] = random_index
    else:
        # keep the word
        pass
input_ids[mask] = replacements

text_hidden, text_cls_head = albert_model(texts, attention_mask=pads)
masked_hidden_state = text_hidden[mask]
output_predictions = F.log_softmax(albertMLMHead(masked_hidden_state), dim=1)
```


Comments:
Hi @rasoolims ,

That sounds reasonable what you are saying! I wanted to take a look into masking optimization in a couple of weeks. If you feel like it, you could also open a PR and we take a look together :-) 

Feel free to open a PR for this :-) Closing for now