noamaanMulla-03/transformers-issues huggingface dataset:
Update the NER TF script
Author: jplu
Pull request: yes

PR to remove the softmax and make the pad token label id to -1.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3511?src=pr&el=h1) Report
> Merging [#3511](https://codecov.io/gh/huggingface/transformers/pull/3511?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/601ac5b1dc1438f00d09696588f2deb0f045ae3b&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3511/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3511?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3511      +/-   ##
==========================================
+ Coverage   77.79%   77.80%   +0.01%     
==========================================
  Files         100      100              
  Lines       17051    17051              
==========================================
+ Hits        13265    13267       +2     
+ Misses       3786     3784       -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3511?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/optimization\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/3511/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb25fdGYucHk=) | `79.27% <100.00%> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3511/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.94% <0.00%> (+0.13%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3511/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.32% <0.00%> (+0.17%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3511?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3511?src=pr&el=footer). Last update [601ac5b...572e04d](https://codecov.io/gh/huggingface/transformers/pull/3511?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Error importing Reformer model
Author: bayomim
Pull request: no

# 🐛 Bug

## Information

I am trying to import the Reformer model as in here:
https://huggingface.co/transformers/model_doc/reformer.html
'''
from transformers import ReformerModel, ReformerConfig
'''
However, I am getting the following error:
ImportError: cannot import name 'ReformerModel' from 'transformers'

The problem arises when using:
* [ √] the official example scripts: (give details below)
* [ X ] my own modified scripts: (give details below)

- `transformers` version: 2.8.0


Comments:
I think you only need to update the library.
Version 2.9, released esterday, contains the reformer model

Thanks, 
Installing Version 2.9.0 solved the issue
how i pass from an AutoModelWithLMHead to a GPT2DoubleHeadsModel ?
Author: nikkon3
Pull request: no

I used your script of run_language_modeling.py to train a gpt2 model from scratch. Here i save it and upload:

from transformers import AutoModelWithLMHead, AutoTokenizer
import os
directory = "./checkpoint"
model = AutoModelWithLMHead.from_pretrained(directory)
tokenizer = AutoTokenizer.from_pretrained(directory)
out = "gpt_2"
os.makedirs(out, exist_ok=True)
model.save_pretrained(out)
tokenizer.save_pretrained(out)
!transformers-cli upload ./gpt_2/

After if i want to use it for fine-tuning in a specific task and i want to use a GPT2DoubleHeadsModel
i can just load it like this?

from transformers import GPT2DoubleHeadsModel,GPT2Tokenizer
model = GPT2DoubleHeadsModel.from_pretrained("ex/gpt_2")
tokenizer = GPT2Tokenizer.from_pretrained("ex/gpt_2")

Or it will not load the weights?

Comments:
It will load the weights, but will not load the multiple choice head as it's not in the checkpoint. That's why fine-tuning it is a good idea, so that this head's weights get trained!
No name 'AlbertMLMHead' in module 'transformers'
Author: zzj0402
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

MLM model head
```
No name 'AlbertMLMHead' in module 'transformers'
```

## Open source status

* [x] the model implementation is available: 
```
class AlbertMLMHead(nn.Module):
    def __init__(self, config):
        super().__init__()

        self.LayerNorm = nn.LayerNorm(config.embedding_size)
        self.bias = nn.Parameter(torch.zeros(config.vocab_size))
        self.dense = nn.Linear(config.hidden_size, config.embedding_size)
        self.decoder = nn.Linear(config.embedding_size, config.vocab_size)
        self.activation = ACT2FN[config.hidden_act]

        # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
        self.decoder.bias = self.bias

    def forward(self, hidden_states):
        hidden_states = self.dense(hidden_states)
        hidden_states = self.activation(hidden_states)
        hidden_states = self.LayerNorm(hidden_states)
        hidden_states = self.decoder(hidden_states)

        prediction_scores = hidden_states

        return prediction_scores


[DOCS]@add_start_docstrings(
    "Albert Model with a `language modeling` head on top.", ALBERT_START_DOCSTRING,
)
```
Include in the __initial__ file and make it available for use.

* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)


Comments:
Hi, you can import it as such:
```py
from transformers.modeling_albert import AlbertMLMHead
```

It's not in the `__init.py__` as it's not a model but part of one.
Make ElectraPreTrainedModel importable
Author: kumapo
Pull request: no

# 🚀 Feature request

Make `ElectraPreTrainedModel` importable

## Motivation

Consistency.

`DistilBertPreTrainedModel` and `AlbertPreTrainedModel` are importable but I cannot import `ElectraPretrainedModel` from v2.8.0
(see https://github.com/huggingface/transformers/issues/1968)

## Your contribution

https://github.com/huggingface/transformers/pull/4173


Comments:
Thanks for contributing #4173 :).
Write with transformers demo hardware
Author: pelegb
Pull request: no

Hi,
Your package is great and your demo is cool too. I was wondering what kind of hardware does it take to generate text with the large language models. I've had a difficult time running GPT-2 with 1.5B params on a Tesla T4 GPU.
Any pointers will be greatly appreciated.
Thanks in advance,
Barak

Comments:
Hi! We run on K80 and V100 GPUs. Running the 1.5B params model was very costly and hard to maintain, so we're not running it anymore, however. It should run fine on a single V100/Titan RTX however.

Did you have issues with the T4 GPU because of memory?

Thanks for you response. This is really beneficial. I think that it is due to memory. 
Are you serving your models with a flask server, TF-serving, or a different serving framework?
Were you serving it using your PyTorch or Tensorflow implementation?

Thanks again

We're serving our models in PyTorch, using a mix of gunicorn/falcon to handle requests. You can see the detailers [here](https://medium.com/huggingface/scaling-a-massive-state-of-the-art-deep-learning-model-in-production-8277c5652d5f)!

Really clear blog post.
Thanks
[Reformer] Add Enwiki8 Reformer Model - Adapt convert script
Author: patrickvonplaten
Pull request: yes

This PR adapts the reformer conversion script to the newest trax code and adds `reformer-enwik8` to the docs.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4282?src=pr&el=h1) Report
> Merging [#4282](https://codecov.io/gh/huggingface/transformers/pull/4282?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b290c32e1617fa74f44ccd8b83365fe764437be9&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4282/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4282?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4282      +/-   ##
==========================================
+ Coverage   78.39%   78.40%   +0.01%     
==========================================
  Files         120      120              
  Lines       19932    19932              
==========================================
+ Hits        15626    15628       +2     
+ Misses       4306     4304       -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4282?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4282/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.10% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4282/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `74.38% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4282?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4282?src=pr&el=footer). Last update [b290c32...5babcfc](https://codecov.io/gh/huggingface/transformers/pull/4282?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Reformer self attention mask value not converted in apex half precision
Author: theblackcat102
Pull request: no
Labels: name: reformer

# 🐛 Bug

## Information

Model I am using Reformer:

Language I am using the model on English:

The problem arises when using:
* [ x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Install the latest transformers 2.9.0 and apex

```
from transformers import ReformerModel, ReformerTokenizer
import torch
from apex import amp

tokenizer = ReformerTokenizer.from_pretrained('google/reformer-crime-and-punishment')
model =  ReformerModel.from_pretrained('google/reformer-crime-and-punishment').cuda()

optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)

model, optimizer = amp.initialize(model, optimizer, opt_level='O1')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0).cuda()  # Batch size 1
outputs = model(input_ids)

last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

```

## Expected behavior

```
File "/home/model/env/lib/python3.7/site-packages/transformers/modeling_reformer.py", line 840, in forward
    query_key_dots = torch.where(mask, query_key_dots, mask_value)
RuntimeError: expected scalar type Half but found Float
```


- `transformers` version: 2.9.0 (latest source from github)
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.5.0 cuda
- Tensorflow version (GPU?): No tensorflow
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
I have submitted a pull request in #4242 

Awesome I will take a look - thanks for the PR :-)
Evaluating a checkpoint
Author: jmamou
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I have trained a NER model with the following command
python3.6 examples/token-classification/run_ner.py --save_steps 100 --data_dir $DIR --model_type bert --model_name_or_path bert-base-cased --output_dir $DIR/model --do_train --do_eval --do_predict --overwrite_output_dir --labels $DIR/labels.txt --evaluate_during_training --max_seq_length 300

Now, I am trying to evaluate and predict at checkpoint 500 (already evaluated during training using --evaluate_during_training argument) using the following command
python3.6 examples/token-classification/run_ner.py --data_dir $DIR --model_type bert --model_name_or_path bert-base-cased --output_dir $DIR/model/checkpoint-500 --do_eval --do_predict --labels $DIR/labels.txt --max_seq_length 300

However, I don't get same performance as during training (as evaluated at the same checkpoint using --evaluate_during_training argument), I get lower performance (by far).

Any idea about it?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
Yes: you need to use the path to the saved model as the `--model_name_or_path` argument.

`--model_name_or_path` is the argument that conditions which model is loaded.

Will close this for now.
T5 
Author: MVsualStdio
Pull request: no
Labels: name: wontfix

    the code is here
    tokenizer = T5Tokenizer.from_pretrained('t5-small')
    model = T5WithLMHeadModel.from_pretrained('t5-small')
    input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
    print(tokenizer._tokenize("Hello, my dog is cute"))
    print(input_ids)
    print(input_ids.shape)
    outputs = model(input_ids=input_ids,attention_mask=torch.tensor([5.]))
    print(outputs[0].shape)
  

and the error is
extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
UnboundLocalError: local variable 'extended_attention_mask' referenced before assignment


Comments:
Your attention_mask should be torch.tensor([[1,1,1,1,1,1]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Allow gpt2 to be exported to valid ONNX
Author: tianleiwu
Pull request: yes

Problem: gpt2 model and other models using gelu_new or gelu_fast cannot be exported to valid ONNX model.

Causes:
PyTorch will export some constants like 1 or 2 as int64, while ONNX operators like [Add](https://github.com/onnx/onnx/blob/master/docs/Operators.md#Add) and [Pow](https://github.com/onnx/onnx/blob/master/docs/Operators.md#Pow) requires data type matches between two inputs. It is invalid to have one input as float and another as int64.

For GPT2 model, another issue is the ONNX operator [Where](https://github.com/onnx/onnx/blob/master/docs/Operators.md#Where). It requires the first input to be bool tensor but currently it is uint8 tensor.

Add explicit data type cast could solve these problems. It is verified in exporting 3 pretrained models to ONNX model and loading by ONNX Runtime: albert-base-v2, distilgpt2, gpt2.

Comments:
@LysandreJik failing unittest seems unrelated to the change-set introduced in this PR

Indeed @mfuntowicz
Wrong returns of API:get_special_tokens_mask()
Author: ishikawa-takumi
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)
I use transformers tokenizer, and created mask using API: get_special_tokens_mask.
[My Code](https://github.com/ishikawa-takumi/transformers-sample/blob/master/tokenizer.ipynb)

In [RoBERTa Doc](https://huggingface.co/transformers/model_doc/roberta.html), returns of this API is "A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token". But I seem that this API returns "0 for a sequence token, 1 for a special token".
## To reproduce


Steps to reproduce the behavior:

1. Use get_special_tokens_mask().
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Returns of this API is "A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token".

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
Yes, this is https://github.com/huggingface/transformers/issues/1784 come again, the issue is in the doc, not in the code

Indeed, thanks for raising this issue.
Unclear documentation for special_tokens_mask
Author: Evpok
Pull request: no

According to [the docs](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.get_special_tokens_mask), the `special_tokens_mask` returned by e.g. `encode_plus` should have

> 0 for a special token, 1 for a sequence token

Yet when I try

```python
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer.encode_plus("This is a test sentence", add_special_tokens=True) 
```

I get

```python
{'special_tokens_mask': [1, 0, 0, 0, 0, 0, 1],
 'input_ids': [101, 2023, 2003, 1037, 3231, 6251, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0]}
```

Which — to me — would suggest that it is in fact 1 for special tokens and 0 for sequence tokens.
The implementations for [BERT](https://github.com/huggingface/transformers/blob/700331b5ece63381ad1b775fc8661cf3ae4493fd/transformers/tokenization_bert.py#L210) and [RoBERTa](https://github.com/huggingface/transformers/blob/1c542df7e554a2014051dd09becf60f157fed524/transformers/tokenization_roberta.py#L110) support this, so should the documentation be made clearer or is it just me who understood it backward?

Comments:
Indeed, this is a documentation error! Thank you for letting us know!

@LysandreJik This was reintroduced in https://github.com/huggingface/transformers/pull/2989 apparently

Thanks @Evpok for letting me know, this flew under my radar.

My pleasure 😄
Simplify cache vars and allow for TRANSFORMERS_CACHE env
Author: BramVanroy
Pull request: yes

As it currently stands, "TRANSFORMERS_CACHE" is not an accepted environment variable. It seems that the these variables were not updated when moving from version pytorch_transformers to transformers. In addition, the fallback procedure could be improved. and simplified. Pathlib seems redundant here.

This PR:

- adds the TRANSFORMERS_CACHE environment variable
- change the fall-back order of the cache env variables

File_utils (and perhaps the rest of the library) might benefit from using pathlib more widely. In a future endeavour I might take this on.

Comments:
[TF 2.2 compat]  use tf.VariableAggregation.ONLY_FIRST_REPLICA
Author: jplu
Pull request: yes

This PR updates the TF gradient accumulator optimization to make it compliant with the new TF 2.2.

Comments:
Good to be merged for me :) We should update the `setup.py` in same time to do not limit anymore to TF 2.1. Should it be done in this PR? /cc @julien-c @LysandreJik 

I think you can cc @LysandreJik @julien-c 

Yes can you do it in the same PR? and it'll be ready to merge.

Nice! Unfortunately there are other issues with TF2.2, hence why the tests fail and why the version was pinned. I'm patching it now.

Yep I think @mfuntowicz is working on it as it is related to the Tokenizer. At least we have identified why it is not working :)
Bug: can not use pretrained BERT on multiple GPUs with DataParallel (PyTorch 1.5.0)
Author: erikchwang
Pull request: no

Python: 3.6.10
PyTorch: 1.5.0
Transformers: 2.8.0 and 2.9.0

In the following code, I wrap the pretrained BERT with a DataParallel wrapper so as to run it on multiple GPUs:

> import torch, transformers
> model = transformers.AutoModel.from_pretrained("bert-base-multilingual-cased")
> model = torch.nn.DataParallel(model)
> model = model.cuda()
> input = torch.ones([16, 10], dtype=torch.long)
> input = input.cuda()
> model(input)

But I got the following error: 

> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
>     outputs = self.parallel_apply(replicas, inputs, kwargs)
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
>     return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
>     output.reraise()
>   File "/home/anaconda/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
>     raise self.exc_type(msg)
> StopIteration: Caught StopIteration in replica 0 on device 0.
> Original Traceback (most recent call last):
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
>     output = module(*input, **kwargs)
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/anaconda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 734, in forward
>     extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
> StopIteration
> 

But it will work if I remove the DataParallel wrapper.

Comments:
By the way, when I downgrade Pytorch 1.5.0 to 1.4.0, the error disappears. 

The same issue: #3936

Closing in favor of #3936
bug in run_glue
Author: rabeehk
Pull request: no
Labels: name: wontfix

Hi I got this error when running the run_glue.py
    from transformers import glue_compute_metrics as compute_metrics
ImportError: cannot import name 'glue_compute_metrics'

Comments:
If you could include your environment and steps to replicate the issue, that would help. Its working for me on version 2.5.1 (from master branch).

Hi. Sorry then I must have used the old version of transformer, I will close the issue and reopen if needed. thanks. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.