noamaanMulla-03/transformers-issues huggingface dataset:
Can we use tf.keras.mixed_precision.experimental.set_policy ?
Author: ydshieh
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I want to use `mixed_precision`, and I found [tf.keras.mixed_precision.experimental.Policy](https://www.tensorflow.org/api_docs/python/tf/keras/mixed_precision/experimental/Policy).

So I put `tf.keras.mixed_precision.experimental.set_policy("mixed_float16")` before `TFBertModel.from_pretrained(pretrained_weights)`. When I run the code, I got the following error:

> InvalidArgumentError: cannot compute AddV2 as input #1(zero-based) was expected to be a half tensor but is a float tensor [Op:AddV2] name: tf_bert_model_1/bert/embeddings/add/

which happened at `ret = model(model.dummy_inputs, training=False)  # build the network with dummy inputs`.

I am not sure if I used it correctly. I think `tf.keras.mixed_precision.experimental.set_policy` is supposed to be used before constructing / build the model, as the tf page says `Policies can be passed to the 'dtype' argument of layer constructors, or a global policy can be set with 'tf.keras.mixed_precision.experimental.set_policy'`.

I wonder if I can use AMP with tf based transformer models and how. Thanks.

[error.txt](https://github.com/huggingface/transformers/files/3907032/error.txt)


Comments:
For now we need to use:

```python
tf.config.optimizer.set_experimental_options({"auto_mixed_precision": True})
```

Please see [example here](https://github.com/huggingface/transformers/blob/master/examples/run_tf_glue.py).

Thanks. I tried it during waiting the answer, and it doesn't speed up the training. I probably can post my model later.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Arxiv README
Author: LysandreJik
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2747?src=pr&el=h1) Report
> Merging [#2747](https://codecov.io/gh/huggingface/transformers/pull/2747?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2184f87003c18ad8a172ecab9a821626522cf8e7?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2747/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2747?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #2747   +/-   ##
======================================
  Coverage    75.1%   75.1%           
======================================
  Files          93      93           
  Lines       15249   15249           
======================================
  Hits        11452   11452           
  Misses       3797    3797
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2747?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2747?src=pr&el=footer). Last update [2184f87...69d18f4](https://codecov.io/gh/huggingface/transformers/pull/2747?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LOTM

(= Looks outstanding to me)

I really did my best
Development Infrastructure for ML Projects
Author: shashankMadan-designEsthetics
Pull request: no

Hi guys, I am sorry to post an issue that is a bit outside the scope of this project. But I have been a consistent watcher of the transformers project and it is excellent on how I can collaborate and coordinate with people and develop something.
But I am a rookie when it comes to working with people and get going with a project and I have been assigned a task to create reliable and scalable infrastructure, where a team can have space for research, development, test, deploy.
I have been dabbling around with bitbucket pipelines and docker but it would be helpful to get your opinions on it.

Comments:
This is an interesting subject but is way too broad for discussing here

@julien-c any forum you know for having this discussion?
Reason for using einsum in xlnet?
Author: SungMinCho
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello.

This might be a newbie question, so I apologize in advance.

While reading your implementation of xlnet, I ran into several usages of `torch.einsum` function.

example) `k_head_h = torch.einsum('ibh,hnd->ibnd', cat, self.k) `

After studying the definition of einsum, I came to a conclusion that the above statement is exactly like using a linear layer (without bias) (from dimension h to n*d), and then resizing the output to be ibnd.

So if I'm not wrong, is there any reason to prefer using `torch.einsum` over `nn.Linear`?

Is it related to performance issues?

I ran a simple test, and `nn.Linear` seems to be a bit faster than `torch.einsum`.

I would really appreciate your help.

Thank you.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add contributors snapshot
Author: clmnt
Pull request: yes

powered by https://github.com/sourcerer-io/hall-of-fame

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2762?src=pr&el=h1) Report
> Merging [#2762](https://codecov.io/gh/huggingface/transformers/pull/2762?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/33d3072e1c54bcd235447b98c6dea1b4cb71234c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2762/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2762?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #2762   +/-   ##
======================================
  Coverage    75.1%   75.1%           
======================================
  Files          93      93           
  Lines       15249   15249           
======================================
  Hits        11452   11452           
  Misses       3797    3797
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2762?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2762/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `89.93% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2762/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `95.77% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2762/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.21% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2762/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `94.27% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2762/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `81.39% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2762?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2762?src=pr&el=footer). Last update [33d3072...8b6a98e](https://codecov.io/gh/huggingface/transformers/pull/2762?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Nice!
[docs] Add menu w/ links to other pages on hf.co
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2761?src=pr&el=h1) Report
> Merging [#2761](https://codecov.io/gh/huggingface/transformers/pull/2761?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/33d3072e1c54bcd235447b98c6dea1b4cb71234c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2761/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2761?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #2761   +/-   ##
======================================
  Coverage    75.1%   75.1%           
======================================
  Files          93      93           
  Lines       15249   15249           
======================================
  Hits        11452   11452           
  Misses       3797    3797
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2761?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2761?src=pr&el=footer). Last update [33d3072...e6944e6](https://codecov.io/gh/huggingface/transformers/pull/2761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


btw @LysandreJik mkdocs looks really cool :)

Yeah I really like mkdocs as well
Fix multi-gpu evaluation in run_glue.py example
Author: peteriz
Pull request: yes

Fix multi-gpu evaluation while training in `examples/run_glue.py`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2752?src=pr&el=h1) Report
> Merging [#2752](https://codecov.io/gh/huggingface/transformers/pull/2752?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9d87eafd118739a4c121d69d7cff425264f01e1c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2752/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2752?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2752   +/-   ##
=======================================
  Coverage   74.51%   74.51%           
=======================================
  Files          87       87           
  Lines       14920    14920           
=======================================
  Hits        11117    11117           
  Misses       3803     3803
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2752?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2752?src=pr&el=footer). Last update [9d87eaf...31218ea](https://codecov.io/gh/huggingface/transformers/pull/2752?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks!
Saving tokenizer vocabulary throws error when passing file name instead of directory.
Author: dchurchwell
Pull request: no

# 🐛 Bug

## Information

Using transfo-xl-wt103

UnboundLocalError: local variable 'vocab_file' referenced before assignment


## To reproduce

Steps to reproduce the behavior:

```
from transformers import TransfoXLTokenizer, TFTransfoXLLMHeadModel

tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
tokenizer.save_vocabulary('vocab.txt')
```

## Pull Request

https://github.com/huggingface/transformers/pull/2754



Comments:
Closed by #2754
Cannot reproduce SQUAD Example
Author: eblaudez
Pull request: no

I'm not able to reproduce the squad experimentation (via the example). I tried the command line;
python run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

That gived very weird results

Then I read a little the forum and I tried:
python3 run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --version_2_with_negative \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 10000 \
  --output_dir debug_squad/ \
  --overwrite_output_dir

(I also tried with v2 dataset it not works anymore). Can you give me some lead to reproduce the results given in the git readme, or the branch to do it ?


Comments:
1. What do you mean "weird results"?
2. What do you mean "v2 dataset it not works anymore"?
3. Please provide all the information required in the template, i.e. python version, transformers version, torch version etc

1. F1 & Exact match:~18 (should be 88/81 no ?)
2. Squad 2.0
3. Python 3.6, last transformer version (clone yesterday), torch 1.4.0

Tensorboard :
![squad_pb](https://user-images.githubusercontent.com/11454249/74007546-dbc43f80-497e-11ea-81bb-f7f8622d1a2a.png)
Loss is calculated on all tokens, including padding, in the LM fine-tuning example
Author: plroit
Pull request: no

# 🐛 Bug
The BERT fine-tuning example uses a special index to mark ignored locations for the loss function:
`loss_fct = CrossEntropyLoss(ignore_index=-1)`
While in the same example, the masking function that samples locations to be included or excluded uses a different index: -100 (which is the default ignored index for the cross-entropy loss function, if one is not supplied):
`labels[~masked_indices] = -100  # We only compute loss on masked tokens`

Model I am using (Bert, XLNet ...): All models.

Language I am using the model on (English, Chinese ...): All languages

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: LM Finetuning
* [ ] my own task or dataset: (give details below)

## Expected behavior
Loss should be computed only on the 15% (mlm_probability) of sampled tokens.
     
- `transformers` version: 2.3, 2.4


Comments:
Hello, `BertForMaskedLM` [does not use an ignore index set to -1](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1018), nor does any other models.

It was updated in [2.4.0](https://github.com/huggingface/transformers/releases/tag/v2.4.0). The scripts should be run against the latest version of the library.

If you want to run against v2.3.0, please use a script from v2.3.0, for example [run_lm_finetuning](https://github.com/huggingface/transformers/blob/v2.3.0/examples/run_lm_finetuning.py).

You're right, I mixed up the versions. closing the issue.
How to generate different suggestions with GPT2 or XLNet like Write With Transformers?
Author: cppntn
Pull request: no

Hello, 

I want to generate with run_generation more different suggestions of the next words, preferably with variable length and different terms or synonyms, like it is done in Write With Transformer.

Any suggestion or idea on how to achieve this?

Thanks

Comments:
I closed this issue since I found useful to set the `do_sample` argument to True, as mentioned in this issue: https://github.com/huggingface/transformers/issues/2415
The prediction output is random
Author: Mozen
Pull request: no

When I  use the official example scripts to predict a text sentence classification model,  I found that the output is different every time。

```
from transformers import BertTokenizer, BertForSequenceClassification
import torch
import numpy as np

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
model.eval()
input_ids = torch.tensor(tokenizer.encode("my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs= model(input_ids)
print(outputs)
```

fist result :(tensor([[-0.1939,  0.1449]], grad_fn=<AddmmBackward>),)
second result:(tensor([[-0.2425, -0.2737]], grad_fn=<AddmmBackward>),)
third result: (tensor([[ 0.0494, -0.7208]], grad_fn=<AddmmBackward>),)
......

I expected the outputs to be the same... am I doing this wrong?

Comments:
Hi @Mozen, 
You will need to train your model for sequence classification first.
The pre-trained models are not yet trained for the downstream task. So right now, you have an untrained sequence classification head on top of Bert. 

I could not find where it was mentioned in the docs, but have a look at [this comment](https://github.com/huggingface/transformers/issues/1979#issuecomment-559597512).

@jwallat ok， thanks  a lot！

Please close the question if your question is answered.
Fix documentation in ProjectedAdaptiveLogSoftmax
Author: ari-holtzman
Pull request: yes

The shape of outputs for forward in ProjectedAdaptiveLogSoftmax is flipped in the documentation: it should be log probabilities when `labels` is `None` and NLLs otherwise. This is what the code does, but the docstring has them flipped.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2766?src=pr&el=h1) Report
> Merging [#2766](https://codecov.io/gh/huggingface/transformers/pull/2766?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/33d3072e1c54bcd235447b98c6dea1b4cb71234c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2766/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2766?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #2766   +/-   ##
======================================
  Coverage    75.1%   75.1%           
======================================
  Files          93      93           
  Lines       15249   15249           
======================================
  Hits        11452   11452           
  Misses       3797    3797
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2766?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_transfo\_xl\_utilities.py](https://codecov.io/gh/huggingface/transformers/pull/2766/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsX3V0aWxpdGllcy5weQ==) | `53.33% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2766?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2766?src=pr&el=footer). Last update [33d3072...8725c54](https://codecov.io/gh/huggingface/transformers/pull/2766?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Perfect, thank you!!
FileNotFoundError: [Errno 2] No such file or directory: 'data/dump.txt'
Author: MrLinNing
Pull request: no
Labels: name: wontfix


@stefan-it
Hello, I am new learner in BERT and I want to have a try the excellent work - distilBert.
But the problem happened when I ran the training step, and Could you tell me where can I download the `dump.txt` file ?
Thank you very much!



Comments:
As stated [here](https://github.com/huggingface/transformers/blob/1ab8dc44b3d84ed1894f5b6a6fab58fb39298fc7/examples/distillation/README.md), the `dump.txt` file is **your training file**. This file will contain one sequence per line (a sequence being composed of one of several coherent sentences).

> @stefan-it
> Hello, I am new learner in BERT and I want to have a try the excellent work - distilBert.
> But the problem happened when I ran the training step, and Could you tell me where can I download the `dump.txt` file ?
> Thank you very much!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BART
Author: clmnt
Pull request: no

# 🌟New model addition

## Model description

method for pre-training seq2seq models by de-noising text. BART outperforms previous work on a bunch of generation tasks (summarization/dialogue/QA), while getting similar performance to RoBERTa on SQuAD/GLUE

## Open Source status

* [ ] the model implementation is available: not yet
* [ ] the model weights are available: not yet
* [ ] who are the authors: @yinhanliu @ernamangoyal 


Comments:
Duplicate of #1676
distilbert-base-cased
Author: VictorSanh
Pull request: yes

- Weights
- Readmes and docs
- Previous omissions

weights are uploaded on s3, along with modelcards.

@LysandreJik Could you make sure I didn't forget anything?
@mfuntowicz Could have a check on the pipeline part?

Comments:
do we want to actually change the model used in the pipeline?

> do we want to actually change the model used in the pipeline?

I'm not sure to understand the rationale behind the question.
Purely from a perf point of view, it's the same inf speed, while having better metrics than before.

Nevermind the failing test, it's a Heisenbug. Merge when ready.
Transformers for WebNLG tasks 
Author: MathewAlexander
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Can we leverage GPT-2 pre-trained model for WebNLG tasks ?http://webnlg.loria.fr/pages/challenge.html

The WebNLG challenge consists in mapping data to text
similar to what is being done in https://github.com/tyliupku/wiki2bio.

Comments:
Actually I'm working on this right now. Interested to know as well if anyone else has done it.
Most probably this is possible.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
how to select best model in run_glue
Author: TLCFYBJJHYYSND
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
i'm a green hand and i know it is a rediculous problem.I just saw 
`    # Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained()
    if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0) and not args.tpu:
        # Create output directory if needed
        if not os.path.exists(args.output_dir) and args.local_rank in [-1, 0]:
            os.makedirs(args.output_dir)`

in row 526,but it seems like that i can hardly find any word about selecting the best model.Would be very appriciate if you can tell me.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.