noamaanMulla-03/transformers-issues huggingface dataset:
When I run `import transformers` , it reports an error. But I had no problem with pyrotch-transformers before. Is it a TensorRT problem?
Author: songyingxin
Pull request: no

# ❓ Questions & Help

when i run `import transformers`, i get this:

2020-04-26 17:02:59.210145: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
*** Error in `/conda-torch/bin/python': double free or corruption (!prev): 0x00007faa0eac23c0 ***

is there anything can fix it?

Comments:
run_xnli doesn't execute
Author: antmarakis
Pull request: no

# 🐛 Bug

## Information

(a related issue is this: #3950 )

I am trying to run the `run_xnli.py` script from the [Examples documentation](https://huggingface.co/transformers/examples.html#xnli), but I am getting an error. I am trying to run the script without training.

## To reproduce

Steps to reproduce the behavior:

I follow the [documentation](https://huggingface.co/transformers/examples.html#xnli)

My args are the following:

```
export XNLI_DIR=/path/to/xnli

python run_xnli.py \
  --model_type bert \
  --model_name_or_path bert-base-multilingual-uncased \
  --language de \
  --train_language en \
  --do_eval \
  --data_dir $XNLI_DIR \
  --per_gpu_train_batch_size 32 \
  --learning_rate 5e-5 \
  --num_train_epochs 1.0 \
  --max_seq_length 128 \
  --output_dir bert-base-multilingual-uncased \
  --save_steps -1
```

I get the following error:

```
Traceback (most recent call last):
  File "run_xnli.py", line 646, in <module>
    main()
  File "run_xnli.py", line 638, in main
    result = evaluate(args, model, tokenizer, prefix=prefix)
  File "run_xnli.py", line 290, in evaluate
    outputs = model(**inputs)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 1139, in forward
    loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 932, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2317, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2115, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 2 is out of bounds.
```

## Expected behavior

I am not sure I am afraid. I am expecting some sort of accuracy measure as shown in the Example.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: Linux-4.15.0-74-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
How to do_predict on run_glue?
Author: azamatolegen
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
I have fine-tuned BERT for sequence classification task by running run_glue script. Now I have trained and evaluated model. My question is how do I make prediction with it on new instances (test set)? Thanks!

Comments:
#2198 Same here. A predict script can be really helpful to researchers.

same here

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Clarification about GPT2LMHeadModel lm_head weights
Author: thesamuel
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->
Each time the GPT2LMHeadModel is loaded from pretrained weights, the following is logged:
```
Weights of GPT2LMHeadModel not initialized from pretrained model: ['lm_head.weight']
```

Just to clarify, is this OK because we tie the output (`lm_head`) weights to the input weights?

Comments:
yes exactly - this should not be a problem :-)
MLM Loss not decreasing when pretraining Bert from scratch
Author: JF-D
Pull request: no

I want to pretrain bert base model from scratch with en-wikipedia dataset, since I havn't found a bookcorpus copy. The code I used was adapted from pytorch-pretrained-bert and Nvidia Megatron-LM. I can finetune BERT on SQuAD and get SOTA result. But for pretraining, the MLM loss stays around 7.3. I have no idea why this happens. Can someone offer some possible solutions? Great thanks!

Comments:
Maybe it is a bug for Roberta vocab file.
Author: czheng17
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [1 ] the official example scripts: (give details below)
* [1] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [1 ] my own task or dataset: (give details below)

Hi Huggingface team,

When I use Roberta, I found that there are so many words start with "Ġ" in the vocab file you provided. I just want to make sure whether it is a bug or not. Around 40,771 words start with "Ġ".

Roberta-base vocab file: https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json

Roberta-large vocab file:
https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-vocab.json

Best Regards,
Chen





    


Comments:
Hi @czheng17,

that's because RoBERTa uses a byte-level BPE (like GPT-2), see section 4.4 in the original paper.

A good explanation can be found here:

https://github.com/pytorch/fairseq/issues/1716#issuecomment-588750983

:)

Hi @stefan-it ,

Interesting! Thank you for your help!

Best,
Remove boto3 dependency
Author: julien-c
Pull request: yes

Downloading a model from `s3://` urls was not documented anywhere and I suspect it doesn't work with our `from_pretrained` methods anyways.

Removing boto3 also kills its transitive dependencies (some of which are slow to support new versions of Python), contributing to a leaner library:

```
boto3-1.12.46
botocore-1.15.46
docutils-0.15.2
jmespath-0.9.5
python-dateutil-2.8.1
s3transfer-0.3.3
six-1.14.0
urllib3-1.25.9
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3968?src=pr&el=h1) Report
> Merging [#3968](https://codecov.io/gh/huggingface/transformers/pull/3968?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/4e817ff41885063e08bb3bcd63e5adfd835b9911&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3968/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3968?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3968      +/-   ##
==========================================
+ Coverage   78.44%   78.51%   +0.07%     
==========================================
  Files         111      111              
  Lines       18518    18486      -32     
==========================================
- Hits        14527    14515      -12     
+ Misses       3991     3971      -20     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3968?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3968/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `72.61% <100.00%> (+3.74%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3968/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.06% <0.00%> (+0.12%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3968?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3968?src=pr&el=footer). Last update [4e817ff...b440a19](https://codecov.io/gh/huggingface/transformers/pull/3968?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix the typos
Author: airsplay
Pull request: yes

Fix the typos of "[0.0, 1.0[" --> "[0.0, 1.0]".

Comments:
That's not a typo :)

https://en.wikipedia.org/wiki/Interval_(mathematics)

Ohhh! I am not aware of this notation before. (only [X, Y) is in my knowledge-base). Thanks for the clarification.
Run Multiple Choice failure with ImportError: cannot import name 'AutoModelForMultipleChoice'
Author: leixiaofeng-astar
Pull request: no

# 🐛 Bug

## Information
Run the example, it returns error as below. The version in April 17 doesn't have the problem.
python ./examples/run_multiple_choice.py \
> --task_name swag \
> --model_name_or_path roberta-base \
> --do_train \
> --do_eval \
> --data_dir $SWAG_DIR \
> --learning_rate 5e-5 \
> --num_train_epochs 3 \
> --max_seq_length 80 \
> --output_dir models_bert/swag_base \
> --per_gpu_eval_batch_size=16 \
> --per_gpu_train_batch_size=16 \
> --gradient_accumulation_steps 2 \
> --overwrite_output
Traceback (most recent call last):
  File "./examples/run_multiple_choice.py", line 26, in <module>
    from transformers import (
ImportError: cannot import name 'AutoModelForMultipleChoice'

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
Follow the exact steps in README.md
1. Download the transformer with git clone
2. git clone https://github.com/rowanz/swagaf.git
3. export SWAG_DIR=/path/to/swag_data_dir
python ./examples/run_multiple_choice.py \
--task_name swag \
--model_name_or_path roberta-base \
--do_train \
--do_eval \
--data_dir $SWAG_DIR \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--max_seq_length 80 \
--output_dir models_bert/swag_base \
--per_gpu_eval_batch_size=16 \
--per_gpu_train_batch_size=16 \
--gradient_accumulation_steps 2 \
--overwrite_output

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
***** Eval results *****
eval_acc = 0.8338998300509847
eval_loss = 0.44457291918821606

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: ubuntu 18.04
- Python version: 3.6.8
- PyTorch version (GPU?): 1.4.0 GPU
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No.


Comments:
You need to install transformers from source as specified in the README
Problem with downloading the XLNetSequenceClassification pretrained xlnet-large-cased
Author: tienpham-dtp
Pull request: no

 I've run this before and it was fine. However, this time, i keep encountering this runtime error. In fact, when i tried to initialize model = XLNetModel.from_pretrained('xlnet-large-cased') - it also gives me the 'negative dimension' error.

`model = XLNetForSequenceClassification.from_pretrained("xlnet-large-cased", num_labels = 2)
model.to(device)`
`---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-37-d6f698a3714b> in <module>()
----> 1 model = XLNetForSequenceClassification.from_pretrained("xlnet-large-cased", num_labels = 2)
      2 model.to(device)

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in __init__(self, num_embeddings, embedding_dim, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse, _weight)
     95         self.scale_grad_by_freq = scale_grad_by_freq
     96         if _weight is None:
---> 97             self.weight = Parameter(torch.Tensor(num_embeddings, embedding_dim))
     98             self.reset_parameters()
     99         else:

RuntimeError: Trying to create tensor with negative dimension -1: [-1, 1024]`

Comments:
Hey mate, I've encountered the same issue. How did you solve it?

I encounter this issue. How can I solve this?

the problem is when l import xlnet from pytorch_transformers. instead, you should be importing it from the module called 'transformers'
fix output_dir / tokenizer_name confusion
Author: antmarakis
Pull request: yes

Fixing #3950 

It seems that `output_dir` has been used in place of `tokenizer_name` in `run_xnli.py`. I have corrected this typo.

I am not that familiar with code style in this repo, in a lot of instances I used:

`args.tokenizer_name if args.tokenizer_name else args.model_name_or_path`

Instead, I believe I could have set `args.tokenizer_name = args.tokenizer_name if args.tokenizer_name else args.model_name_or_path` somewhere at the start of the script, so that the code could be cleaner below.

Comments:
No, this isn't correct. This was consistent with other scripts (before they were ported to the new Trainer in #3800) where we always saved the model and its tokenizer before re-loading it for eval 

So I think that code is currently right. (but again, should be re-written to Trainer pretty soon)
DistributedSampler can't shuffle the dataset
Author: elk-cloner
Pull request: no

# 🐛 Bug

## Information

I'm trying to fine-tune BERT model using ```run_language_modeling.py```.  

Language I am using the model on is Persian:

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

But according to this [issue](https://github.com/pytorch/pytorch/issues/31771) there is a bug in ```torch.utils.data.distributed.DistributedSampler``` so that during different epochs shuffling operation doesn't work properly(it's not working). 
To solve this problem: according to pytorch official example [here](https://github.com/pytorch/examples/blob/ad775ace1b9db09146cdd0724ce9195f7f863fff/imagenet/main.py#L238), we should add ```train_sampler.set_epoch(epoch)``` before each new epoch at this [line](https://github.com/huggingface/transformers/blob/f8208fa456039b46873a2e497b6318d30a4fc84e/examples/run_language_modeling.py#L322) 

## To reproduce

Steps to reproduce the behavior:

1. compare batches between different epoch like mentioned [issue](https://github.com/pytorch/pytorch/issues/31771) 

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: transformers==2.8.0
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): torch==1.4.0 (Yes)
- Tensorflow version (GPU?): tensorflow-gpu==2.1.0 (Yes)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: distributed


Comments:
I think you are right

Isn't there the same issue in other places?
E.g. in trainer.py: https://github.com/huggingface/transformers/blob/97a375484c618496691982f62518130f294bb9a8/src/transformers/trainer.py#L305-L307

I forgot to re-add this in Trainer when merging #3800 

It's on my todo-list, but feel free to open a PR if you can do it faster than I can

Great. Personally I've not yet upgraded to the newer version with trainer.py, so I'll leave it for you, thanks.
Model cards for KoELECTRA
Author: monologg
Pull request: yes
Labels: name: model card

Hi:)

I've recently uploaded `KoELECTRA`, pretrained ELECTRA model for Korean:)
Thanks for supporting this great library and S3 storage:)


Comments:
That's great! Model pages: 
https://huggingface.co/monologg/koelectra-base-generator
https://huggingface.co/monologg/koelectra-base-discriminator

cc'ing @LysandreJik and @clarkkev for information
[model_cards] Add model card for Hindi-Bert
Author: mapmeld
Pull request: yes
Labels: name: model card

Trained with Google ELECTRA and Hindi Corpus (OSCAR CommonCrawl and latest Hindi Wikipedia)

README offers more details and links to notebooks showing pretraining and finetuning on movie reviews

Comments:
[Model page](https://huggingface.co/monsoon-nlp/hindi-bert)
Create model card
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4007?src=pr&el=h1) Report
> Merging [#4007](https://codecov.io/gh/huggingface/transformers/pull/4007?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/4e817ff41885063e08bb3bcd63e5adfd835b9911&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4007/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4007?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4007      +/-   ##
==========================================
- Coverage   78.44%   78.44%   -0.01%     
==========================================
  Files         111      111              
  Lines       18518    18518              
==========================================
- Hits        14527    14526       -1     
- Misses       3991     3992       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4007?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4007/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `68.49% <0.00%> (-0.37%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4007?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4007?src=pr&el=footer). Last update [4e817ff...0bb517d](https://codecov.io/gh/huggingface/transformers/pull/4007?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Summarisation tuning
Author: dimagalat
Pull request: no

Hi everybody,

I’ve tried using BART summarisation code, and I had a question about finetune.py 

Can SummarisationTrainer checkpoint be loaded as a BartForConditionalGeneration model from the evaluation script?

Comments:
Great, thanks @sshleifer 

(Duplicate of https://github.com/huggingface/transformers/issues/3853)


sorry, mentioned wrong issue
Minor fix in Transformers Notebook
Author: stefan-it
Pull request: yes

Minor fix for the German example sentence in notebook :)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4025?src=pr&el=h1) Report
> Merging [#4025](https://codecov.io/gh/huggingface/transformers/pull/4025?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2fade302ac55a289f00c61b0947a8e00dadf41fe&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4025/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4025?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4025   +/-   ##
=======================================
  Coverage   78.51%   78.51%           
=======================================
  Files         111      111           
  Lines       18486    18486           
=======================================
+ Hits        14514    14515    +1     
+ Misses       3972     3971    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4025?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4025/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.59% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4025/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.06% <0.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4025/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.02% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4025?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4025?src=pr&el=footer). Last update [2fade30...3a3c460](https://codecov.io/gh/huggingface/transformers/pull/4025?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Perfekt, vielen Dank haha :-)