noamaanMulla-03/transformers-issues huggingface dataset:
Always got RuntimeError while converting ALBERT model to TorchScript (.pt file)
Author: xf05888
Pull request: no

I am trying to convert ALBERT to a `.pt` file from the original albert model from transformers.(I am not very familiar with TorchScript so I want the `.pt` to be clean)

The code I ran (following the tutorial from [https://huggingface.co/transformers/torchscript.html](https://huggingface.co/transformers/torchscript.html)):
```

from transformers import AlbertModel, AlbertTokenizer, AlbertConfig
import torch

enc = AlbertTokenizer.from_pretrained("albert-xxlarge-v2")
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]
config = AlbertConfig(vocab_size_or_config_json_file=73000, hidden_size=4096,
    num_hidden_layers=12, num_attention_heads=64, intermediate_size=16384, torchscript=True)
model = AlbertModel(config)
model.eval()
model = AlbertModel.from_pretrained("albert-xxlarge-v2", torchscript=True)
traced_model = torch.jit.trace(model, [tokens_tensor, segments_tensors])
torch.jit.save(traced_model, "albert-xxlarge-v2.pt")
```
But the second last line threw out a error:
`RuntimeError: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3`

From the tutorial:
```
The trace is created relatively to the inputs’ dimensions. It is therefore constrained by the dimensions of the dummy input, and will not work for any other sequence length or batch size. When trying with a different size, an error such as:

The expanded size of the tensor (3) must match the existing size (7) at non-singleton dimension 2
```

So I tried changing `vocab_size_or_config_json_file` to a larger value, but still got the same error.
Am I doing something wrong? Thanks for any advice.



Comments:
Issue with HANS evaluation
Author: prajjwal1
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Albert, Bert

Language I am using the model on (English, Chinese ...): GLUE (MNLI), HANS

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
I have tried to evaluate ALBERT and BERT (pretrained, trained on MNLI) on HANS. I'm using the official example. 

Everytime I run evaluation, I'm getting
```
Heuristic entailed results:
lexical_overlap: 0.0
subsequence: 0.0
constituent: 0.0

Heuristic non-entailed results:
lexical_overlap: 1.0
subsequence: 1.0
constituent: 1.0
```
Results can be reproduced by:
```
python3 hans/test_hans.py --task_name hans --model_type bert/albert --do_eval --data_dir $HANS_DIR --model_name_or_path $MODEL_PATH --max_seq_length 128 --output_dir $MODEL_PATH --per_gpu_eval_batch_size 1024 --overwrite_cache
```


## Expected behavior
It shouldn't be exactly 0 and 1 always and entailment score should be higher than non entailment.

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Linux 5.4.0-29-generic
- Python version: 3.8.2
- PyTorch version (GPU?): 1.5.0+cu101
- Tensorflow version (GPU?): NA
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
Could you please provide some possible areas to look at ? I can look at it and send a PR.

@sgugger @julien-c Thank you for the [PR](https://github.com/huggingface/transformers/issues/4742). There is a problem associated with this PR, that is, if you run `python3 evaluate_heur_output.py /path_to_hans_predictions.txt`, I'm getting an error:
```
guess = guess_dict[key]
Keyerror: 'ex0'
```
which suggests that a key is missing, indicating that `hans_predictions.txt` is not being generated in the expected manner as [HANS repo](https://github.com/huggingface/transformers/issues/4742) indicates. 
For fixing, it would great if evaluation is carried within transformers just like GLUE tasks and that user doesn't have to rely on external [repo](https://github.com/huggingface/transformers/issues/4742) to get predictions.  
It would be good if you can check this, as this error makes the example unusable.

I forgot the header on that file, #5082 should fix.

Evaluation seems to be working fine now. Feel free to close this issue @sgugger . Thanks again. Would be great if evaluation method can be integrated.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
broken ONNX slow test
Author: sshleifer
Pull request: no
Labels: name: Tests

```
    def test_quantize_pytorch(self):
        for model in OnnxExportTestCase.MODEL_TO_TEST:
            path = self._test_export(model, "pt", 12)
>           quantized_path = quantize(Path(path))
```
tests/test_onnx.py:75:  `path` is None

https://github.com/huggingface/transformers/runs/960368281?check_suite_focus=true

Comments:
This seems to be a bit of a flaky test, doesn't it?

There is strange try/except syntax in `_test_export ` that I think can be trivially improved.
allow spaces in bash args with "$@"
Author: sshleifer
Pull request: yes

extends the bugfix for #6477 to more scripts.



Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6521?src=pr&el=h1) Report
> Merging [#6521](https://codecov.io/gh/huggingface/transformers/pull/6521?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/fe61c05b85f98846779bb490a747875e7d54ec2a&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6521/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6521?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #6521   +/-   ##
=======================================
  Coverage   80.59%   80.59%           
=======================================
  Files         156      156           
  Lines       28058    28058           
=======================================
  Hits        22612    22612           
  Misses       5446     5446           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6521?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6521?src=pr&el=footer). Last update [2060181...4032ff5](https://codecov.io/gh/huggingface/transformers/pull/6521?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[sched] polynomial_decay_schedule use default power=1.0
Author: stas00
Pull request: yes

As discussed in https://github.com/huggingface/transformers/pull/6361 we weren't sure why fairseq's `polynomial_decay_schedule` `power` default was `1.0`, and decided to go with `2.0` as the latter does something polynomial.

I got the devs at fairseq to answer this question: https://github.com/pytorch/fairseq/issues/2466#issuecomment-673146603

> myleott wrote:
> This is based on the original BERT code, which implemented a linear decay via a polynomial schedule with power=1.0: https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L37

So, perhaps we do the same or we don't. 

If we don't - then the doc needs to be fixed that the default is `power=2.0` as currently it says `1.0` - my mistake. If we do (this PR), then the doc is already correct.

Thanks.

Comments:
bart-base config.*attention_heads (should be 12 was 16)
Author: ibeltagy
Pull request: no

[Bart-base configuration](https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-base/config.json) has the wrong values for `decoder_attention_heads` and `encoder_attention_heads`. This messes up the self-attention computation and make model totally unusable.  

### Who can help
@sshleifer


Comments:
thx!

Should it be 12?

yes

Fixed, great catch, sorry for the annoyance.

cc @VictorSanh if you are still using.
Can't load t5-11b from pre-trained
Author: saareliad
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform:
- Python version: 3.8.2
- PyTorch version 1.6


### Who can help

 T5: @patrickvonplaten

## Information

The model I am using: T5


## To reproduce

Steps to reproduce the behavior:

```
import transformers
transformers.T5ForConditionalGeneration.from_pretrained("t5-11b")
```

```


OSError: Can't load weights for 't5-11b'. Make sure that:

- 't5-11b' is a correct model identifier listed on 'https://huggingface.co/models'

- or 't5-11b' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.

```


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
the model should be loaded.


Comments:
Hey @saareliad, 
can you try:
```python
t5 = transformers.T5ForConditionalGeneration.from_pretrained('t5-11b', use_cdn = False)
```

Also, see: https://github.com/huggingface/transformers/issues/5423

But the model cannot really be run before we take a closer look at: https://github.com/huggingface/transformers/pull/3578.

@patrickvonplaten mind adding a big disclaimer to the model card for this particular checkpoint? About what you just said (CDN limitation + model parallelism)

Thanks @patrickvonplaten ,
Our work successfully adds (several types of) model parallellism and trains T5 and several other large transformers and is integrated with HF for quite a while.

Will opensource it soon :)
Model Upload does not show up `german-nlp-group/electra-base-german-uncased`
Author: PhilipMay
Pull request: no

Hi,
yesterday I uploaded a new model to `german-nlp-group/electra-base-german-uncased`:

```bash
$ transformers-cli s3 ls --organization german-nlp-group
Neither PyTorch nor TensorFlow >= 2.0 have been found.Models won't be available and only tokenizers, configurationand file/data utilities can be used.
Filename                                          LastModified             ETag                               Size      
------------------------------------------------- ------------------------ ---------------------------------- --------- 
electra-base-german-uncased/config.json           2020-08-14T17:13:01.000Z "10c75064301189f269b4898d4265cd61"       467 
electra-base-german-uncased/pytorch_model.bin     2020-08-14T17:13:37.000Z "a621e1cb07af0a08aaa643af52f9f189" 444881731 
electra-base-german-uncased/tokenizer_config.json 2020-08-14T17:43:33.000Z "7f6d7cb22bc6342b9c942da874754264"        86 
electra-base-german-uncased/vocab.txt             2020-08-14T17:43:31.000Z "e9fa1e40c556fc02c62ebaa214a52dc4"    275501
```

But it does not show up. See here: https://huggingface.co/german-nlp-group

What happened here? Could you fix that?

Thanks
Philip

Comments:
The files are there: https://cdn.huggingface.co/german-nlp-group/electra-base-german-uncased/tokenizer_config.json

But it simply does not show up...

Maybe related to #6478

Having the same problem here. Uploaded a new model (`salti/xlm-roberta-large-arabic_qa`) earlier this morning and it doesn't show up in the model hub, although I can download it and use it using the `from_pretrained` method.

@julien-c @Pierrci 

Maybe some sync service just needs a restart? :-)

Not a sync service, but a (uncaught) user error :)

Fixed: https://huggingface.co/german-nlp-group/electra-base-german-uncased#german-electra-uncased
Upladed model is not indexed
Author: mrm8488
Pull request: no

# ❓ Questions & Help

Hi guys, I uploaded a model several hours ago (t5-base-finetuned-boolq) and it is not indexed in the model hub search engine yet!

Thanks, Manu

Comments:
The model is also not listed on your [page ](https://huggingface.co/mrm8488). Can you try re-uploading ?

If you load it in your code ```mrm8488/t5-base-finetuned-boolq``` it works! Maybe a problem indexing it.

cc @julien-c 

Hi everyone, has there been a way fix this? I also uploaded a model (t5-podcast-summarisation) that hasn't shown up on the model hub. I am able to load it in my code using `paulowoicho/t5-podcast-summarisation` though

Fixed:
- https://huggingface.co/mrm8488/t5-base-finetuned-boolq
- https://huggingface.co/paulowoicho/t5-podcast-summarisation
Error : forward() got an unexpected keyword argument 'inputs_embeds'
Author: CNelias
Pull request: no

Hello,

I am trying to train a GPT2 from scratch using ```modeling_gpt2.py``` as a base.
I declared the model as follow : 
```Python 
config = GPT2Config(vocab_size = VELSIZE, n_positions = SEQLEN, n_embd = EMBEDSIZE, n_layer = NUMLAYER,n_ctx = SEQLEN, n_head = NUMHEAD)  
model = GPT2Model(config)  
``` 
I don't need to use the buid-in embeddings for my application and would like to pass my input tensor as-is, but trying ```model(inputs_embeds = test)```,  ```model.forward(inputs_embeds = test)```, ```model(input_ids=None,
      past=None,
      attention_mask=None,
      token_type_ids=None,
      position_ids=None,
      head_mask=None,
      inputs_embeds=test)``` or any other variant I can think of always results in the following error :
```Python 
Traceback (most recent call last):

  File "<ipython-input-52-616a2eb9b3f4>", line 7, in <module>
    inputs_embeds=testx)

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)

TypeError: forward() got an unexpected keyword argument 'inputs_embeds'
```

Is this a bug or am I doing it wrong ?

Comments:
I can run the following code succesfully: 

```
from transformers import GPT2Model, GPT2Tokenizer
import torch

model = GPT2Model.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<PAD>')

input_ids = tokenizer.encode("Hello, how are you?", return_tensors='pt')
inputs_embeds = model.wte(input_ids)

model(inputs_embeds=inputs_embeds)  # runs without error
```

Can you update `transformers` to the most current version and verify that you can run the code snippet I posted? 



Running ```conda update transformers``` returned that I already have the latest version.
As for you snippet, I still get the same error : 
```Python
Traceback (most recent call last):

  File "<ipython-input-136-d0df910b9d57>", line 10, in <module>
    model(inputs_embeds=inputs_embeds)  # runs without error

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)

TypeError: forward() got an unexpected keyword argument 'inputs_embeds'
```
Is this because I installed ```transformers``` with ```conda``` instead of ```pip``` ?

Edit : this is indeed probably a conda issue. When I run the snippet in ```atom``` (with the python depedency and not anaconda) instead of ```spyder```, then it works.

I get the same error . upgrading transformers vai pip doesn't solve the problem. any solution?
[docs] Fix wrong newline in the middle of a paragraph
Author: romainr
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6573?src=pr&el=h1) Report
> Merging [#6573](https://codecov.io/gh/huggingface/transformers/pull/6573?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cfa26d2b412ac3494eef06506004ca857c115ad9&el=desc) will **increase** coverage by `0.14%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6573/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6573?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6573      +/-   ##
==========================================
+ Coverage   79.24%   79.38%   +0.14%     
==========================================
  Files         156      156              
  Lines       28130    28130              
==========================================
+ Hits        22292    22332      +40     
+ Misses       5838     5798      -40     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6573?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_mbart.py](https://codecov.io/gh/huggingface/transformers/pull/6573/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWJhcnQucHk=) | `56.25% <0.00%> (-39.07%)` | :arrow_down: |
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/6573/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `93.76% <0.00%> (+3.52%)` | :arrow_up: |
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/6573/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `99.16% <0.00%> (+32.50%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6573?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6573?src=pr&el=footer). Last update [cfa26d2...0a20942](https://codecov.io/gh/huggingface/transformers/pull/6573?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
[Pegasus Doc] minor typo
Author: patil-suraj
Pull request: yes

Minor typo correction
@sshleifer

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6579?src=pr&el=h1) Report
> Merging [#6579](https://codecov.io/gh/huggingface/transformers/pull/6579?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7516bcf27319a2aea9bbe927f8e4d8e501e23c99&el=desc) will **decrease** coverage by `0.52%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6579/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6579?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6579      +/-   ##
==========================================
- Coverage   80.53%   80.00%   -0.53%     
==========================================
  Files         156      156              
  Lines       28130    28130              
==========================================
- Hits        22654    22506     -148     
- Misses       5476     5624     +148     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6579?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/6579/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb24ucHk=) | `25.55% <0.00%> (-70.00%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6579/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `69.06% <0.00%> (-29.32%)` | :arrow_down: |
| [src/transformers/optimization\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/6579/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb25fdGYucHk=) | `33.33% <0.00%> (-24.33%)` | :arrow_down: |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6579/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `88.26% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/6579/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `79.94% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6579/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `88.34% <0.00%> (+63.80%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6579?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6579?src=pr&el=footer). Last update [7516bcf...1e290b6](https://codecov.io/gh/huggingface/transformers/pull/6579?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
fixed fast tokenizer use in QA pipeline and added corresponding test
Author: bdalal
Pull request: yes

This PR fixes https://github.com/huggingface/transformers/issues/6545

The problem is that the behavior of the python tokenizer and the rust based fast tokenizer is very different.
The python tokenizer handles cases where inputs are in different formats (str tokens and int tokens and vice versa), where as the fast tokenizer is unable to do so.

My modifications include pretokenizing the query and along side the context, but not encoding it as is done now.
Both the query and context are encoded together by the the tokenizer's `encode_plus` method with `is_pretokenized` set to True.

Some edge cases where the fast tokenizer fails because of limited functionality (compare to python tok) have been added and I've included comments in those places.

Comments:
The failed test seems to be failing for merged PRs too. I'll fix up the formatting with black