noamaanMulla-03/transformers-issues huggingface dataset:
Tokenizers v3.0.0
Author: mfuntowicz
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3185?src=pr&el=h1) Report
> Merging [#3185](https://codecov.io/gh/huggingface/transformers/pull/3185?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7420a6a9cc1750c2bd2c2c245d00048ec36d3bf0?src=pr&el=desc) will **decrease** coverage by `0.24%`.
> The diff coverage is `80.42%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3185/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3185?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3185      +/-   ##
==========================================
- Coverage   77.79%   77.55%   -0.25%     
==========================================
  Files         100      100              
  Lines       17025    17105      +80     
==========================================
+ Hits        13245    13265      +20     
- Misses       3780     3840      +60
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3185?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/3185/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `95.33% <ø> (-1.7%)` | :arrow_down: |
| [src/transformers/tokenization\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/3185/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcm9iZXJ0YS5weQ==) | `94.36% <100%> (-5.64%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/3185/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `74.51% <100%> (-0.28%)` | :arrow_down: |
| [src/transformers/tokenization\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/3185/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdHJhbnNmb194bC5weQ==) | `40.67% <100%> (-0.43%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3185/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `86.18% <78.61%> (-5.81%)` | :arrow_down: |
| ... and [1 more](https://codecov.io/gh/huggingface/transformers/pull/3185/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3185?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3185?src=pr&el=footer). Last update [7420a6a...860cf66](https://codecov.io/gh/huggingface/transformers/pull/3185?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Really like the new typings!
Sentence pair classification
Author: Mahmedturk
Pull request: no
Labels: name: wontfix

name: Ex: Sequence Classification

name: Usage

# ❓ Questions & Help
Hi,

I want to do sentence pair classification on Quora Questions Dataset by fine-tuning BERT. I am new to this and do not know where to start? Can anyone let me know how do i get started with this? 




<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update doc for {Summarization,Translation}Pipeline and other tweaks
Author: julien-c
Pull request: yes


Comments:
@sshleifer It's not 100% needed to have model cards but definitely encouraged as they unlock new features e.g. here, discoverability of the models in the model hub, etc.

Yeah, adding an item to the checklist (I guess in `templates/adding_a_new_model`) would be nice, do you want to do it?
Bug in ElectraForTokenClassification
Author: LiZongyue
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): ElectraForTokenClassification

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [√] the official example scripts: (give details below)
  huggingface.co/transformers/model_doc/electra.html#electrafortokenclassification
* [×] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ×] an official GLUE/SQUaD task: (give the name)
* [ √] my own task or dataset: (give details below)
  Ran the example given in the documentation


        from transformers import ElectraTokenizer, ElectraForTokenClassification  
        import torch  

        tokenizer = ElectraTokenizer.from_pretrained('google/electra-small-discriminator')  
        model = ElectraForTokenClassification.from_pretrained('google/electra-small-discriminator')  

        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute",   
        add_special_tokens=True)).unsqueeze(0)  # Batch size 1  
        labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1  
        outputs = model(input_ids, labels=labels)  

        loss, scores = outputs[:2]
## To reproduce

Steps to reproduce the behavior:

1. Start Google Colab
2. Install Transformer and  requriements
3. run the code

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
AttributeError: 'ElectraForTokenClassification' object has no attribute 'num_labels'
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: Windows 10
- Python version: 3.7.6
- PyTorch version (GPU?): Colab
- Tensorflow version (GPU?): Colab
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Oh no, it should be `self.config.num_labels` instead of `self.num_labels` [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_electra.py#L665)!

My bad, it should be fixed now!
Can't update the train_batch_size and eval_batch_size for the training image in a docker container
Author: tbs17
Pull request: no

Hi,

I tried to train a couple multi-label models with the fast-bert libary using the container files to build up the docker image and uploaded to AWS ECR and used the aws helper notebook that's included in the 'sample notebook' folder in the repo. I have trained for 3 models and **regardless I changed to a different train_batch_size in the hyperparameters.json file, the model when training still outputs total train batch size is 64 and eval batch size is 128.**

**My question here:** 

-  Am i not able to update the train batch size?  if the training is happening in a container? 

-  Does the training and eval batch size have some relationship? from a glance, it looks like eval_batch_size is doubled train_batch_size. I'm gonna say there shouldn't be any relationship. However, **why there's no parameter set in the hyperparameters.json to specify the eval_batch_size?**

-  The three models i have trained all got really good accuracy_thresh with above 0.97. **However, one of the models only outputs 2 classes as the top probability class.** The original data has about 9455 rows and 113 classes. I have also trained it on BERT tensorflow version, i was able to get multiple labels as top predicted class. **what could be possibly wrong?** Note that my other 2 models has about 36 classes and 11 classes. The top predicted classes all came out reasonable, meaning all the 36 and 11 classes showed up for the top predicted class.  In addition, i don't see the performance change whenever i changed the accuracy_thresh change after epoch 2.

**Please provide some guidance as this is going into deployment soon** but I'm still struggling to figure out why....
    



Comments:
Updating the TensorFlow models to work as expected with tokenizers v3.0.0
Author: LysandreJik
Pull request: yes

Models and tokenizers should work in harmony; this is why it is an API design choice to be able to send the output of `encode_plus` and `batch_encode_plus` straight to the model, in both PyTorch and TensorFlow:

```py
encoded_sequence = tokenizer.encode_plus(sequence)
model(encoded_sequence)  # for TensorFlow
model(**encoded_sequence)  # for PyTorch
```

With the recent changes of tokenizers-v3.0.0 and the introduction of `BatchEncoding`, the way the TensorFlow models usually identified such inputs didn't work, as it was looking for a `dict` instead of a `BatchEncoding`. This PR patches this.

This feature was previously untested; this PR addresses this by adding four different tests on each tokenizers; testing that the tokenizers return correct `BatchEncoding` objects in both PyTorch and TensorFlow which can be fed directly to the model. Both `encode_plus` and `batch_encode_plus` are tested.

Some issues were found, and were patched with this PR.

Comments:
Summarization pipeline - Couldn't reach server at 'https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-cnn/modelcard.json'
Author: metahgva
Pull request: no

# 🐛 Bug

## Information
Loading the summarization pipeline will result in below assertion: 

> Couldn't reach server at 'https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-cnn/modelcard.json' to download model card file.
> Creating an empty model card.


The problem arises when using:
* [V] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
`!pip install -q transformers --upgrade
from transformers import pipeline
summarizer = pipeline(task="summarization")`

1. Execute above code


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
@julien-c This is issue has been closed, but I continue to get the same exact error. Can you reopen it? Or do I have to open a new issue?

Did you update from master. 

Can you paste the output of `transformers-cli env`

I just upgraded to the latest version:
```
- `transformers` version: 2.8.0
- Platform: Linux-5.3.0-46-generic-x86_64-with-Ubuntu-19.10-eoan
- Python version: 3.7.5
- PyTorch version (GPU?): 1.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```

The fix is not in a released version yet so you need to install from source.
Question Answering with Japanese
Author: Mukei
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Hi @singletongue,

I am trying to use Question-Answering for Japanese, however I could not find any model trained for that. 
I tried with the available models but the results were way off (as expected...).
Any suggestions on available models, or other library that already handle QnA with Japanese?

If it is supposed to work as-is, could you share a simple example?

Thank you in advance!

Comments:
Hi @Mukei,

As far as I know, there is no Transformer-based model fine-tuned for Japanese question answering tasks.
It is partly due to the scarcity of Japanese QA datasets (like SQuAD) to train the models on.

(Of course, we do wish to release models for QA, and it is left for our future work.)

As a workaround you could load the [bert-base-japanese](https://huggingface.co/bert-base-japanese) weights for the BertForQuestionAnswering model and just finetune the qa_outputs layer (in case of a single span prediction task). It will be quickly trained and maybe produces already sufficient results. 

@singletongue Thank you for your reply!

You might already know about it, but I found this [project](https://github.com/AkariAsai/extractive_rc_by_runtime_mt) with SQuAD V1.1 partially translated to Japanese: [Context](https://github.com/AkariAsai/extractive_rc_by_runtime_mt/blob/master/data/ja_question_v5_context.csv), [QA](https://github.com/AkariAsai/extractive_rc_by_runtime_mt/blob/master/data/ja_question_v5.csv)

@cronoik Thank you for the advice.
I tried but unfortunately the results were pretty bad even for some simple phrase.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
🐛 Summarization pipeline : T5-base much slower than BART-large
Author: astariul
Pull request: no
Labels: name: Core: Pipeline

# 🐛 Bug

## Information

Model : `bart-large-cnn` and `t5-base`

Language : English

The problem arises when using : [this colab notebook](https://colab.research.google.com/drive/1iAIFX1QQiFm1F01vMmnAgFh4oH1H-K8W), using both BART and T5 with pipeline for Summarization.

Dataset : CNN/DM

## To reproduce

Run the notebook and measure time for inference between the 2 models. On my run, I have :

```
BART = 73s
T5 = 369s
```

## Expected behavior

I expected T5 to be at least as fast as BART, since there is less parameters (for the base version at least). Instead it takes much longer with T5...

@patrickvonplaten Do you happen to know why T5 is so slow ?

Comments:
Hi @Colanim, thanks a lot for your speed comparison :-). 

It might be possible that the pipelines used different default parameters for `T5` and `Bart` under the hood which strongly influence their running times.
Besides `min_length` and `max_length` could you also insert those parameters into both `T5` and `Bart` to overwrite the default parameters: 

```
      "early_stopping": True
      "length_penalty": 2.0
      "no_repeat_ngram_size": 3
      "num_beams": 4
```

If there is still a big difference in time, then I guess we have to take a closer look! 



Thanks for your fast answer @patrickvonplaten 

Here is the link to the modified notebook, with the parameters you mentioned :
https://colab.research.google.com/drive/1kCm5ew8qDQqguZjbsC6Ujs9KZBaSfafi

---

Unfortunately, there is still a **huge** difference...

```
BART = 66s
T5 = 226s
```

Ok, good to know! thanks for doing the comparison @Colanim. This might interest you as well @sshleifer :-) 

Oh actually I just remember that Bart caches the decoder hidden key/value outputs when doing auto-regressive decoding (similar to GPT2 - check Visuals under "GPT-2 Masked Self-Attention" in this [post](http://jalammar.github.io/illustrated-gpt2/)) and I think T5 does not. 

But T5 could cache the decoder key/value outputs to speed up decoding as well since it uses a causal mask for the decoder. This could definitely be a Feature Request. What do you think
@sshleifer @craffel @thomwolf ?

Sounds worth it!
Typo in warning message
Author: elgeish
Pull request: yes

`T5Tokenizer` instead of `XLNetTokenizer`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3219?src=pr&el=h1) Report
> Merging [#3219](https://codecov.io/gh/huggingface/transformers/pull/3219?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d6de6423baf02a971d38ee69824104a1f0f85ad2&el=desc) will **not change** coverage by `%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3219/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3219?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3219   +/-   ##
=======================================
  Coverage   78.14%   78.14%           
=======================================
  Files          98       98           
  Lines       16668    16668           
=======================================
  Hits        13026    13026           
  Misses       3642     3642           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3219?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3219/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `95.83% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3219/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.54% <0.00%> (-0.20%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3219/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `94.56% <0.00%> (+0.15%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3219?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3219?src=pr&el=footer). Last update [d6de642...0a77ca6](https://codecov.io/gh/huggingface/transformers/pull/3219?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
cased -> uncased in BERT GLUE example
Author: boy2000-007man
Pull request: no

similar to https://github.com/huggingface/transformers/issues/3183, the GLUE readme also have this issue, the MRPC example use `bert-base-cased` while at the same time `--do_lower_case`.

Comments:
additionally, the `xlnet-large-cased` should not combine with `--do_lower_case` as `xlnet` model only has a cased version.
cased -> uncased for example cmdline consistency
Author: mattolson93
Pull request: yes


Comments:
In the same file, there is also `--do_lower_case` wrongly apply to `roberta-base` and `xlnet-large-cased` model. It will be great if this PR also includes that fix.

> In the same file, there is also `--do_lower_case` wrongly apply to `roberta-base` and `xlnet-large-cased` model. It will be great if this PR also includes that fix.

The intention is to have consistency across all the experiments, where all input is lowercase for all examples. Just because roberta and xlnet don't have lowercased models, doesn't mean their input has to be the cased version.

Thanks for raising this. Should be fixed by #3738

if one saves their tokenizers using `save_pretrained()` there shouldn't be a need for passing the `do_lower_case` manually anymore.
Fine tuning XLMRoberta for Question Answering
Author: wasiahmad
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I am trying to fine-tune XLM Roberta for SQuAD using the run_squad.py script available in examples. I have simply written an `XLMRobertaForQuestionAnswering` class as suggested at https://github.com/huggingface/transformers/issues/3694. However, the performance is extremely poor. I am wondering, do I need to perform anything special during preprocessing the SQuAD dataset when I use XLMRoberta?

I am using all the default hyper-parameters provided in the example script.

Comments: