noamaanMulla-03/transformers-issues huggingface dataset:
Tokenizer methods and padding
Author: r0mainK
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I wanted to know whether there was a perticular reason why the `get_special_tokens_mask` method of the tokenizer does not also return as mask over padding tokens, only <CLS> and <SEP> tokens, in the case where `already_has_special_tokens=True` ? I had to rewrite a custom function for my usecase, but it seemed off.

Also, I think there should be an additional `padding` kwarg in the method, which if provided would return a longer mask then the sum of lenfgths of `token_ids_0` and `token_ids_1`, in the case where `already_has_special_tokens=False`. The same should be true for `build_inputs_with_special_tokens` IMO. What do yoy think ?


Comments:
I have no strong opinion about this. Wdyt @LysandreJik?

Related to this though, this is how I'm proposing to mask the padding tokens in Masked language modeling batches in the `run_lm_finetuning` script: https://github.com/huggingface/transformers/pull/2570/commits/55939b5707066f612b0b2390787b325d30af728c#diff-713f433a085810c3d63a417486e56a88R205-R206

Since you are already caching the encoded examples, I think you can do: `batch_encode_plus.(..., pad_to_max_length=True)` in both Dataset's `__init__`, instead of repeating this for each epoch. This will also get rid of the introduced `collate_fn` logic you introduce.

Regarding the issue, I just think it's surprising `get_special_tokens_mask` does not consider padding tokens as special tokens, requiring them to be handled separately, for instance as you did.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to start a server and client to get feature vectors
Author: duan348733684
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
How to start a server and client to get feature vectors，or Which part of the code should I study in https://github.com/huggingface/transformers.git.




Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add t5 to pipeline(task='summarization')
Author: patrickvonplaten
Pull request: yes

This PR:

- adds T5 to summarization piplines.
- adds warnings and better defaults to Bart/T5 summarization
- removes unnecessary assert in generate() function 



Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3413?src=pr&el=h1) Report
> Merging [#3413](https://codecov.io/gh/huggingface/transformers/pull/3413?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e392ba6938f50655a195ea7ec8a260b1e9fc6058&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `93.75%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3413/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3413?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3413      +/-   ##
==========================================
+ Coverage   77.56%   77.58%   +0.02%     
==========================================
  Files         100      100              
  Lines       16970    16993      +23     
==========================================
+ Hits        13162    13184      +22     
- Misses       3808     3809       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3413?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3413/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.71% <ø> (-0.02%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/3413/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `73.05% <93.10%> (+0.52%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3413/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `84.44% <100.00%> (+0.52%)` | :arrow_up: |
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3413/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `95.89% <100.00%> (+0.05%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3413?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3413?src=pr&el=footer). Last update [e392ba6...23778d1](https://codecov.io/gh/huggingface/transformers/pull/3413?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Update model card huseinzol05/bert-base-bahasa-cased
Author: huseinzol05
Pull request: yes
Labels: name: model card


Comments:
Hi @huseinzol05! can you rebase on master so that it's easy to merge?

@julien-c , done! thank u very much!

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3425?src=pr&el=h1) Report
> Merging [#3425](https://codecov.io/gh/huggingface/transformers/pull/3425?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9c683ef01e19c4dc1216dcd1ae3c8e7c44d7b2b9&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3425/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3425?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3425   +/-   ##
=======================================
  Coverage   77.76%   77.76%           
=======================================
  Files         100      100           
  Lines       16995    16995           
=======================================
+ Hits        13216    13217    +1     
+ Misses       3779     3778    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3425?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3425/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.32% <0.00%> (+0.17%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3425?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3425?src=pr&el=footer). Last update [9c683ef...33d0e10](https://codecov.io/gh/huggingface/transformers/pull/3425?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@julien-c , added xlnet-base README
run_lm_finetuning on multiple training files
Author: ghtaro
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

Hi,

I would like to fine-tune huggingface's pre-trained BERT model on a relatively big text data.
I split the data into multiple files (~10k files) in .raw format in the same folder.

First, I succeeded to run run_lm_finetuning.py on one of the raw files I generated.

Now, I would like to run that in my 10k raw files.
I realised that that train_data_file argument does not accept a folder name (only the path to a single raw file).
I believe that we can do a loop over the 10k train files and incrementally fine-tuning the model, but it does not look like a best solution for me...

Could you please tell me if it exists a simple way to achieve the above?

Thank you very much for your help.

Comments:
Hi, this is not currently supported, you would need to implement this yourself. Feel free to open a PR if you do
Adds translation pipeline
Author: patrickvonplaten
Pull request: yes

The API for translation is as follows: 

```
en_fr_translation = pipeline("translation_en_to_fr")
en_fr_translation("How old are you?")
```

for English to French translation. 

PR adds tests and gives an example in the docstring.
PR builds on #3413 and should be merged after this one.

Example:
![Screenshot from 2020-03-26 11-10-52](https://user-images.githubusercontent.com/23423619/77637172-899ea400-6f55-11ea-83a6-0a476fd33430.png)




Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3419?src=pr&el=h1) Report
> Merging [#3419](https://codecov.io/gh/huggingface/transformers/pull/3419?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9c683ef01e19c4dc1216dcd1ae3c8e7c44d7b2b9&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `94.44%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3419/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3419?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3419      +/-   ##
==========================================
+ Coverage   77.76%   77.80%   +0.03%     
==========================================
  Files         100      100              
  Lines       16995    17025      +30     
==========================================
+ Hits        13216    13246      +30     
  Misses       3779     3779              
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3419?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/3419/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `98.92% <ø> (ø)` | |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/3419/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `74.78% <94.44%> (+1.51%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3419/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.86% <0.00%> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3419?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3419?src=pr&el=footer). Last update [9c683ef...a5160a6](https://codecov.io/gh/huggingface/transformers/pull/3419?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
NER Pipeline returns null
Author: Realvincentyuan
Pull request: no

# 🐛 Bug

## Information

Model I am using (NER Pipeline):

Language I am using the model on (English):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import pipeline
# Allocate a pipeline for named entity recognition
nlp = pipeline('ner')
nlp(['We are very happy to include pipeline into the transformers repository.'])

```
**returns null.**

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Expect to have the named entity label for each token. But it returns null.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Jupyter lab
- Python version: 3.6.1
- PyTorch version (GPU?): CPU-1.4.0
- Tensorflow version (GPU?): CPU-1.15.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
By "it returns null" you mean it returns an empty array? That's because it didn't identify any named entity in your sequence.
Add missing token classification for XLM
Author: sakares
Pull request: yes

The current [modeling_xlm.py](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_xlm.py) did not have alike `ForTokenClassification` class like others, which helps for NER task comparison across all existing models.

now `XLMForTokenClassification` can be called via:
```python
from transformers import XLMForTokenClassification

model = XLMForTokenClassification.from_pretrained('xlm-mlm-100-1280')

```

Comments:
🐛Bugs in run_tf_ner.py
Author: jia-zhuang
Pull request: no

Found a bug in [run_tf_ner.py](https://github.com/huggingface/transformers/blob/master/examples/ner/run_tf_ner.py) at line 170 and 325:

```python
loss_fct = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE)
```
refer to [tf.keras.losses.SparseCategoricalCrossentropy](https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy)

```python
tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=False, reduction=losses_utils.ReductionV2.AUTO,
    name='sparse_categorical_crossentropy'
)
```

> `from_logits`: Whether y_pred is expected to be a logits tensor. By default, we assume that y_pred encodes a probability distribution. Note: Using from_logits=True may be more numerically stable.

So I think `loss_fct` should be init with `from_logits=True` if the `TFBertForTokenClassification` just return pure logits rather than softmax output.

Comments:
Hey @jia-zhuang,

As you can see [here](https://github.com/huggingface/transformers/blob/master/examples/ner/run_tf_ner.py#L525) I update the classification layer to add a softmax as activation, then the `from_logits=True` is not necessary.

@jplu Thanks for your reply! I learn a lot from your code.
Add t5 summarization example
Author: patrickvonplaten
Pull request: yes

Adds TF 2.0 Example for T5 summarization. 

Adds dataset download file via `tensorflow_datasets` and a rouge scorer. 


Example is currently being tested on T5-large on GPU to see how rouge scorer performs in comparsion to `examples/summarization/bart` rouge scorer.

Comments:
> pending my comments

Very much down to share the summarization code in another PR!

Code quality test fails because of unpinned isort library (see https://github.com/huggingface/transformers/pull/3449)
revert unpin isort commit
Author: patrickvonplaten
Pull request: yes

This PR reverts the change of 
https://github.com/huggingface/transformers/commit/fbc5bf10cfe4d4ca81f8daacc148b0abd51dda5a

Using the unpinned version of `isort` makes black and `isort` disagree in some cases. 
In this PR the unpinned version of `isort` leads to a falied code quality test:
https://github.com/huggingface/transformers/pull/3411 



Comments:
This syntax is not support by PyPi unfortunately, so we had to revert this.

> This syntax is not support by PyPi unfortunately, so we had to revert this.

I see...is there another way of  getting the pinned version? 
In general, do we need isort? Doesn't black also sort the import statements? 

Other ways are:
- bug the isort maintainer to release a version containing this commit
- push a new forked package to PyPI, like `isort-pvp` or `isort-black-compat` or whatever.

Yes we need isort.

I have also observed this issue with #3402
Add wmt translation example
Author: patrickvonplaten
Pull request: yes

PR adds translation example for T5. 
It uses the `sacrebleu` BLEU scorer. 

I adapted the README.md a bit so that users are aware that models in official paper were attained with finetuned T5 @craffel 

Comments:
> long

Yeah we should definitely try to run it on a GPU - will take a look at that :-) 

Not sure whether we need fp16 and multi-gpu training. I think single GPU training is enough and t5 + wmt does not take much memory. But happy to take a look into it if you guys think it's worth it :-) @thomwolf @LysandreJik @julien-c 

Code quality test fails because of unpinned isort library (see https://github.com/huggingface/transformers/pull/3449)

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3428?src=pr&el=h1) Report
> Merging [#3428](https://codecov.io/gh/huggingface/transformers/pull/3428?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b4fb94fe6d831b17c0df364b2848c80ef3add154?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3428/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3428?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3428   +/-   ##
=======================================
  Coverage   52.51%   52.51%           
=======================================
  Files         100      100           
  Lines       17051    17051           
=======================================
  Hits         8954     8954           
  Misses       8097     8097
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3428?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3428?src=pr&el=footer). Last update [b4fb94f...713524e](https://codecov.io/gh/huggingface/transformers/pull/3428?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Does loss  function in the run_tf_ner.py takes logits or probabilities?
Author: andrey999333
Pull request: no

The model used for ner model is TFBertForTokenClassification in case of Tensorflow. According to documentation this model produces logits for every token. But the loss function that is used in `run_tf_ner.py` is `tf.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE)` which has default argument `from_logits=False`. In this case the loss function expects probabilities but not the logits. So i would think that correct loss function should have additional argument `from_logits=True`. I'm building my own code using the `run_ner_tf.py` as an example, that is why i want to understand it. Am i missing something here? Because it looks like people can train the model and get some reasonable results...

Comments:
I think you are right, I also find this bug, see [🐛Bugs in run_tf_ner.py](https://github.com/huggingface/transformers/issues/3389)

Hey @andrey999333! I answered to the question just here https://github.com/huggingface/transformers/issues/3389

Hey @jplu Thanks a lot, i have not noticed that line for some reason. I was wandering coz my code was working fine with the flag and yours without. Now the mystery is solved :)
[Seq2Seq Generation] Call encoder before expanding input_ids
Author: sshleifer
Pull request: yes

Proposing to call model.encoder before expanding `input_ids` to `effective_batch_size*num_beams`.

For Bart, this saves 1.5 GB of GPU mem on batch_size=6. Savings probably similar for T5 (untested).

Requires knowing which index of the encoder_outputs is associated with the batch dim (we need to expand this dimension), which is different between `Bart` and `T5`. This difference is encoded in the `self.encoder_outputs_batch_idx` variable.


This PR is WIP because `encoder_outputs_batch_idx` could be avoided if we transposed Bart's encoder_outputs, which I haven't tried. 


Comments:
Like the change a lot! 
One question I asked myself: With this change the `encoder_outputs` which are the same point to the same memory address -> could that lead to problems? Probably not because the `encoder_outputs` are never changed, right? 

I'd just propose some renaming.
[Bart/Memory] SelfAttention only returns weights if config.output_attentions
Author: sshleifer
Pull request: yes

**Previously**, `SelfAttention` would always return `attn_weights`, and then `BartDecoder` and `BartEncoder` would decide whether to return them to the user.
The `attn_weights` tensor is fairly large, with shape = `(bs, num_heads, tgt_len, src_len)`
This meant that the memory allocated for `attn_weights` could not be freed until after the forward pass of `BartDecoder`.

Now: `SelfAttention` returns (output, None) if `config.output_attentions=False` and the memory can be freed 

Impact: memory can be freed after SelfAttention returns. -600MB peak GPU consumption for batch_size=6, tgt_len=src_len=1024, num_heads=16

Speed impact: negligible

Comments: