noamaanMulla-03/transformers-issues huggingface dataset:
Fix QA models binding for Flaubert, XLNet and XLM.
Author: mfuntowicz
Pull request: yes

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Fix #2893

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3100?src=pr&el=h1) Report
> Merging [#3100](https://codecov.io/gh/huggingface/transformers/pull/3100?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/eec5ec807135ae61fa2266f3c7ad947cc207abda?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3100/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3100?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3100      +/-   ##
==========================================
- Coverage   77.59%   77.59%   -0.01%     
==========================================
  Files          98       98              
  Lines       16250    16250              
==========================================
- Hits        12610    12609       -1     
- Misses       3640     3641       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3100?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/3100/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `75.47% <ø> (ø)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3100/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.86% <0%> (-0.16%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3100?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3100?src=pr&el=footer). Last update [eec5ec8...44215a4](https://codecov.io/gh/huggingface/transformers/pull/3100?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Pipelines: add PoS support
Author: arnaudmiribel
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

## 🚀 Feature

As `Pipelines` were recently added for many tasks including NER, Sentiment Analysis, it'd be great to also enable Part-of-Speech tagging.

## Motivation

PoS tagging is a very useful task, and often used as an evaluating downstream task for new models.

## Additional context

Current available tasks for `Pipelines` are described [here](https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py#L831).


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


We now have a more general `TokenClassificationPipeline`, @arnaudmiribel (this is just an alias to the `NerPipeline`)
[Bart] _prepare_decoder_inputs should use large negative
Author: sshleifer
Pull request: yes

Also renames some things and adds a nice test.
I suspect that this didn't break integration tests because we don't have a serious integration test with decoder_input_ids set (e.g. calculating loss for a summarization example)

Comments:
Thanks @tomhosking for noticing!

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3158?src=pr&el=h1) Report
> Merging [#3158](https://codecov.io/gh/huggingface/transformers/pull/3158?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6ffe03a0a1d472a4e5941793fd361d2b82c8be3f?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3158/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3158?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3158      +/-   ##
==========================================
+ Coverage   78.11%   78.12%   +<.01%     
==========================================
  Files          98       98              
  Lines       16651    16651              
==========================================
+ Hits        13007    13008       +1     
+ Misses       3644     3643       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3158?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/3158/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `93.57% <100%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3158?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3158?src=pr&el=footer). Last update [6ffe03a...71e626b](https://codecov.io/gh/huggingface/transformers/pull/3158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Remove excess line breaks in DeepPavlov model cards
Author: yoptar
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3139?src=pr&el=h1) Report
> Merging [#3139](https://codecov.io/gh/huggingface/transformers/pull/3139?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8a2d9bc9ef38452e80ce872505a5ad5623c12657?src=pr&el=desc) will **decrease** coverage by `0.51%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3139/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3139?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3139      +/-   ##
==========================================
- Coverage   78.45%   77.94%   -0.52%     
==========================================
  Files          98       98              
  Lines       16561    16561              
==========================================
- Hits        12993    12908      -85     
- Misses       3568     3653      +85
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3139?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/3139/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `28.89% <0%> (-27.6%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3139?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3139?src=pr&el=footer). Last update [8a2d9bc...129d1ab](https://codecov.io/gh/huggingface/transformers/pull/3139?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[model_cards]Add albert chinese model
Author: voidful
Pull request: yes
Labels: name: model card

Hi,

this PR adds the model card for albert-chinese model
- albert_chinese_tiny
- albert_chinese_small
- albert_chinese_base
- albert_chinese_large
- albert_chinese_xlarge
- albert_chinese_xxlarge


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3157?src=pr&el=h1) Report
> Merging [#3157](https://codecov.io/gh/huggingface/transformers/pull/3157?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6ffe03a0a1d472a4e5941793fd361d2b82c8be3f?src=pr&el=desc) will **decrease** coverage by `0.15%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3157/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3157?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3157      +/-   ##
==========================================
- Coverage   78.11%   77.96%   -0.16%     
==========================================
  Files          98       98              
  Lines       16651    16651              
==========================================
- Hits        13007    12982      -25     
- Misses       3644     3669      +25
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3157?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3157/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.35% <0%> (-5.23%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3157/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `94.56% <0%> (+0.15%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3157?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3157?src=pr&el=footer). Last update [6ffe03a...491bea5](https://codecov.io/gh/huggingface/transformers/pull/3157?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for sharing. I'll just switch the language tag to chinese (tags are case-sensitive)

Model pages:
[`voidful/albert_chinese_tiny`](https://huggingface.co/voidful/albert_chinese_tiny)
[`voidful/albert_chinese_small`](https://huggingface.co/voidful/albert_chinese_small)
[`voidful/albert_chinese_base`](https://huggingface.co/voidful/albert_chinese_base)
[`voidful/albert_chinese_large`](https://huggingface.co/voidful/albert_chinese_large)
[`voidful/albert_chinese_xlarge`](https://huggingface.co/voidful/albert_chinese_xlarge)
[`voidful/albert_chinese_xxlarge`](https://huggingface.co/voidful/albert_chinese_xxlarge)
Change back pipeline signatures
Author: LysandreJik
Pull request: yes

As discussed with @julien-c in the merged #3055.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3105?src=pr&el=h1) Report
> Merging [#3105](https://codecov.io/gh/huggingface/transformers/pull/3105?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b31f7150190cdf13950607f8ee1efe11b352c909?src=pr&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3105/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3105?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3105      +/-   ##
==========================================
+ Coverage    77.6%   77.62%   +0.02%     
==========================================
  Files          98       98              
  Lines       16250    16230      -20     
==========================================
- Hits        12611    12599      -12     
+ Misses       3639     3631       -8
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3105?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/3105/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `70.95% <ø> (+0.52%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3105?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3105?src=pr&el=footer). Last update [b31f715...da52b39](https://codecov.io/gh/huggingface/transformers/pull/3105?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Classification of sentence pair with two different languages
Author: chiragsanghvi10
Pull request: no
Labels: name: wontfix

I have been working on multi-lingual sentence similarity (English-Hindi)

### for example:

> Sentence 1 (English)

> Sentence 2 (Translation in Hindi of Sentence 1)

> Sentence Similarity Score.

Any idea on how do I train for sentence similarity using `xlm-mlm-xnli15-1024`?



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Do we have a whole-word-masked version of BERT?
Author: ridiculouz
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Thanks for the excellent work! But I'm wonder is there a whole-word-masked version of BERT? Moreover, how can I adapt the Tokenizer class to make it support other parsing method (e.g. In Chinese the BERTTokenizer simply parses the sequences in character level, while parser like jieba can parse them into Chinese words. How can I keep the features of Tokenizer class while using other parsing method?)?

Comments:
Yes, there are a few available on the hub, you can search for [whole word masking](https://huggingface.co/models?search=whole-word-masking) or [wwm](https://huggingface.co/models?search=wwm).

For your second question, you would have a better answer if you opened an issue on [huggingface/toeknizers](https://github.com/huggingface/tokenizers/issues) instead.
Quick tour TF 2.0 
Author: celsofranssa
Pull request: no

# 🐛 Bug

## Information

I am trying **Quick tour TF 2.0**.

The problem arises when using a quick example: **How a TensorFlow 2.0 model can be trained in 12 lines of code**:

```python
import tensorflow as tf
import tensorflow_datasets
from transformers import *

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/mrpc')

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)
```
which produces the output below:
```python
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-4-ea94e17f2f79> in <module>()
      1 tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
----> 2 model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
      3 data = tensorflow_datasets.load('glue/mrpc')

NameError: name 'TFBertForSequenceClassification' is not defined
```



The above behavior can be reproduced using this [Colab ](https://colab.research.google.com/drive/1aAmOVlvkuP9PLOuGKVx7-k0vsVBoD506)notebook.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Any help will be much appreciated.


Comments:
Our models are TensorFlow 2.x only, while your notebook is in TensorFlow 1.x:

```
The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.
We recommend you upgrade now or ensure your notebook will continue to use TensorFlow 1.x via the %tensorflow_version 1.x magic: more info.
```

You can use the following command at the start of your notebook to use TensorFlow 2.x:

```
%tensorflow_version 2.x
```

@LysandreJik thank you.
LM Fine-tuning for XLNET?
Author: kevinmandich
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi there, is there any plan to add support for fine-tuning XLNET? This one currently isn't available in run_lm_finetuning.py.

Thanks!

Comments:
I'm also looking forward to it! 

I might be naive here, but will adding xlnet's config, LMHeadModel, and tokenizer to MODEL_CLASSES work?
https://github.com/huggingface/transformers/blob/74ce8de7d8e0375a9123f9542f3483f46cc8df9b/examples/run_lm_finetuning.py#L61

I don't think adding the Python classes for XLNet is enough. The code distinguishes between two modes of fine-tuning a language model: 1. masked language models (MLM), i.e. BERT and company 2. Sequential left-to-right LM, GPT and GPT2. 

Check this, for example:
https://github.com/huggingface/transformers/blob/74ce8de7d8e0375a9123f9542f3483f46cc8df9b/examples/run_lm_finetuning.py#L224

XLNet is neither. Its LM task is to predict factorization permutations (check paper for more details) of a given sequence. In simple words, it splits in a sequence into two sequential sub-sequences in all possible ways and it uses the first one as input for predicting the second one.

The original implementation has sample code on how to produce the factorization permutation:
https://github.com/zihangdai/xlnet/blob/5cd50bc451436e188a8e7fea15358d5a8c916b72/data_utils.py#L579

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Pretty much looking forward to it
Why cosine similarity of BERT, ALBERT, Robert is so big, almost near 1.0?
Author: lowoodz
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I tried to use bert models to do similarity comparison of words/sentences, but I found that the cosine similarities are all very high, even for very different words/sentences in meaning. Why?

Does all the vector are located in a small portion the vector-space?

Comments:
BERT was not designed to produce useful word / sentence embeddings that can be used with cosine similarities. Cosine-similarity treats all dimensions equally which puts high requirements for the created embeddings.

BERT as not intended for this. See this post by Jacob Devlin:
https://github.com/UKPLab/sentence-transformers/issues/80#issuecomment-565388257

If you want to use BERT with cosine similarities, you need to fine-tune it on suitable data. You can find data, code and examples in our repository:
https://github.com/UKPLab/sentence-transformers

@nreimers I have read your paper, it's great and thanks for the answer!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
finetune transformer
Author: jackfeinmann5
Pull request: no
Labels: name: wontfix

Hi
I greatly appreciate showing me how to pretrain a transformer model like BERT, I mean not finetuning but pretraining, Is there any code in your repo doing this? thanks a lot for your help. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Updated `Tokenw   ise` in print statement to `Token wise`
Author: param087
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3175?src=pr&el=h1) Report
> Merging [#3175](https://codecov.io/gh/huggingface/transformers/pull/3175?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e03129ad447ad7670fcc6206e5eb27a5435d4d86?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3175/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3175?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3175   +/-   ##
=======================================
  Coverage   78.15%   78.15%           
=======================================
  Files          98       98           
  Lines       16641    16641           
=======================================
  Hits        13006    13006           
  Misses       3635     3635
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3175?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3175?src=pr&el=footer). Last update [e03129a...70d11c4](https://codecov.io/gh/huggingface/transformers/pull/3175?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).