noamaanMulla-03/transformers-issues huggingface dataset:
multitask learning 
Author: antgr
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I would like to apply multitask learning, using for example two tasks, and the one (and maybe both) of the two tasks is sequential labeling like NER. To my understanding, in order to apply the library on such a task, the way to go is with BertForTokenClassification. Is that correct? But also what I think is that I do not have enough flexibility to use it/ adapt it to create a multitask model. 

Could you share your thoughts on that? Any help would be much appreciated.


Comments:
from the documentation of the class,
https://github.com/huggingface/transformers/blob/933841d903a032d93b5100220dc72db9d1283eca/pytorch_transformers/modeling_bert.py#L1100

I understand that I could use the ```scores```,  as input to an additional module that I could stack on top of BertForTokenClassification, for example for a second task. Is that correct? 

```loss, scores = outputs[:2] ```

But what I am thinking right now, is that scores could have small dimensions, so probably I would need the weights of the last layer. How could I extract them?

Always your thoughts on that would be much appreciated!

for every other suffered person that  needs an answer on that last question above, I think that the way to extract those weights, is if you open the black box of the implementation of this class, and here it is what you want:
outputs = self.bert(..)
so I think that reimplementation/enhancment is needed to support my needs as was given above.
Am I missing something if I reimplement this class adding more functionality? I think that no, and that it is safe. 

Hi, there are several things you can do to obtain the last layer representation. 

- First of all, you can use a standard `BertModel` on which you add your own classifier for token classification (that's what's done with `BertForTokenClassification`). This will allow you to easily switch the heads for your multi-task setup.

- You could also use the `BertForTokenClassification`, as you have said, and use the inner model (model.bert) to obtain the last layer.

- Finally, the cleanest way would be to output hidden states directly by specifying the option in the configuration:

```py
from transformers import BertConfig, BertForTokenClassification
import torch

config = BertConfig.from_pretrained("bert-base-cased")
config.output_hidden_states = True

model = BertForTokenClassification.from_pretrained("bert-base-cased", config=config)

inputs = torch.tensor([[1, 2, 3]])

outputs = model(inputs)
token_classification_outputs, hidden_states = outputs

last_layer_hidden_states = hidden_states[-1]
```

The variable `last_layer_hidden_states` is of shape `[batch_size, seq_len, hidden_size]` and is the output of the last transformer layer.

I hope this clears things up.

https://github.com/jiant-dev/jiant
Why is `encoder_extended_attention_mask = None` when `config.is_decoder == False`
Author: UsmannK
Pull request: no

Potential Bug(?)

Reading the codebase I see that attention masks are ignored for many of the pretrained model configs such as `'bert-base-uncased'`. We can see [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L743) that the attention mask is simply cleared out. Is this intentional? 

```
from transformers import BertModel
config_path = 'bert-base-uncased'
config = BertModel.config_class.from_pretrained(config_path)
print(f'is_decoder: {config.is_decoder}')
```

outputs False

Comments:
The `encoder_attention_mask` is only relevant if BERT is uses as a Encoder-Decoder model using the `EncoderDecoderModel` wrapper class. In this case the decoder should be able to accept an `encoder_attention_mask` for its cross-attention layers. 

In all other cases this mask is not relevant and should be set to None. 

I agree that the check `if self.is_decoder` is probably not the best one here it should rather be `if self.is_encoder_decoder and self.is_decoder`. will update this soon.

Feel free to reopen if this does not answer your question

Hi Patrick, thanks for the swift response. I’m not sure if I understand: shouldn’t we always want to mask the padded tokens, even in the encoder?

In fact the canonical BERT model suggests this, where they have no such check: https://github.com/google-research/bert/blob/master/modeling.py#L200

@patrickvonplaten Sorry for the noise. Noticed you said to reopen the issue but I think only maintainers have this permission :)

This `encoder_attention_mask` is only relevent for a Bert EncoderDecoder model. It is not the same as the usual `attention_mask`

Ah, I see. Looking again at the code I definitely misunderstood that. Thanks a ton.
Can't pickle tokenizers ...
Author: ohmeow
Pull request: no

Tokenizers are losing all their special tokens when un-pickled.  Not sure if there are other attributes that aren't being rehydrated as well ...

```
hf_tokenizer
# <transformers.tokenization_roberta.RobertaTokenizer at 0x7fc4ce625f10>

hf_tokenizer.special_tokens_map
# {'bos_token': '
',
# 'eos_token': '
',
# 'unk_token': '<unk>',
# 'sep_token': '
',
# 'pad_token': '<pad>',
# 'cls_token': '
',
# 'mask_token': '<mask>'}

pickle.dump( hf_tokenizer, open( "save.p", "wb" ) )
hf_tokenizer = pickle.load( open( "save.p", "rb" ) )

hf_tokenizer.special_tokens_map
# {'bos_token': '',
# 'eos_token': '',
# 'unk_token': '',
# 'sep_token': '',
# 'pad_token': '',
# 'cls_token': '',
# 'mask_token': ''}

```

Comments:
Hi, ok I can reproduce, this is a bug in the `AddedToken`class of `huggingface/tokenizers`. Moving this up.
[ERROR] Tokenizer and TokenizerFast ??? 
Author: 1512262
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): 'bert-base-multilingual-cased'

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x] my own task or dataset: (give details below)
## To reproduce

Steps to reproduce the behavior:

1. `from transformers import *`
2. `tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-cased')`
3. `tokenizer.decode(tokenizer.encode('mở bài lạc trôi'))` --> wrong 


but:

1. `from transformers import *`
2. `tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')`
3. `tokenizer.decode(tokenizer.encode('mở bài lạc trôi'))` --> true

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
the decode sentence after encoding and decoding using TokenizerFast should be true
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Pytorch and TF
- Python version: 3.6
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?): 2.2
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
This is related to https://github.com/huggingface/transformers/issues/2917. In the slow tokenizers, when `do_lower_case=False` we don't strip accents, while we do it when `do_lower_case=True`. In the fast tokenizers, this is controlled by the `strip_accents` option, which is `True` here.

@thomwolf How do you think we should fix this?

Yes let's do it @n1t0  and stick to the official bert tokenizer behavior in the fast tokenizers as well.
Quantized model not preserved when imported using from_pretrained()
Author: trevorpfiz
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
When I import the saved quantized model using `from_pretrained()`, the model's size is inflated to the pre-quantized version.  The model also takes a significant performance hit, both accuracy and time, from the original quantized model.

The tasks I am working on is:
This is the official Pytorch notebook from: https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html

https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/dynamic_quantization_bert_tutorial.ipynb

## To Reproduce

Steps to reproduce the behavior:

1.  Run through the notebook fully.
2.  Load the quantized model with from_pretrained()
3.  Run `3.1 Check the model size` and `3.2 Evaluate the inference accuracy and time`

You will see that the size is of the pre-quantized model (>400 MB), and the accuracy AND time take a huge hit from the original quantized model.

## Expected behavior

1.  The quantized model can be loaded in at its original size of <200 MB
2.  The quantized model preserves its accuracy when loaded in
3.  The quantized model preserves its time to run when loaded in

## Environment

Colab


Comments:
@LysandreJik any ideas on this?  I am itching to use a quantized BERT model in production, but it does not work when loaded in :(

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi @ElektrikSpark , I am also facing similar issue, were you able to resolve this? 
I am unable to get good results while loading the quantized bert model.
Create model card
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5523?src=pr&el=h1) Report
> Merging [#5523](https://codecov.io/gh/huggingface/transformers/pull/5523?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/58cca47c16149e43d1b516623d59e3c5d97f695e&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5523/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5523?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5523      +/-   ##
==========================================
+ Coverage   77.83%   77.85%   +0.02%     
==========================================
  Files         141      141              
  Lines       24634    24634              
==========================================
+ Hits        19175    19180       +5     
+ Misses       5459     5454       -5     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5523?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5523/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+1.25%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5523?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5523?src=pr&el=footer). Last update [58cca47...0780d53](https://codecov.io/gh/huggingface/transformers/pull/5523?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Create model card
Author: mrm8488
Pull request: yes
Labels: name: model card

Create model card for electra-small-discriminator finetuned on SQUAD v1.1

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5430?src=pr&el=h1) Report
> Merging [#5430](https://codecov.io/gh/huggingface/transformers/pull/5430?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d60d231ea497aa2ed46226f51e360b207a79682e&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5430/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5430?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5430      +/-   ##
==========================================
- Coverage   77.61%   77.60%   -0.01%     
==========================================
  Files         140      140              
  Lines       24343    24343              
==========================================
- Hits        18893    18892       -1     
- Misses       5450     5451       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5430?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5430/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `79.26% <0.00%> (-0.34%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5430?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5430?src=pr&el=footer). Last update [d60d231...e3436bf](https://codecov.io/gh/huggingface/transformers/pull/5430?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
BertEmbeddings code for position_embeddings and word_embeddings
Author: shampp
Pull request: no

   I am trying to figure out how exactly positions and words are embedded in Bert.  I checked the code, and it seems following code is used for the purpose.   This implies that both the embedding function are calling same `__init__' function from the class `nn.Embedding`.  Where exactly is the code doing the actual position embedding (sine and cosine based). Where can I see that code ?.  

    self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
    self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
    inputs_embeds = self.word_embeddings(input_ids)
    position_embeddings = self.position_embeddings(position_ids)`

Comments:
Hi, BERT uses absolute positional embeddings, not based on sinusoidal functions. The positional embeddings are learned. 

You can check the [XLNet code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_xlnet.py#L704-L712) or the [TransfoXL code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_transfo_xl.py#L170-L186) for examples showcasing sinusoidal (relative) embeddings.
[Bart] enable test_torchscript, update test_tie_weights
Author: sshleifer
Pull request: yes

This sets `test_torchscript=True` for BART and removes unneeded asserts in `test_tie_weights`.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5457?src=pr&el=h1) Report
> Merging [#5457](https://codecov.io/gh/huggingface/transformers/pull/5457?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/306f1a269504b781f886d75105acabf8ae95bd11&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5457/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5457?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5457      +/-   ##
==========================================
- Coverage   77.86%   77.57%   -0.29%     
==========================================
  Files         141      141              
  Lines       24608    24608              
==========================================
- Hits        19160    19089      -71     
- Misses       5448     5519      +71     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5457?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5457/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxuZXQucHk=) | `32.43% <0.00%> (-55.86%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5457/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `66.25% <0.00%> (-32.52%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5457/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.18% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5457/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `79.59% <0.00%> (+0.33%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/5457/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `98.37% <0.00%> (+25.00%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5457?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5457?src=pr&el=footer). Last update [306f1a2...e08f8b7](https://codecov.io/gh/huggingface/transformers/pull/5457?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
xlm-mlm-17-1280: after run model to get embeddings shape 20000
Author: vvssttkk
Pull request: no

i want to get embeddings for :ru: text by `xlm-mlm-17-1280`
at the end get embeddings with shape 2k

example code (using the last version transformers & torch on ubuntu):
```
from transformers import AutoTokenizer, AutoModelWithLMHead

xlm_mlm = 'xlm-mlm-17-1280'
tokenizer_xlm_mlm = AutoTokenizer.from_pretrained(xlm_mlm)
model_xlm_mlm =  AutoModelWithLMHead.from_pretrained(xlm_mlm) 

input_ids = torch.tensor([tokenizer_xlm_mlm.encode(my_input_text)]) # batch size of 1
print(f'{input_ids.shape=}')
# input_ids.shape=torch.Size([1, 373])

lang_id_ru = tokenizer_xlm_mlm.lang2id['ru']

langs_ru = torch.tensor([lang_id_ru] * input_ids.shape[1])  # torch.tensor([0, 0, 0, ..., 0])
print(f'{langs_ru.shape=}')
# langs_ru.shape=torch.Size([373])

langs_ru = langs_ru.view(1, -1) # is now of shape [1, sequence_length]
print(f'{langs_ru.shape=}')
# langs_ru.shape=torch.Size([1, 373])

outputs = model_xlm_mlm(input_ids, langs=langs_ru)
outputs[0].shape
# torch.Size([1, 373, 200000])
```

so it's a bug or my bad?

Comments:
hmm, i found `(proj): Linear(in_features=1280, out_features=200000, bias=True))`
why `out_features = 200000`?

That's the vocab size, which is of size 200 000. You can see it in the [config.json](https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-17-1280-config.json).

it's true, my bad
and how i can get embeddings 1280 for input text? 

@LysandreJik maybe u can help, how i can get embeddings to sentence after run `outputs = model_xlm_mlm(input_ids, langs=langs_ru)`
at the older version 2.3.0 the `outputs` had the last size <emb_dim> but no it's <vocab_size>

That's probably because you were using the `AutoModel` factory instead of `AutoModelWithLMHead`. The former returns the transformer embeddings of dimension `hidden_size` (1280 in your case), while the latter returns the projected embeddings on the vocabulary, of dimension `vocab_size` (200 000 in your case). 

Change the two lines:

```py
from transformers import AutoTokenizer, AutoModelWithLMHead

model_xlm_mlm =  AutoModelWithLMHead.from_pretrained(xlm_mlm)
```

to

```py
from transformers import AutoTokenizer, AutoModel

model_xlm_mlm =  AutoModel.from_pretrained(xlm_mlm)
```
Add "labels" functionality for all TF Causal LM and Masked LM models
Author: patrickvonplaten
Pull request: no

# 🚀 Feature request

Currently, it is not possible to calculate the loss of a CLM or MLM model using TF:

```python
from transformers import TFBertForMaskedLM, BertTokenizerFast

model = TFBertForMaskedLM.from_pretrained("bert-base-uncased")
tok = BertTokenizerFast.from_pretrained("bert-base-uncased")

input_ids = tok("This is a test string.").input_ids
loss = model(input_ids, labels=input_ids)[0]  # This is currently not possible
```
Currently the `call(...)` function does not accept a `labels` input and does not return the 
loss. This should be implemented similar to how it was done for `BertForSequenceClassification`:
https://github.com/huggingface/transformers/blob/393b8dc09a97197df1937a7e86c0c6b4ce69c7e9/src/transformers/modeling_tf_bert.py#L918

All CLM and MLM TF models should be updated to accept `labels` as an input and return the corresponding loss.

## Motivation

1. This allows to use TFTrainer for CLM and MLM models.
2. This aligns TF with PT API

## Your contribution

Starting from 29.06/30.06 I want to implement all these features (with some help from @jplu if possible ;-)) 

Pinging @LysandreJik @julien-c @sgugger for notification.


Comments:
Awesome! I'm in!!! 

Yes please! Happy to help if you need it.
🐛 Can't use `AutoTokenizer` with `sshleifer/mbart-large-cc25`
Author: astariul
Pull request: no

# 🐛 Bug

From [`sshleifer/mbart-large-cc25`](https://huggingface.co/sshleifer/mbart-large-cc25) :

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sshleifer/mbart-large-cc25")
```

---

Running this code yield an error :

>OSError: Model name 'sshleifer/mbart-large-cc25' was not found in tokenizers model name list (facebook/bart-large, facebook/bart-large-mnli, facebook/bart-large-cnn, facebook/bart-large-xsum). We assumed 'sshleifer/mbart-large-cc25' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.

---

Which Tokenizer should I use with this model ?

@sshleifer 

Comments:
Ha, I didn't know the Tokenizer between Bart and mBart is different. I just noticed there is a class `MBartTokenizer`.

It seems like this class is not documented on [HuggingFace documentation](https://huggingface.co/transformers/model_doc/bart.html). Maybe we should consider adding it ?

_Also the model card for `sshleifer/mbart-large-cc25` may need an update_

---

Also, the following is still not working :

```python
from transformers import MBartTokenizer

tokenizer = MBartTokenizer.from_pretrained("sshleifer/mbart-large-cc25")
```

Only `tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-en-ro')` seems to work.

**Can I use Tokenizer from checkpoint `facebook/mbart-large-en-ro` for the model `sshleifer/mbart-large-cc25` ?**

mbart-large-cc25 does not work well yet, the PR is still open #3513.
Nonetheless these are all things I should fix, thanks!

For your second question, no. At the moment that tokenizer will not work well. Do you have fairseq cc25 working well?

**Update:** just moved it to `facebook/mbart-large-cc25`. AutoTokenizer should work. 

I didn't try the fairseq model, went directly for HF implementation ^^