noamaanMulla-03/transformers-issues huggingface dataset:
Is summing of attention_mask intended?
Author: shaoyent
Pull request: no

# ❓ Questions & Help
Is summing of `attention_mask` intended?

## Details
Documents describe `attention_mask` as a mask to avoid performing attention on padding token indices.
However from the [code](https://github.com/huggingface/transformers/blob/2ffef0d0c7a6cfa5a59c4b883849321f66c79d62/src/transformers/modeling_bert.py#L243) in `BertSelfAttention` the attention mask is added to the scores.
```
attention_scores = attention_scores + attention_mask
```

Is this intended? Does this fully mask out the context?



Comments:
Yes, it's intended. The attention mask has values of 0 where its attending to the tokens, and has a value of `-10000` for tokens it's not attending. By summing this attention mask, it zeros out the attentions that should not be kept. See [here](https://github.com/huggingface/transformers/blob/2ffef0d0c7a6cfa5a59c4b883849321f66c79d62/src/transformers/modeling_utils.py#L228) for the implementation.

@LysandreJik  I found that `attention_mask` is simply computed by `encoded_inputs["attention_mask"] = [1] * len(encoded_inputs["input_ids"])` in [tokenization_utils_base.py](https://github.com/huggingface/transformers/blob/2ffef0d0c7a6cfa5a59c4b883849321f66c79d62/src/transformers/tokenization_utils_base.py#L1944). Should `attention_mask` be synchronized with `encoded_inputs["input_ids"]` when a [MASK] appears in the input?  For example,

Input: This is [MASK] pen.

Should the `attention_mask` be `1 1 0 1` rather than `1 1 1 1`?

The attention mask is computed according to the padding tokens, not masking tokens. See [here](https://github.com/huggingface/transformers/blob/2ffef0d0c7a6cfa5a59c4b883849321f66c79d62/src/transformers/tokenization_utils_base.py#L1932) for a sequence that requires padding.
Gpt2 model card
Author: sgugger
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5283?src=pr&el=h1) Report
> Merging [#5283](https://codecov.io/gh/huggingface/transformers/pull/5283?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0e1fce3c0129d05b65a83cdd89e8eadded553f2e&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5283/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5283?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5283      +/-   ##
==========================================
- Coverage   79.11%   79.09%   -0.02%     
==========================================
  Files         138      138              
  Lines       24080    24080              
==========================================
- Hits        19050    19046       -4     
- Misses       5030     5034       +4     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5283?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5283/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `79.51% <0.00%> (-1.39%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5283/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.62% <0.00%> (-0.15%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5283/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.26% <0.00%> (+0.12%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5283?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5283?src=pr&el=footer). Last update [0e1fce3...a1c3ed8](https://codecov.io/gh/huggingface/transformers/pull/5283?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


(@julien-c approved offline)
Added a model card README.md for my pretrained model.
Author: Pradhy729
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5325?src=pr&el=h1) Report
> Merging [#5325](https://codecov.io/gh/huggingface/transformers/pull/5325?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5543b30aa6b52da3c8f7d9e525b0edc26226d717&el=desc) will **increase** coverage by `0.47%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5325/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5325?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5325      +/-   ##
==========================================
+ Coverage   77.42%   77.90%   +0.47%     
==========================================
  Files         138      138              
  Lines       24284    24284              
==========================================
+ Hits        18803    18919     +116     
+ Misses       5481     5365     -116     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5325?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5325/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `66.25% <0.00%> (-32.52%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5325/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.48% <0.00%> (+1.32%)` | :arrow_up: |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/5325/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `87.50% <0.00%> (+58.65%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/5325/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `95.38% <0.00%> (+68.46%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5325?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5325?src=pr&el=footer). Last update [5543b30...523ec13](https://codecov.io/gh/huggingface/transformers/pull/5325?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Nice work. What's your intended use case, @Pradhy729? Topic classification? Something else?

Thanks! A few different use cases actually. Better entity recognition, topic classification, sentiment extraction and hopefully play around with summarization as well.
T5 Warning: embeddings are not initialized
Author: sshleifer
Pull request: no

```python
from transformers import *
model = T5ForConditionalGeneration.from_pretrained("t5-base")
```

Is this concerning?

```
Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at t5-base and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

Relatedly, I am porting a summarization checkpoint and wondering whether I should initialize lm_head.

Comments:
This [comment](https://github.com/huggingface/transformers/issues/3553#issuecomment-624306027) answers this issue.
XLMRobertaTokenizer vocab size
Author: ssdorsey
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

name: Should Fix

I think the XLMRobertaTokenizer vocab_size is off. Currently double counts ```'<unk>' | '
' | '
'```

Maybe change it to
```
def vocab_size(self):
    return len(self.sp_model) + self.fairseq_offset
```


Comments:
Running the following code caused error for me: 

> import transformers
> tokenizer = transformers.AutoTokenizer.from_pretrained("xlm-roberta-base")
> tokenizer.convert_ids_to_tokens(range(tokenizer.vocab_size))

Actually, the `tokenizer.vocab_size` is `250005`, the last id `250004` is `<mask>`, but the ids from `250001` to `250003` do not exist.

> Actually, the `tokenizer.vocab_size` is `250005`, the last id `250004` is `<mask>`, but the ids from `250001` to `250003` do not exist.

Ya ok this is definitely the problem. Either way, it's an issue for the current implementation of get_vocab which will crash at 25001:

```
    def get_vocab(self):
        vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
        vocab.update(self.added_tokens_encoder)
        return vocab
```


I wonder if this issue will be fixed? Currently it is not...

This issue is known and will be fixed.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This should have been fixed with https://github.com/huggingface/transformers/pull/3198
Upload model card with the CLI
Author: borisdayma
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Upload model card README.md with transformers CLI.

## Motivation

Some applications may require to upload multiple models. It would be nice to be able to upload associated model card with model/tokenizer/config files.

At the moment a PR is required making it more difficult for users to upload their models and nearly impossible for apps relying on training and uploading a lot of different models.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I could help propose a PR for the client side.


Comments:
This is not currently documented (cc @sgugger) but it's already possible, see for instance https://huggingface.co/Helsinki-NLP/opus-mt-fr-en

At some point it might become the preferred way of publishing a model card too.

Will update the tutorial today.

Thanks, this is great!
run_squad.py :: ValueError: Input [] is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.
Author: SarangSanjayGujar-lilly
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
[*] the official example scripts: (give details below)
[ ] my own modified scripts: (give details below)

The tasks I am working on is:
[*] an official GLUE/SQUaD task: (give the name)
[ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. while running run_squad.py 
2.  Training & testing with
https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Error
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 320, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
ValueError: Input [] is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.


## Expected behavior

To work

## Environment info
- `transformers` version: 2.11.0
- Platform: Linux-5.3.0-1017-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No



Comments:
Hi! Do you mind pasting the command you use to run the script?

I'm facing the same error while training Electra and MiniLM on Squad:
ValueError: Input [] is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.

This error is thrown up after reading of about 2% of the examples.
My env is the same as above. The command I use is:

python examples/question-answering/run_squad.py \
  --model_type bert \
  --model_name_or_path microsoft/MiniLM-L12-H384-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file "/content/transformers/dev-v1.1.json" \
  --predict_file "/content/transformers/dev-v1.1.json" \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir "/content/drive/My Drive/bert/newdir5"



Thanks, looking into it.

I works now!

Yes, it should be fixed in v3.0.0!
When is 2.12 coming out?
Author: Laksh1997
Pull request: no

Hi when is 2.12 coming out? Cheers.

(Interested in AutoModelForCausalLM)

Comments:
For now you can install from source to use `AutoModelForCausalLM`

Yes but you can't push out libraries with packages installed from master

Probably this week or the next!

+1

Version v3.0.0 was released this morning!
BertForPreTraining and BertModel when loading TF checkpoints
Author: jungwhank
Pull request: no

Hello, 
I want to load tensorflow checkpoints and when I run code below

```python
config = BertConfig.from_json_file('./bert_config.json')
config.output_hidden_states = True
bert_model = BertForPreTraining.from_pretrained('./model.ckpt', from_tf=True, config=config)
......
self.bert = bert_model
.......
bert_seq_out, _ = self.bert(input_ids, token_type_ids=segment_ids, attention_mask=input_mask,
output_hidden_states=False)

```

got error like this

```
Traceback (most recent call last):
  File "train_tf.py", line 738, in <module>
    neg_log_likelihood = model.neg_log_likelihood(input_ids, segment_ids, input_mask, label_ids)
  File "train_tf.py", line 605, in neg_log_likelihood
    bert_feats = self._get_bert_features(input_ids, segment_ids, input_mask)
  File "train_tf.py", line 542, in _get_bert_features
    bert_seq_out, _ = self.bert(input_ids, token_type_ids=segment_ids, attention_mask=input_mask, output_hidden_states=False)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'output_hidden_states'
```


I think it is because I use `BertForPreTraining` instead of `BertModel`,
but when I use `BertModel`, I got error like below

```
/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    592                 return modules[name]
    593         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 594             type(self).__name__, name))
    595 
    596     def __setattr__(self, name, value):

AttributeError: 'BertModel' object has no attribute 'bias'
```

How can I fix and load TF checkpoint correctly?
any help will be appreciated

Comments:
Hi! What is your library version? The `output_hidden_states` as argument to the call method was only added in the v3.0.0 version, which was released this morning.
Do T5 have the next-sentence-predict loss?
Author: guotong1988
Pull request: no

Input the previous sentence and predict/generate the next sentence.

Thank you very much.

Comments:

Hi @guotong1988   It's possible to do these tasks with text-to-text approach.

for predicting next sentence you can process input like this
`sentence1: sentence1_text  sentence2: sentence2_text`
and ask the model to predict `true` if sentence2 is the next sentence else `false`

and for generating next sentence, provide first sentence as input and next sentence as the target.

Thank you. It is not generating the whole next sentence. But it is predicting is_next or not_next.
Update README.md
Author: mrm8488
Pull request: yes
Labels: name: model card

- Model pic uploaded to a more permanent place

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5371?src=pr&el=h1) Report
> Merging [#5371](https://codecov.io/gh/huggingface/transformers/pull/5371?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b9ee87f5c730d72b326ef65089a574a0b519e827&el=desc) will **increase** coverage by `0.06%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5371/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5371?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5371      +/-   ##
==========================================
+ Coverage   77.49%   77.56%   +0.06%     
==========================================
  Files         138      138              
  Lines       24314    24314              
==========================================
+ Hits        18843    18858      +15     
+ Misses       5471     5456      -15     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5371?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5371/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `79.93% <0.00%> (+0.33%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5371/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `84.75% <0.00%> (+2.05%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5371?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5371?src=pr&el=footer). Last update [482a599...812f63a](https://codecov.io/gh/huggingface/transformers/pull/5371?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[Reformer] Include char lm to Trainer
Author: patrickvonplaten
Pull request: yes

Trainer currently expects every model to have a tokenizer. The reformer: `google/reformer-enwik8` is a char lm which does not require a tokenizer.

Comments:
@patrickvonplaten Thank you your awesome work. I´m super excited about a char-lm. 
I´m trying to can  train the "reformer" model, using the "google/reformer-enwik8".

But using this script or the run_language_modeling.py I get the error about the lack of tokenizer ( that is expected to a char only LM).
`Model name 'google/reformer-enwik8' was not found in tokenizers`

Can you give me some pointer how I could train it?

Hi bratao! Good point I will consult with our team on how to include models that don't have don't need a tokenizer! Let me get back to you in a couple of days :-)
Update Bertabs example to work again
Author: MichaelJanz
Pull request: yes

Fix the bug 'Attempted relative import with no known parent package' when using the bertabs example. Also change the used model from bertabs-finetuned-cnndm, since it seems not be accessible anymore

Comments:
There are `remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization` and also `remi/bertabs-finetuned-extractive-abstractive-summarization`. Which should be used here? @sshleifer 

Also #5234

Model remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization did not provide GPU support, which I (newbie) would expect from cnndm prefix.
I had to use remi/bertabs-finetuned-extractive-abstractive-summarization for GPU support.

Thanks! Circleci wants you to run `make style` I believe.

Ty, will do the next time. Is there a guideline for how to properly set up pull requests? I was not able to find one

Yes: https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests
How to interpret/act on this warning:  "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM"?
Author: ohmeow
Pull request: no

```
model = AutoModelForMaskedLM.from_pretrained("bert-base-cased")
```
Returns this warning ...
```
Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMaskedLM were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['cls.predictions.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

What is the recommended way to understand/act on this warning message?  For example, from what pre-trained model should I use for the MaskedLM task (and how would I know which to use for any other task)?

Comments:
How to interpret/act on this warning:  "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM
Author: ohmeow
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments: