noamaanMulla-03/transformers-issues huggingface dataset:
Some questions in Loss Function for MaskedLM
Author: wlhgtc
Pull request: no

Use the same sentence in your **Usage** Section:
```
# Tokenized input
text = "Who was Jim Henson ? Jim Henson was a puppeteer"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 6
tokenized_text[masked_index] = '[MASK]'
```

Q1. 
When we use this sentence as training data，according to your code
```
 if masked_lm_labels is not None:
            loss_fct = CrossEntropyLoss(ignore_index=-1)
            masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
            return masked_lm_loss
```
seem the loss is a sum of all word in this sentence, not the single word "henson", am I  right? But in my opinion, we only need to calculate the **masked** word's loss, not the whole sentence?

Q2.
It's also a question about masked, "chooses 15% of tokens at random" in the paper, I don't know how to understand it... For each word, a probability of 15% to be masked or just 15% of the sentence is masked?
Hope you could help me fix them.

By the way, the  notes in line 731 in: pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling.py should be :  if `masked_lm_labels` is  not `None`, missed a word "not".

Comments:
@julien-c  Seem some conflict with the original BERT in tf.
The code in tf is as follows:

```
def gather_indexes(sequence_tensor, positions):
  """Gathers the vectors at the specific positions over a minibatch."""
...
input_tensor = gather_indexes(input_tensor, positions)
```

Or dose you mean that we could set all words that are not masked(random pick from the sentence) and the padding(add to reach max_length) to "-1"(in order to ignore)?

> Q1:[...] But in my opinion, we only need to calculate the masked word's loss, not the whole sentence?

It's exactly what is done in the current implementation. The labels of not masked tokens are set to -1 and the loss function ignores those tokens by setting ignore_index=-1 (see [documentation](https://pytorch.org/docs/stable/nn.html#crossentropyloss))

> Q2.
It's also a question about masked, "chooses 15% of tokens at random" in the paper, I don't know how to understand it... For each word, a probability of 15% to be masked or just 15% of the sentence is masked?

Each token has a probability of 15% of getting masked. You might wanna checkout [this code](https://github.com/deepset-ai/pytorch-pretrained-BERT/blob/master/examples/run_lm_finetuning.py#L288) to get a better understanding 

So nice to see your reply, it do fix my problem,4K.


@tholor  I want to rebuild BERT on a single GPU, still some problems. May I know your email address ?

malte.pietsch [at] deepset.ai 
But if you have issues that are of interest for others, please use github.

Thanks @tholor!
BertForSequenceClassification() model TF to pytorch conversion
Author: altsoph
Pull request: yes
Labels: name: wontfix

I added a script convert_bert_seqclass_tf_checkpoint_to_pytorch.py for the conversion a trained BertForSequenceClassification model from TF to pytorch.
I had to modify modeling_bert.py to support it, as well.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2170?src=pr&el=h1) Report
> Merging [#2170](https://codecov.io/gh/huggingface/transformers/pull/2170?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c8ed1c82c8a42ef700d4129d227fa356385c1d60?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2170/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2170?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2170      +/-   ##
==========================================
- Coverage   80.35%   80.34%   -0.01%     
==========================================
  Files         114      114              
  Lines       17095    17097       +2     
==========================================
  Hits        13736    13736              
- Misses       3359     3361       +2
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2170?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2170/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `87.43% <0%> (-0.32%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2170?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2170?src=pr&el=footer). Last update [c8ed1c8...e3c65da](https://codecov.io/gh/huggingface/transformers/pull/2170?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
help...
Author: ghost
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
---------------------------------------------------------------------------

i saw great example in (https://huggingface.co/transformers/main_classes/model.html?highlight=from_pretrained#pretrainedmodel) but i got an error please help 


![image](https://user-images.githubusercontent.com/48268050/72089118-d0193500-334e-11ea-9b23-a63bcfe67e44.png)


config = BertConfig.from_json_file('./tf_model/my_tf_model_config.json')
model = BertModel.from_pretrained('./tf_model/my_tf_checkpoint.ckpt.index', from_tf=True, config=config)

i followed this 2 lines code but get error please help...

![image](https://user-images.githubusercontent.com/48268050/72089259-0a82d200-334f-11ea-9489-f97be66bd67d.png)

here is my code

![image](https://user-images.githubusercontent.com/48268050/72089322-2a19fa80-334f-11ea-91ae-297c0b20b852.png)


Comments:
I haven't worked on TF code like this personally, but by looking [https://github.com/huggingface/transformers/blob/master/README.md#quick-tour-tf-20-training-and-pytorch-interoperability](url) It shows that they don't override the config like you have done.

Now if that doesn't work - which I don't think it will work to be fair - my guess is that the model file your attempting to load is of type `BertModel` when it should be ` TFBertForSequenceClassification`

Have a look at the link and let us know how you get on.

Please also change the title of this issue to something meaningful.

First of all: please change your title and please post code snippets in tags and not images. They load slow, are hard to read, and impossible to copy-paste - just plain annoying. :-)

Second, it seems that your checkpoint contains additional layers, particularly a classifier layer. So you probably want to load the weights into another model architecture. Probably one of these (instead of just `BertModel`):

- BertForSequenceClassification
- BertForTokenClassification
- BertForMultipleChoice

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Strange behaviour after using BertTokenizer.add_tokens()
Author: Saltychtao
Pull request: no

When a word is in origin Bert vocabulary, it would not be split.  But after adding several its subtokens, it would be split into pieces, which seems to inconsistent with the longest match first algorithm.
To reproduce:
```
import transformers
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased)
tokenizer.tokenize('involve')
tokenizer.add_tokens(['inv','ol','ve'])
tokenizer.tokenize('involve')
```
Line 2 returns 'involve' , and line 4 returns ['inv','##ol','##ve']. 
Is this behaviour expected or is it a bug?

Comments:
No, this is expected behavior. If it was returning `['inv', 'ol', 've']`, then the model would identify each token as being a beginning of word token, whereas the last two are actually part of words following a beginning of a word.

Thanks for your response. But since both 'involve' and `['inv', 'ol', 've']` are in the vocabulary, shouldn't 'involve' be kept unsplit instead of split into subword? I am expecting the output to be `'involve'` instead of `['inv','##ol','##ve']`.
Imports likely broken in examples
Author: aaugustin
Pull request: no
Labels: name: wontfix

## 🐛 Bug

While cleaning up imports with isort, I classified them all. I failed to identify the following four imports:

1. model_bertabs
2. utils_squad
3. utils_squad_evaluate
4. models.model_builder

These modules aren't available on PyPI or in the transformers code repository.

I think they will result in ImportError (I didn't check).

I suspect they used to be in transformers, but they were renamed or removed.



Comments:
@aaugustin   it seems like that i met the same problem when i use convert_bertabs_original_pytorch_checkpoint.py ,have you ever fixed it or find any way to make it work.
Appriciate it if you can tell me!

I didn't attempt to fix this issue. I merely noticed it while I was working on the overall quality of the `transformers` code base.

I suspect these modules used to exist in `transformers` and were removed in a refactoring, but I don't know for sure.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Handle pinned version of isort
Author: BramVanroy
Pull request: yes

The CONTRIBUTING file pins to a specific version of isort, so we might as well install that in `dev` . This makes it easier for contributors so they don't have to manually install the specific commit.

Comments:
I didn't know that worked, thanks @BramVanroy. @aaugustin what do you think?

I'm not familiar with the syntax, but if it works, go for it. I really hope we have a release of sort and we can remove this soon.

Works on my machine so I'll merge :)

Thanks @BramVanroy, this will simplify @LysandreJik and @thomwolf's lives a lot!
Run_tf_ner.py error on TPU 
Author: dlauc
Pull request: no
Labels: name: wontfix

## 🐛 Bug
run_tf_ner.py does not work with TPU. The first error is:
`File system scheme '[local]' not implemented `
When the script is changed and .tfrecord file is moved to gs:// address (and also hardcoded "/tmp/mylogs" is replaced with gs:/// dir)  there is an error with optimiser:
`AttributeError: 'device_map' not accessible within a TPU context.`

## To Reproduce

Steps to reproduce the behaviour:

python run_tf_ner.py.1 --tpu grpc://10.240.1.2:8470 --data_dir gs://nomentech/datadir --labels ./datasets/labels.txt --output_dir gs://nomentech/model1 --max_seq_length 40 --model_type bert --model_name_or_path bert-base-multilingual-cased --do_train --do_eval --cache_dir gs://nomentech/cachedir --num_train_epochs 5 --per_device_train_batch_size 96


## Environment

* OS: Ubuntu 18
* Python version: 3.7
* Tensorflow version: 2.1.0-dev20191222 (tf-nightly)
* PyTorch Transformers version (or branch): 2.3.0
* Distributed of parallel setup ? TPU


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have the same issue on Colab TPU with tf-nightly 2.2.0. @dlauc did you solve the problem?

Hi @BOUALILILila, I've switched to the TF/Keras - it works well with TPU-s

Hi @dlauc. Could you elaborate how you fixed this? I am having the same problem.
BertTokenizerFast.encode() ignores max_length
Author: yonigottesman
Pull request: no
Labels: name: wontfix

## 🐛 Bug
for this input: (from yelp review)
`text = "After a morning of Thrift Store hunting, a friend and I were thinking of lunch, and he suggested Emil's after he'd seen Chris Sebak do a bit on it and had tried it a time or two before, and I had not."`

If i use the standard BertTokenizer it works fine:
`tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')`
`len(tokenizer.encode(text, max_length=32))   `  
`output: 32`

but if i use the fast version:
`tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')`
`len(tokenizer.encode(text, max_length=32))   `  
**`output: 55`**




* OS: macos
* Python version: 3.6.5
* PyTorch version:  1.3.1
* PyTorch Transformers version (or branch): master branch
* Using GPU ? no
* Distributed or parallel setup ? no
* Any other relevant information: no

Comments:
I'll let @n1t0 chime in if needed, but in the `Fast` versions of the tokenizers you have to define the `max_length` at initialization, not when calling `.encode()`

Can you try this and let me know if it works?

Oh You are right. When i init with max_length it works.
Is this documented?
Thanks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Does calling fit() method on TFBertForSequenceClassification change the weights of internal pre-trained bert?
Author: hamidreza-ghader
Pull request: no
Labels: name: wontfix

Hi all,

Let's say, I have a TFBertForSequenceClassification object and I call fit method on it. Does it change the weights of the internal TFBertMainLayer too or it only trains the weights of the Dropout and the classifier Dense layers?

Best

Comments:
I can't tell for sure about the TF version, but I would assume it's the same as the one in PyTorch, in which case yes: all weights are changed. You can freeze layers, though. Here (for PyTorch nn.Module) only freezing the embeddings:

```python
bert = BertModel.from_pretrained('bert-base-uncased')
for name, param in bert.named_parameters():                
    if name.startswith('embeddings'):
        param.requires_grad = False
```

If your problem is solved, please close this.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Generate all possible sentences using a fine-tuned GPT-2 model
Author: zeyuhuan
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Is there a way to generate all possible sentences using a fine-tuned GPT-2 model given a certain sampling technique? For some reason I wan to exhaust all possible combination of tokens given a fine-tuned GPT-2 model with certain sampling technique. Is it doable? If it is not, how do we get an estimate of how many possible sentences are there in the latent space?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
What do you mean exactly be all possible sentences? 

The space of possible sentence that could be generated grows exponentially with the length of the sentences. Having **V** words in your vocabulary, there are **V^N** possible sentences of length **N** that could be sampled. If **V ~ 50,000** and **N = 10**, you are already at > 10^40 possibilities which is intractable.

Thanks for the reply. By all possible sentences I meant all possible sentences under some certain sampling technique. For example, if I want all possible sentences with top-k=1, there would be just 1 in the space. I can control the desired number of possible sentences by choosing sampling techniques at a certain level of strictness. However, I don't want to do a sampling, I want to find all of them by some BFS or DFS. What I plan to do is find some desired sampling technique so that there are say 1 million unique sentences in the space, then find out all of them. The provided util in the package only does sampling which could generate duplicate sentences. Does that make sense?
Will you release the pre-train script for T5?
Author: g-jing
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
There is a fine-tune script for BERT/GPT https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py. Will you include T5 into this script? Thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error when running demo script in T5Model
Author: g-jing
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): T5

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: When I use the example script in https://github.com/huggingface/transformers/blob/dfe012ad9d6b6f0c9d30bc508b9f1e4c42280c07/src/transformers/modeling_t5.py#L716-L724

It shows 
```
Traceback (most recent call last):
  File "hello.py", line 6, in <module>
    model = T5Model.from_pretrained('t5-small')
  File "/home/jinggu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jinggu/anaconda3/lib/python3.7/site-packages/transformers/modeling_t5.py", line 859, in forward
    encoder_outputs = self.encoder(hidden_states, **kwargs_encoder)
  File "/home/jinggu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'lm_labels'
```

Comments:
I found the error is because the `lm_labels` is not poped out. One possible solution: change https://github.com/huggingface/transformers/blob/dfe012ad9d6b6f0c9d30bc508b9f1e4c42280c07/src/transformers/modeling_t5.py#L864 to 
```
        lm_labels = kwargs.pop('decoder_lm_labels', None)
        if not lm_labels:
            lm_labels = kwargs.pop('lm_labels', None)
```


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
add crf layer to BERT, RoBERTa, XLMR
Author: mezig351
Pull request: yes

Building on pull request [ bert（+lstm）+crf #2249 ], I make it more generic and similar to run_ner.py . There is a CRF layer adapted from https://github.com/jiesutd/LatticeLSTM , which does not have additional package requirements, and was published in ACL 2018. 

Comments:
What do you think about just using https://github.com/harvardnlp/pytorch-struct (I'm biased)? It should remove the need for most of this code. 

Also we now have a pytorch-lightning NER example which should make this much simpler. 

Would also be helpful to know if any of this improves prediction accuracy.

@srush thanks for the comment, I'll definitely check out the repo. Regarding accuracy, there are studies that suggest using CRF yields better results in some languages, for example in Portugese and Slavic languages:

article{souza2019portuguese,
  title={Portuguese Named Entity Recognition using BERT-CRF},
  author={Souza, F{\'a}bio and Nogueira, Rodrigo and Lotufo, Roberto},
  journal={arXiv preprint arXiv:1909.10649},
  year={2019}
}

article{arkhipov2019tuning,
  title={Tuning Multilingual Transformers for Named Entity Recognition on Slavic Languages},
  author={Arkhipov, Mikhail and Trofimova, Maria and Kuratov, Yuri and Sorokin, Alexey},
  journal={BSNLP’2019},
  pages={89},
  year={2019}
}


this looks promising. is this going to be merged?

So as per my above comment, I'm not going to merge this example which feels too complex to me. But I will start an issue to put together some work `crf/` examples for ner/chunking/parsing that people can build on.
[Question] Add extra sublayer for each layer of Transformer
Author: robinsongh381
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello !

BERT base has 12 layers and each layer includes follwing sublayer ( and of course, add norm)

` {self-attention -> feed-foward} `

I was wondering if there is a way of adding extra unit to this sublayer , for example,

`{self-attention -> feed-foward -> **LSTM**}`

Thanks
 

Comments:
Anything is possible, if you want to! But it's not straightforward, I think. You can have a look at `BertLayer` where I would assume that you make your changes.

https://github.com/huggingface/transformers/blob/dfe012ad9d6b6f0c9d30bc508b9f1e4c42280c07/src/transformers/modeling_bert.py#L365-L373

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
SentencePiece Error with AlbertTokenizer using google pretrained chinese model
Author: ubuntu733
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using albert:

Language I am using the model on , Chinese:

The problem arise when using:
* [ ] the official example scripts: (give details)
`AlbertTokenizer.from_pretrained(vocab)`

It shows:

> Traceback (most recent call last):
  File "/home/shenchengen/venv/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 438, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/shenchengen/venv/lib/python3.6/site-packages/transformers/tokenization_albert.py", line 90, in __init__
    self.sp_model.Load(vocab_file)
  File "/home/shenchengen/venv/lib/python3.6/site-packages/sentencepiece.py", line 118, in Load
    return _sentencepiece.SentencePieceProcessor_Load(self, filename)
RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(73) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 
sentencepiece_processor.cc(558) LOG(ERROR) /sentencepiece/src/sentencepiece_processor.cc(124) [model_] Model is not initialized.
Returns default value 0
sentencepiece_processor.cc(558) LOG(ERROR) /sentencepiece/src/sentencepiece_processor.cc(124) [model_] Model is not initialized.
Returns default value 0


**It seems origin google chinese pretrained model do not use sentence piece,but word piece, so the pretrained model has no sentence piece model but has vocab_chinese.txt** [albert-issues](https://github.com/google-research/ALBERT/issues/58)

### Expected behavior:
   AlbertTokenizer support word piece method


Comments:
Indeed, this implementation of ALBERT only supports SentencePiece as its tokenizer.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.