noamaanMulla-03/transformers-issues huggingface dataset:
upgrading new transformer doesn't work
Author: ehsan-soe
Pull request: no

## ❓ Questions & Help

Hi,

I have pulled the repo again since a lot of stuff changed/added.
When I try to use ```pip install --upgrade .``` command, nothing changes and I am stuck in the following step forever:
```
(py36) pytorch-transformers$ pip install --upgrade .
Processing /home/pytorch-transformers

```

Plus, since a lot of folders are renamed and scripts moved to ```src``` folder, whenI try to do ```from transformers import BertTokenizer, BertModel, BertForMaskedLM```
I get following error:
```
ImportError                               Traceback (most recent call last)
<ipython-input-1-34ecfe73cb1a> in <module>
      1 import torch
----> 2 from transformers import BertTokenizer, BertModel, BertForMaskedLM, BertConfig, BertForPreTraining, BertConfig
      3 
      4 # OPTIONAL: if you want to have more information on what's happening, activate the logger as follows
      5 import logging

ImportError: cannot import name 'BertTokenizer' from 'transformers' (unknown location)
```

Would you please help with this? What is the reason to move to src?

Comments:
Previously you were relying on `transformers` being implicitly added to `PYTHONPATH` when you were working from the source of the repository. This breaks if you move to another directory, like `examples`.

`pip install .` makes `transformers` available in your virtualenv regardless of where you're working.

It's unclear to me why the installation is stuck.

Could you run the following commands?

```
pip uninstall transformers
pip --version
pip install --verbose .
```


Thanks for replying @aaugustin .
my pip version is 19.1.1 
However, I ended up cloning the whole repo again and install fresh.
I will close it for now.

OK, that was going to be my next suggestion if the situation didn't improve!
Perm Mask in XLNet
Author: PierreColombo
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
"Mask to indicate the attention pattern for each input token with values selected in [0, 1]: **If perm_mask[k, i, j] = 0, i attend to j in batch k; if perm_mask[k, i, j] = 1**, i does not attend to j in batch k. If None, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation)."
Can you confirm this is not the reverse i attend to j in batch k **if perm_mask[k, i, j] = 1** ?
Thanks a lot for your great work!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Typo in Documentation for GPT2LM Output "past"
Author: patrickvonplaten
Pull request: no

I think the described output shape for "past" for GPT-2 is wrong.
In:
[https://github.com/huggingface/transformers/blob/0cdfcca24b5739e25b584c05b866baa19ea382ef/transformers/modeling_gpt2.py#L332](url)
it says the output shape of each key, value tensor in each self attention layer is 
``(batch_size, num_heads, sequence_length, sequence_length)``
, but it should be 
``(batch_size, num_heads, sequence_length, hidden_size / num_heads)``
 or 
``(batch_size, num_heads, sequence_length, embed_size_per_head)``
.Also since the past tensor per layer always shows both key and value tensors it might even be clearer to write:
``(2, batch_size, num_heads, sequence_length, embed_size_per_head)``

Comments:
Yes, feel free to open a PR to fix this Patrick ;-)
Readme installation/test order can lead to confusion when running example unit tests
Author: internetcoffeephone
Pull request: no

## ❓ Questions & Help

When following the main readme installation/testing instructions in order, it is not mentioned that in order to let the examples tests pass, the separate examples/requirements.txt must be installed.
Thus, `pip install -r ./examples/requirements.txt` should come before `python -m pytest -sv ./transformers/tests/` in the main readme to avoid confusion.

Additionally, the readme line `python -m unittest discover -s examples -p "*test.py" -t examples` cannot find any tests, and produces the following output;

`Ran 0 tests in 0.000s
OK`

I don't think this is the intended behavior - is the line redundant?

Comments:
related to what @aaugustin is working on

Yes I'm planning to rework the contributor documentation. Currently it's a bit haphazard, sorry.

This is now clarified. The general README points to the README for examples which is unambiguous.
ModuleNotFoundError: No module named 'git'
Author: wrzzd
Pull request: no

## 🐛 Bug
`
Traceback (most recent call last):

  File "train.py", line 32, in <module>

    from distiller import Distiller

  File "~/transformers/examples/distillation/distiller.py", line 40, in <module>

    from utils import logger

  File "~/transformers/examples/distillation/utils.py", line 18, in <module>

    import git

ModuleNotFoundError: No module named 'git'

`
how to install git package?


Comments:
with referring to that file https://github.com/huggingface/transformers/blob/master/examples/distillation/requirements.txt

run:
`pip install -r requirements.txt`

> with referring to that file https://github.com/huggingface/transformers/blob/master/examples/distillation/requirements.txt
> 
> run:
> `pip install -r requirements.txt`

thx, got it~
clarification on output
Author: vr25
Pull request: no

Hi,

On using bert_uncased on the following sentence:

`Hello this is my dog`

and to get attentions I use:

`last_hidden_states, pooler_outputs, hidden_states, attentions = outputs`

`attentions` gives:
a tuple of 12 tensors where each tensor is of size [1,12,5,5]

I was wondering if the 12 tensors in the tuple are for each attention head or hidden layer.

Thanks!

Comments:
how to use  BertForMaskedLM
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I want to use  BertForMaskedLM as a decoder, apparently I need to give ids and then this function generates the ids and computes the loss. could you tell me how the generation with this function work? I see for instance in run_generation.py codes you use neucleus sampling or beam search, I see none of them used here, could you explain how this works. Also I want to see the generated sequence as text, could you tell me how from the output of this function I can get this information? thanks 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
FineTuning using single sentence document
Author: tuhinjubcse
Pull request: no
Labels: name: wontfix

Hello

roBERTa is not using the next sentence prediction objective. I want to fine-tune the pre-trained model on an unlabelled corpus of domain-specific text (ULMFIT style intermediate pretraining).
The bottleneck is my examples are single short sentences, instead of a document with multiple sentences 

The INPUT format here says it requires a file with one sentence per line and one blank line between documents. For me, each document has a single sentence
https://github.com/huggingface/pytorch-transformers/tree/b62abe87c94f8df4d5fdc2e9202da651be9c331d/examples/lm_finetuning


The last time I raised an issue this was an expected behavior I know
Re : https://github.com/huggingface/pytorch-transformers/issues/272

How to do it now :) ? Any help is appreciated . 

Comments:
Hi Tuhin, you can use `examples/run_lm_finetuning.py` now. The scripts in `examples/lm_finetuning` are deprecated (removed on master now).

Thomas i checked the wiki text 2 format and its confusing to me . Do we have to seperate documents by new lines ? My input file is a set of single sentence documents one per line . Do i need a new line after each sentence ?

My format is
sent1
sent2
sent3

Do i need to have
sent1

sent2

sent3 ?

I am currently running without a new line after each sentence and getting 09/19/2019 03:37:22 - WARNING - pytorch_transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (3989110 >
 512). Running this sequence through the model will result in indexing errors




Also while running
def mask_tokens(inputs, tokenizer, args):

- [ ] labels = inputs.clone()


   

- [ ]  # We sample a few tokens in each sequence for masked-LM training (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa)
- [ ]     **masked_indices = torch.bernoulli(torch.full(labels.shape, args.mlm_probability)).bool()**
- [ ] 

Getting error 
AttributeError: 'Tensor' object has no attribute 'bool'
Line 110


you need to update your pytorch version.  I believe bool() function was intrcoduced in torch 1.1.0 or 1.2.0

> I am currently running without a new line after each sentence and getting 09/19/2019 03:37:22 - WARNING - pytorch_transformers.tokenization_utils - Token indices sequence length is longer than the specified maximum sequence length for this model (3989110 >
> 512). Running this sequence through the model will result in indexing errors

I'm also getting this error. Is there more detail on the expected input format for the LM pretraining?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
attention_mask added, not multiplied ... is this correct?
Author: fginter
Pull request: no

https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L233

This line adds (+ operator) the attention mask. I wonder whether this is correct, as I would have very much expected the mask to be multiplied.

Comments:
Ping. :) To me this still looks like the code actually fails to apply the attention mask and also the parts of the sequence intended to be masked are accessible. The line is now https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L237

Hi, yes this is correct. Inside the `BertModel` forward method, the `attention_mask` is set to `0` for the tokens which should be attended (no modification) and `-10000` for the tokens which must be ignored, resulting in nullification of their attention scores.

You can read the relevant source code [here](https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L683).

I see. Thank you very much for the informative answer!

Thank you for the explanation. I was thinking the same thing when reading the code.

In that case, shouldn't the `attention_mark` input for BertEncoder, BertLayer, ...  be renamed to `extended_attention_mask` or `scaled_attention_mark`? Because those inner modules do expect the scaled (and reshaped?) mark, not the user input attention_mark.

Just a suggestion.
How to fine tune xlm-mlm-100-128 model.
Author: Radeeswar
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

How to fine tune xlm-mlm-17-128 model for own dataset. Since, run_lm_finetuning.py has no option to fine tune XLM models.


Comments:
If you're looking to fine-tune it on an MLM task you could simply re-use some parts of the `run_lm_finetuning.py` script to do it. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
What is currently the best way to add a custom dictionary to a neural machine translator that uses the transformer architecture?
Author: moyid
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

It's common to add a custom dictionary to a machine translator to ensure that terminology from a specific domain is correctly translated. For example, the term server should be translated differently when the document is about data centers, vs when the document is about restaurants.

With a transformer model, this is not very obvious to do, since words are not aligned 1:1. I've seen a couple of papers on this topic, but I'm not sure which would be the best one to use. What are the best practices for this problem?

One paper I found that seem to describe what I'm looking for is [here](aclweb.org/anthology/W18-6318.pdf ) - I have a bunch of questions regarding the paper, which I'm happy to discuss here as well. I'm also wondering if there are other approaches.


Comments:
This question is too general for this repo. It's not specific to anything this repository offers. Perhaps it's better to ask this on one of the Stack Exchange sites. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
transformers command not found after installing transformers using pip
Author: ManasRMohanty
Pull request: no

I wanted to convert TF checkpoints to pytorch saved files and thus I followed instructions as mentioned in the link https://huggingface.co/transformers/converting_tensorflow_models.html 

Thus, I installed PyTorch, Tensorflow and then transformers. But after doing so, when I ran the command, my system prompted me transformers command not found. 

What can be the issue here.

Comments:
I have this problem also. 
Installed with pip3 (maybe this is the necessary information)

Hi @ManasRMohanty, @DaniilRoman,

In 2.3.0 we introduced some new commands from the cli, which are now provided through **transformers-cli**.

Can you please try the following: 

```bash
transformers-cli convert --model_type <model_type> --tf_checkpoint /path/to/tf_model.ckpt --config /path/to/model.json --pytorch_dump_output /path/to/pytorch_model.bin
```

Let us know :) 

@mfuntowicz Do you want to update the doc? (i can do it too if needed)

@mfuntowicz @julien-c 
Yes, the above worked for me in linux. Thank you.

Also, I checked in https://huggingface.co/transformers/converting_tensorflow_models.html and I can see that the document is updated, but the convert parameter is missing there, so please update that.

@ManasRMohanty I've updated the documentation with the missing keyword. Thanks for reporting 👍
BERT multi heads attentions
Author: alshahrani2030
Pull request: no
Labels: name: wontfix

Hello,
I would like to analysis the effect of specific heads' attention.
Is it possible to turn off some heads attentions in a particular layer?
if yes, can you please tell me how to do that or share any helpful document? 

Thank you in advance

Comments:
I think the [BERTology](https://huggingface.co/transformers/bertology.html) section could help, especially the [run_bertology.py](https://github.com/huggingface/transformers/blob/master/examples/run_bertology.py) script can perform pruning and includes other useful functions :)

I am beginner to BERT, can you please tell me to turn off  the second  head in the ninth layer for example.
Here is my model config

config = BertConfig.from_pretrained("bert-base-uncased",output_attentions=True,output_hidden_states=True, num_labels=2)

model = BertForSequenceClassification.from_pretrained("bert-base-uncased", config= config)
model.cuda()

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Finetuning OpenAI GPT-2 for another language.
Author: 0x01h
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi,

Is there any option to finetune and use OpenAI GPT-2 for another language except English?

Comments:
Hello, if you want to try and fine-tune GPT-2 to another language, you can just give the `run_lm_finetuning` script your text in the other language on which you want to fine-tune your model. 

However, please be aware that according to the language and its distance to the English language (language on which GPT-2 was pre-trained), you may find it hard to obtain good results. 

@0x01h

GPT-2 can produce great results given a proper vocabulary. If you just run `run_lm_finetuning` on your lang dataset it will give you poor results, regardless of language distance from English because the vocab. 

I'd suggest that you train your tokenizer model first and then fine-tune GPT-2 with it. I'm doing that here 
https://github.com/mgrankin/ru_transformers



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Distilbert predicting mask
Author: VanOvermeire
Pull request: no

Hi,

This is probably me doing something wrong, but I can't get distilbert to give me a sensible prediciton when I mask part of a sentence. 

This setup for BERT (based on the examples):

```
import logging
logging.basicConfig(level=logging.INFO)

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = "Hello how are you doing?"
tokenized_text = tokenizer.tokenize(text)

masked_index = 2
tokenized_text[masked_index] = '[MASK]'
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0]

tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()

with torch.no_grad():
    outputs = model(tokens_tensor, token_type_ids=segments_tensors)
    predictions = outputs[0]

predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token)
```
gives the correct answer _are_ for _How are you doing?_.

But when I try the same with distilbert:

```
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')

text = "Hello how are you doing?"
tokenized_text = tokenizer.tokenize(text)

masked_index = 2
tokenized_text[masked_index] = '[MASK]'

indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
tokens_tensor = torch.tensor([indexed_tokens])

model = DistilBertModel.from_pretrained('distilbert-base-uncased')
model.eval()

with torch.no_grad():
    # not adding/adding the segment tokens. when I give those to the model, it throws an error
    last_hidden_states = model(tokens_tensor)
    outputs = last_hidden_states[0]
    
predicted_index = torch.argmax(outputs[0], dim=1)[masked_index].item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token)
```

I practically always get some _unusedxxx_ as a result. At first I thought this was because distilbert is a smaller model, but no matter what I try, I keep getting unused, so I am guessing it's something else.

Thanks in advance!


Comments:
In full `bert` case you are using `BertForMaskedLM` but for distill bert you are using `DistilBertModel` which is not for masked language modelling. Try using `DistilBertForMaskedLM`. Check it, it works:

https://colab.research.google.com/drive/1GYt9H9QRUa5clFfAke6KPYl0mi4H1F3H

Well, in hindsight that was obvious. :) Thanks!
Proposition to include community models in readme
Author: clmnt
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2382?src=pr&el=h1) Report
> Merging [#2382](https://codecov.io/gh/huggingface/transformers/pull/2382?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/629b22adcfe340c4e3babac83654da2fbd1bbf89?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2382/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2382?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2382   +/-   ##
=======================================
  Coverage   73.24%   73.24%           
=======================================
  Files          87       87           
  Lines       14989    14989           
=======================================
  Hits        10979    10979           
  Misses       4010     4010
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2382?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2382?src=pr&el=footer). Last update [629b22a...a229a68](https://codecov.io/gh/huggingface/transformers/pull/2382?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
fix #2399 an ImportError in official example
Author: karajan1001
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2400?src=pr&el=h1) Report
> Merging [#2400](https://codecov.io/gh/huggingface/transformers/pull/2400?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/78528742f169fb9481865aa25726ceca5499e036?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2400/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2400?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2400   +/-   ##
=======================================
  Coverage   73.24%   73.24%           
=======================================
  Files          87       87           
  Lines       14989    14989           
=======================================
  Hits        10979    10979           
  Misses       4010     4010
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2400?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2400?src=pr&el=footer). Last update [7852874...71292b3](https://codecov.io/gh/huggingface/transformers/pull/2400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
i want to use bert pre-trained modle in a text classification problem which the text with Multi-label. But，there are some problems .
Author: lushishuai
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
i want to use bert pre-trained modle in a text classification problem which the text with Multi-labels. Which program and task should i select between run_glue.py、run_multiple_choice.py、run_squad.py and so on？
for example of my text：“I'd like 2 tickets to see Zoolander 2 tomorrow at Regal Meridian 16 theater in Seattle at 9:25 PM”     
 in this text include this labels: request_ticket;inform_moviename;inform_date;inform_theater;inform_city;inform_starttime;inform_numberofpeople	
which program  should i select?  Thanks very much!

Comments:
run_multiple_choice.py will be a good choice.
in case ‘bert’, it uses 
https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L1021

but i think your problem formulation is odd.
what about classifying ‘request_*’ intent as normal classification problem and slot tagging ‘inform_*’ as sequence classification?


hi. i just start to learn bert for a very short time,so i could not have learned it clearly. 
‘request_’ intent classifying can be very different with the normal classification problem?
i have not understand this.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.