noamaanMulla-03/transformers-issues huggingface dataset:
question for one parameter matrix in transformers/GPT2
Author: weiguowilliam
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

In transformers/gpt-2 model, there's a weight matrix called "transformer.h.0.attn.bias" whose size is torch.Size([1, 1, 1024, 1024]). I checked the original paper but still get confused by what it is for? 
This parameter matrix is between the Layernomalization layer and the attention layer. Can anyone explain that? Thank you in advance.


Comments:
I dont think something like that is there, please have a look. Or paste everything from h.0 as it is here. Then it will be easy.

> I dont think something like that is there, please have a look. Or paste everything from h.0 as it is here. Then it will be easy.

Hi, thank you for your reply. Here's the parameter list from embedding layer to the first decoder layer.

['transformer.wte.weight', 'transformer.wpe.weight', 'transformer.h.0.ln_1.weight', 'transformer.h.0.ln_1.bias', 'transformer.h.0.attn.bias', 'transformer.h.0.attn.c_attn.weight', 'transformer.h.0.attn.c_attn.bias', 'transformer.h.0.attn.c_proj.weight', 'transformer.h.0.attn.c_proj.bias', 'transformer.h.0.ln_2.weight', 'transformer.h.0.ln_2.bias', 'transformer.h.0.mlp.c_fc.weight', 'transformer.h.0.mlp.c_fc.bias', 'transformer.h.0.mlp.c_proj.weight', 'transformer.h.0.mlp.c_proj.bias']

The fifth item is 'transformer.h.0.attn.bias'. I guess it may be some random noise but I can't find any reference for that.

Hi! Indeed there is a `bias` item in the attention layer. The name is probably not as accurate as it could be, as it does not represent a bias but a triangular matrix that is used when computing the attention score.

As a causal language model, GPT-2 should only look at its left context. This triangular matrix makes sure that the values on the right of the focused token are set to zero so that they do not affect the resulting attention score.

You don't need to worry about this matrix as the model initializes it on its own :).

> Hi! Indeed there is a `bias` item in the attention layer. The name is probably not as accurate as it could be, as it does not represent a bias but a triangular matrix that is used when computing the attention score.
> 
> As a causal language model, GPT-2 should only look at its left context. This triangular matrix makes sure that the values on the right of the focused token are set to zero so that they do not affect the resulting attention score.
> 
> You don't need to worry about this matrix as the model initializes it on its own :).

Hi, thank you for your reply. So the 'attn.bias' is actually used as the masking matrix for attention here. That's why its size is 1024, which is the length of the context in gpt2. Hope this can help others!
Remove unnecessary use of FusedLayerNorm
Author: bryant1410
Pull request: yes

Fix #1172

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1315?src=pr&el=h1) Report
> Merging [#1315](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1315?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/a2d4950f5c909f7bb4ea7c06afa6cdecde7e8750?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1315/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1315?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1315      +/-   ##
==========================================
- Coverage   80.77%   80.76%   -0.01%     
==========================================
  Files          57       57              
  Lines        8092     8091       -1     
==========================================
- Hits         6536     6535       -1     
  Misses       1556     1556
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1315?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1315/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfYmVydC5weQ==) | `88.3% <100%> (-0.03%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1315?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1315?src=pr&el=footer). Last update [a2d4950...98dd19b](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1315?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Note this change makes the codebase to be compatible with apex amp O1.

Ok great, thanks @bryant1410!
Problem with word prediction with GPT2
Author: RuiPChaves
Pull request: no

## ❓ Questions & Help
I'm trying to understand how to obtain the probability of specific word predictions, but I am getting bad results.  For example, according to the code below, the sequence "It seems that" is more likely followed by "ago" than by "we", which surely is not correct. What am I doing wrong?

```import sys
import torch
import numpy
from scipy.special import softmax
from pytorch_transformers import GPT2Config, GPT2Tokenizer, GPT2LMHeadModel

config = GPT2Config.from_pretrained('gpt2-medium')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
model = GPT2LMHeadModel(config)

item = "It seems that"
indexed_tokens = tokenizer.encode(item)
tokens_tensor = torch.tensor([indexed_tokens])

with torch.no_grad():
	predictions = model(tokens_tensor)
	results = predictions[0]    
	temp = results[0,-1,:]
	temp = temp.numpy()
	result = softmax(temp)	
	word_1 = tokenizer.encode('we')[0]
	word_2 = tokenizer.encode('ago')[0]
	print(result[word_1])
	print(result[word_2])
```

This outputs:
1.0500242e-05
5.1639265e-05
But it should be the other way around (see [here](https://books.google.com/ngrams/graph?content=seems+that+ago%2C+seems+that+we&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t1%3B%2Cseems%20that%20we%3B%2Cc0)).
Thanks in advance.

Comments:
Hi, indeed it should be the other way around! I believe it's due to a misconception, you're initializing your model as follows:
```py
config = GPT2Config.from_pretrained('gpt2-medium')
model = GPT2LMHeadModel(config)
```
However, as noted in the [documentation](https://huggingface.co/transformers/main_classes/configuration.html#transformers.PretrainedConfig): _A configuration file can be loaded and saved to disk. Loading the configuration file and using this file to initialize a model does not load the model weights. It only affects the model’s configuration._

In order to initialize the model weights as well, you should do:
```py
config = GPT2Config.from_pretrained('gpt2-medium')
model = GPT2LMHeadModel.from_pretrained("gpt2-medium", config=config)
```
or since you're loading the pre-trained configuration of the same pre-trained model, you could simply do:
```py
model = GPT2LMHeadModel.from_pretrained("gpt2-medium")
```
which already loads this configuration file.

Once you have done this change you should get as output:

```
1.248501e-06
3.727657e-09
```

which is more accurate :)

Yes! Thank you! I was looking for an error on the completely wrong place (must have re-written the latter part of the code about 5 different ways).
Fix some typos in README
Author: chrisgzf
Pull request: yes

This PR fixes some typos in README.md and overall makes it slightly more readable.

No code changes.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1433?src=pr&el=h1) Report
> Merging [#1433](https://codecov.io/gh/huggingface/transformers/pull/1433?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f3e0218fbb6bcc40b40f10089dae8876654edb23?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1433/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1433?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1433   +/-   ##
=======================================
  Coverage   84.72%   84.72%           
=======================================
  Files          84       84           
  Lines       12591    12591           
=======================================
  Hits        10668    10668           
  Misses       1923     1923
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1433?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1433?src=pr&el=footer). Last update [f3e0218...85d7c84](https://codecov.io/gh/huggingface/transformers/pull/1433?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks for the update!
ERNIE 2.0 ?
Author: MuruganR96
Pull request: no
Labels: name: wontfix

Latest NLP Language Model.:)

[ERNIE 2.0](https://arxiv.org/pdf/1907.12412.pdf?source=post_page)

Comments:
This is relevant.
https://medium.com/syncedreview/baidus-ernie-2-0-beats-bert-and-xlnet-on-nlp-benchmarks-51a8c21aa433

We don't have any plan to add ERNIE in the short-term but if someone wants to do a (clean) PR with this model, happy to have a look and add it to the library.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fix pytorch-transformers migration description in README
Author: SeanBE
Pull request: yes


Comments:
Yes! Thanks!

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1438?src=pr&el=h1) Report
> Merging [#1438](https://codecov.io/gh/huggingface/transformers/pull/1438?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/904158ac4dbce046dd02be8382fdb8e52f0e691c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1438/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1438?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1438   +/-   ##
=======================================
  Coverage   84.72%   84.72%           
=======================================
  Files          84       84           
  Lines       12591    12591           
=======================================
  Hits        10668    10668           
  Misses       1923     1923
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1438?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1438?src=pr&el=footer). Last update [904158a...6dc6c71](https://codecov.io/gh/huggingface/transformers/pull/1438?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
How to install transformers with pytorch only?
Author: leitro
Pull request: no

## ❓ Questions & Help
Hi! Pytorch1.0 is installed and I'm installing the transformers with pip, everything is fine. But when I try:
```
import torch
from transformers import BertModel
```
then, an error occurred:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pc/miniconda3/lib/python3.7/site-packages/transformers/__init__.py", line 20, in <module>
    from .file_utils import (TRANSFORMERS_CACHE, PYTORCH_TRANSFORMERS_CACHE, PYTORCH_PRETRAINED_BERT_CACHE,
  File "/home/pc/miniconda3/lib/python3.7/site-packages/transformers/file_utils.py", line 30, in <module>
    assert int(tf.__version__[0]) >= 2
AttributeError: module 'tensorflow' has no attribute '__version__'
```

It seems like it cannot work unless both the tensorflow and pytorch have been installed, is that right? And is there a way to run transformers with pytorch only? (I don't want to install tensorflow)

Thanks in advance!

Comments:
Hi! No, you should be able to import every torch-related model without having TensorFlow installed. 

As I understand it, our method for identifying if you had TensorFlow 2.0 installed broke because the TensorFlow version you have in your environment does not have the attribute `__version__`.

Could you provide the TensorFlow version you have installed so that we may patch this bug? In the meantime, uninstalling TensorFlow from this environment or creating a new environment without this TensorFlow version should work fine. Thanks.

Thanks a lot! Now I got it, I think there is something wrong with my miniconda, because there is a build-in incomplete tensorflow, which has no version, no functions,...nothing but a box. My conda version is 4.7.11. As I cannot uninstall the incomplete version of tensorflow, I just install the latest tensorflow and leave it there, not it works fine with pytorch. Cheers!
Fine-tune specific layers
Author: hsajjad
Pull request: no

Is there any easy way to fine-tune specific layers of the model instead of fine-tuning the complete model?



Comments:
In Pytorch or Tensorflow? If Pytorch, [this issue](https://github.com/huggingface/transformers/issues/400) might be of help.

In my scripts, I use the following code. Passing down a parameter 'freeze' (list) to the config that I use. All layers that start with any of the given strings will be frozen. 

```python
# Freeze parts of pretrained model
# config['freeze'] can be "all" to freeze all layers,
# or any number of prefixes, e.g. ['embeddings', 'encoder']
if 'freeze' in config and config['freeze']:
    for name, param in self.base_model.named_parameters():
        if config['freeze'] == 'all' or 'all' in config['freeze'] or name.startswith(tuple(config['freeze'])):
            param.requires_grad = False
            logging.info(f"Froze layer {name}...")
```

Thanks. Your code works fine. I did the following:

 ```
if freeze_embeddings:
              for param in list(model.bert.embeddings.parameters()):
                  param.requires_grad = False
              print ("Froze Embedding Layer")

# freeze_layers is a string "1,2,3" representing layer number
    if freeze_layers is not "":
         layer_indexes = [int(x) for x in freeze_layers.split(",")]
         for layer_idx in layer_indexes:
              for param in list(model.bert.encoder.layer[layer_idx].parameters()):
                  param.requires_grad = False
              print ("Froze Layer: ", layer_idx)
```
GPT2 Tokenizer 
Author: lumliolum
Pull request: no

I want to know the pad token value for the gpt2 tokenizer. I have checked the [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json) but couldn't find any.


Thanks,
Suchith

Comments:
Hi! GPT-2 doesn't use padding tokens in its tokenizer. In order to manage padding, you should use the `attention_mask` detailed in the [documentation](https://huggingface.co/transformers/model_doc/gpt2.html#transformers.GPT2Model).

Closing as of now, feel free to reopen if @LysandreJik did not answer your question completely.
Provide requirements.txt for development dependencies
Author: rlouf
Pull request: yes

This PR adds the list of requirements needed to run the tests to the repo. Makes it easier for newcomers to contribute.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1447?src=pr&el=h1) Report
> Merging [#1447](https://codecov.io/gh/huggingface/transformers/pull/1447?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1615360c71f75da7b8aefd14c5d8a461486f865b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1447/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1447?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1447   +/-   ##
=======================================
  Coverage   84.72%   84.72%           
=======================================
  Files          84       84           
  Lines       12591    12591           
=======================================
  Hits        10668    10668           
  Misses       1923     1923
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1447?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1447?src=pr&el=footer). Last update [1615360...7afd00a](https://codecov.io/gh/huggingface/transformers/pull/1447?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Performance degradation with new version of this library (inference)
Author: mgrankin
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT-2

Language I am using the model on (English, Chinese....): Russian

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ x] my own modified scripts: (give details)

I do inference with a bit modified run_generation.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x] my own task or dataset: (give details)
I'm training Russian GPT-2

## To Reproduce

Steps to reproduce the behavior:

1. Use pytorch-tranformers library (1.2)
2. Use sample_sequence from run_generation.py on GPU
3. Use tranformers library (2.0)
4. Use sample_sequence from run_generation.py on GPU
5. Step 4 is running 5 times slower than step 2. 

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

The speed probably should stay the same.

## Environment

* OS: Ubuntu 18.04
* Python version: 3.7.3
* PyTorch version: 
* PyTorch Transformers version (or branch): 1.2 vs 2.0
* Using GPU  - yes
* Distributed of parallel setup - no
* Any other relevant information:

## Additional context
https://github.com/mgrankin/ru_transformers
<!-- Add any other context about the problem here. -->


Comments:
I'm so sorry, I haven't replaced all occurrences of `pytorch-tranformers` to `tranformers`. That was the source of the problem.

Glad to hear that!
integer representation ambuiguty in tokenizer
Author: weiguowilliam
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I use GPT-2 transformers model. Tokenizer.encode(' man') = 805, tokenizer.encode('man') = 805. But when in the sentence(e.g. Tokenizer.encode(' the man is a teacher') = [1169, 582, 318, 257, 4701], here the integer representing 'man' is 582. I think the problem is BPE used in transformer, where 850 is the integer representing the subtoken '-man' but not the word 'man'.
I wonder how I can set the tokenizer so I can get Tokenizer.encode(' man') = 582?

Comments:
Hello! You should specify `add_prefix_space=True` in your encode method to obtain that behavior.

Thank you! That works!
Input length is not equal to output length?
Author: RichardHWD
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->


Comments:
Hi @RichardHWD, I'm afraid we'll need a bit more information than what you have given.

@LysandreJik  Sorry. In your example:
```
import torch
from transformers import *

model_class = BertModel
tokenizer_class = BertTokenizer
pretrained_weights = 'bert-base-uncased'

tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights)

 # Encode text
input_ids = torch.tensor([tokenizer.encode("Here is some text to encode", add_special_tokens=False)])
with torch.no_grad():
    last_hidden_states = model(input_ids)[0]  # Models outputs are now tuples
    print(last_hidden_states.size())
    print(last_hidden_states)
```

I set add_special_tokens=False, and sentence "Here is some text to encode" has 6 words. But the output size is [1, 7, 768]. I want an equal length embedding, how to fix it?

What’s the shape of input_ids?

(I suspect it's gonna be 7, you should look into what BertTokenizer does. Thanks!).
Contribution guidelines
Author: rlouf
Pull request: yes

Here is a first draft to serve as a basis for discussion around contribution guidelines. Please mention anything that seems relevant to you / that you care about.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1448?src=pr&el=h1) Report
> Merging [#1448](https://codecov.io/gh/huggingface/transformers/pull/1448?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8fcc6507ce9d0922ddb60f4a31d4b9a839de1270?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1448/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1448?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1448   +/-   ##
=======================================
  Coverage   84.72%   84.72%           
=======================================
  Files          84       84           
  Lines       12591    12591           
=======================================
  Hits        10668    10668           
  Misses       1923     1923
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1448?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1448?src=pr&el=footer). Last update [8fcc650...45de313](https://codecov.io/gh/huggingface/transformers/pull/1448?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Can't replicate Language Model finetuning
Author: kristjanArumae
Pull request: no

I cannot replicate BioBERT results by using finetune_on_pregenerated.py with data generated using pregenerate_training_data.py.  

I've noticed that the LM code has been removed from the repo in that last couple versions.  Does this mean there were known issues with this process?



Comments:
Hello, this language model fine-tuning was community-maintained and is now deprecated. The example script to fine-tune on language modeling is now `run_lm_finetuning.py`.