noamaanMulla-03/transformers-issues huggingface dataset:
Masking in Bert
Author: shashankMadan-designEsthetics
Pull request: no

I am not able to grasp the concept of attention masking in bert or the other transformers.
<img width="790" alt="Screenshot 2020-05-01 at 7 56 06 PM" src="https://user-images.githubusercontent.com/45225143/80812561-d37a4a00-8be5-11ea-9da6-469b0d2d3f8f.png">
According to the documentation I tried to experiment it out.
Here it is clearly mentioned that the 1's in position are going to be masked and 0's are not.
So i tried to experiment it out and got this result.
<img width="510" alt="Screenshot 2020-05-01 at 8 00 02 PM" src="https://user-images.githubusercontent.com/45225143/80813016-d1fd5180-8be6-11ea-8fcc-c86943fc003f.png">
Here are my results the numbers on the left are 0 and right non zero.
I still dont get the whole picture of how this code fits into the whole picture.

TLDR
In general terms, if any one could explain me attention masking in the self attention part of the code in transformers and its variants(possibly with code) it would be great.

Comments:
I don't fully understand the question, but love the color theme :)

> I don't fully understand the question, but love the color theme :)

Hey thanks😄its cobalt on vscode. Have edited the question with tldr pls let me know on it.

That being said, please don't use screenshots. They are hard to read (especially on phones) and make it impossible to copy-and-paste your code. Use [code blocks](https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks) instead. I also don't quite understand your question. Do you have a general question about how attention works? Please use [Stack Overflow](https://stackoverflow.com/) for this, as the template clearly mentions.

> That being said, please don't use screenshots. They are hard to read (especially on phones) and make it impossible to copy-and-paste your code. Use [code blocks](https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks) instead. I also don't quite understand your question. Do you have a general question about how attention works? Please use [Stack Overflow](https://stackoverflow.com/) for this, as the template clearly mentions.

Sure will follow the guidelines, closing the issue.
run_tf_glue with AdamW optimizer and distributed training
Author: BOUALILILila
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
Can you please update the run_tf_glue example (Tensorflow version) that includes the Adam optimizer with Warm-up and weight decay.  If you can also show how to run the code on multiple GPUs and on TPU. 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error: `cannot import name 'TFBertForMaskedLM'`
Author: danyaljj
Pull request: no

When I try the following import: 

```python 
from transformers import TFBertForMaskedLM
```

I get the following error: 

```
Traceback (most recent call last):
  File "/Users/danielk/ideaProjects/farsi-language-models/src/try_bert_lm.py", line 3, in <module>
    from transformers import TFBertForMaskedLM
ImportError: cannot import name 'TFBertForMaskedLM' from 'transformers' (/usr/local/lib/python3.7/site-packages/transformers/__init__.py)
```

My versions are: 
```
torch                              1.4.0    
transformers                       2.8.0    
```


Comments:
The issue was that I did not have TensorFlow installed.
Model output should be dict
Author: max-yue
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
I think model output should be a dict. Since dict are more clearly than tuple, we do not know what output[1] means, but we do know output["pooled_ouptut"] means.
 
https://github.com/huggingface/transformers#models-always-output-tuples

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
We will soon convert the outputs of models to named tuples to that it will be more readable :-)
Padding Token in Transformer XL
Author: sb1992
Pull request: no
Labels: name: wontfix

I have sentences of varying lengths and I was wondering how to handle that as I could not see any padding token present. The index 0 refers to <eos> in the vocab, so any help on addition of padding would be appreciated

Comments:
For these causal models that consider the left-context only, it's ok not to worry too much about padding since the attention modules only look to the previous tokens. Just be careful when you compute the loss to ignore the out-of-sentence-tokens (using loss functions `ignore_index` for instance).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, I have a related doubt. In the example code [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py), GPT-2 and transformer-xl throw error due to lacking padding token.

Please check recent comments in this [Issue](https://github.com/huggingface/transformers/issues/3021)
Adding Transformer-XL pre-trained model
Author: thomwolf
Pull request: yes

Add Transformer-XL (https://github.com/kimiyoung/transformer-xl) with the pre-trained WT103 model (maybe also the 1B-Word model).

The original Google/CMU PyTorch version  (https://github.com/kimiyoung/transformer-xl/tree/master/pytorch) has been slightly modified to better match the TF version which has the SOTA results. I've mostly untied the relative positioning/word biases and changed the initialization of memory states (TODO: PR these modifications back up to the original repo).

The Google/CMU model can be converted with:
```bash
pytorch_pretrained_bert convert_transfo_xl_checkpoint [PATH_TO_TRANSFO_XL_FOLDER]/model.ckpt-0 [PATH_TO_SAVE_PT_DUMP]
```
And the corpus and vocabulary with:
```bash
pytorch_pretrained_bert convert_transfo_xl_checkpoint [PATH_TO_TRANSFO_XL_DATA_FOLDER]/cache.pkl [PATH_TO_SAVE_DATA_AND_VOCABULARY]
```

The evaluation can be run with
```bash
cd ./examples
python eval_transfo_xl.py --cuda --model_name [PATH_TO_PT_DUMP] --work_dir [PATH_TO_SAVE_LOG]
```

Currently I have slightly higher values for the perplexity with the PyTorch model (using the TF pre-trained weights) `20.4` versus `18.3` for the TF version, might try to investigate this a little bit further.

Comments:
Awesome work!

I'm not sure I understand how this model integrates with BERT. Did Google release weights for the MLM + next sentence prediction task using the Transformer XL? And if they did not, how well do the classical LM weights performs for finetuning tasks?

Oh it's not integrated with BERT, that just another model. I'm adding it here to make it use the same easy-to-use interface (with pretrained/cached model and tokenizer) like I do for the OpenAI GPT in the other PR (#183)
It's a language model like OpenAI GPT (trained with just a classical LM loss).
Check the Transformer-XL repo/paper for more details!

Ok, managed to find the bug in this conversion.
The custom-made AdaptiveSofmax used in Transformer-XL indexes the next-cluster probability tokens in reverse-order (see indexing by `-i` on [line 141 of the original repo](https://github.com/kimiyoung/transformer-xl/blob/master/pytorch/utils/proj_adaptive_softmax.py#L141)) so we got worse performances on less frequent words.
Tough to find!

Fixing this conversion we got a test set perplexity of `18.213` on Wiki-103 to be compared to a reported result of `18.3` with the TensorFlow model.

#254 is also the main PR for the inclusion of Transformer-XL. Closing this PR.

Hi, I am assuming the pre-trained model available with huggingface is the `large` variant and not `base`?
Issue with Makefile
Author: surya-narayanan
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert ...):

Language I am using the model on (English): 

The problem arises when using:
* [x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. pip install -e ".[testing]"
2. pip install -r examples/requirements.txt
3. make test-examples

Running 3 throws a few errors and warnings including a   AttributeError: 'TestCaseFunction' object has no attribute 'get_marker'. Wasn't happening a few days ago. What should I do?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
TypeError: apply_gradients() missing 1 required positional argument: 'clip_norm'
Author: dimitreOliveira
Pull request: no

# 🐛 Bug

## Information

Model I am using (TFBertModel):

Language I am using the model on (English):

Also I'm using `tensorflow==2.1.0` and `transformers==2.3.0`

The problem arises when using:
* [x] the official example scripts: (give details below)
I'm trying to use the `optimization_tf.create_optimizer` from the source code.

The tasks I am working on is:
* [x] my own task or dataset: (give details below)
Just a text classification task.

## To reproduce

Steps to reproduce the behavior:

1. Try to use the model the regular way
2. When running the model with "optimization_tf.create_optimizer"

## Environment info

```
/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py in _process_single_batch(model, inputs, targets, output_loss_metrics, sample_weights, training)
    271                         loss_scale_optimizer.LossScaleOptimizer):
    272             grads = model.optimizer.get_unscaled_gradients(grads)
--> 273           model.optimizer.apply_gradients(zip(grads, trainable_weights))
    274       else:
    275         logging.warning('The list of trainable weights is empty. Make sure that'

TypeError: apply_gradients() missing 1 required positional argument: 'clip_norm'
```

## How I am able to run

On the class `AdamWeightDecay` and the method `apply_gradients` I just call the  supper function like this:

```
def apply_gradients(self, grads_and_vars, name=None):
    return super().apply_gradients(grads_and_vars)
```

but as you can see I'm not using the `clip_norm` as the source example uses.

Is there a  way to use the original source function as described in the source code?

Comments:
Hi, indeed this optimizer `AdamWeightDecay` requires an additional argument for truncating the gradient norm.

It essentially feeds the `clip_norm` argument (which is the second required argument in `apply_gradients`) to [tf.clip_by_global_norm](https://www.tensorflow.org/api_docs/python/tf/clip_by_global_norm).

You can see a usage example in our [run_tf_ner.py example](https://github.com/huggingface/transformers/blob/master/examples/run_tf_ner.py#L203)

I wasn't able to implement this fix on my problem but I think this answer closes the issue, thank!

This problem occurs if you don't specify `clip_norm` when calling `apply_gradients`.
If using a custom training loop, the fix is easy :)
If you are using `keras.model.fit`, you can do it the following way:

```
from functools import partialmethod

AdamWeightDecay.apply_gradients = partialmethod(AdamWeightDecay.apply_gradients, clip_norm=1.0)
optimizer = create_optimizer(p.learning_rate, num_train_steps=total_steps, num_warmup_steps=warmup_steps)
```
Set specific hidden_size for ClassificationHead
Author: GuillaumeDesforges
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Add a config term `head_hidden_size` to the model configurations that will be used for the head of models such as `RobertaForSequenceClassification`

## Motivation

HuggingFace transformers library provides a very accessible API and PyTorch models that can be used "plug n play" for various task such as classification.
In many case, varying the hidden size of the last layer (that outputs the logits) is one of the first thing we tweak to improve the performance on such a task.

Currently, the dense layer uses the `hidden_size` config parameter, which is the same as the one used in the transformer (BERT). One cannot change the hidden size of the last layer without changing the hidden size of the entire transformer model behind it.

This means we have to code a new PyTorch module in order to do something as simple as that.

## Your contribution

I could PR this should the change be welcomed.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Regarding attention received by the distilbert model
Author: divyag11
Pull request: no
Labels: name: wontfix

I receive the attention of all the 6 layers with all the 12 heads while exporting the tfdistilbert model. I just want to take the attention of dimension equal to the sequence length. Which layer and head attention is the most effective attention value that I should take in order to get the best value of attention scores with respect to my sentence.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pass existing tensorboard SummaryWriter to Trainer. 
Author: jaymody
Pull request: no

# Feature request
The [`Trainer`](https://github.com/huggingface/transformers/blob/41750a6cff55e401364568868d619747de3db037/src/transformers/trainer.py#L102) class should let us pass our own `tb_writer` (Currently, a new `tb_writer` is created using the `args.logging_dir` argument if it exists).

## Motivation
This let's us have more detailed logs since we can add add things like hyper-params, histographs, and whatnot if we choose so.

Comments:
Negative dimension when initialising the XLNetModel
Author: yxu132
Pull request: no

# 🐛 Bug

## Information

XLNetModel:

## To reproduce

Steps to reproduce the behavior:

```
import torch
from pytorch_transformers import *

# PyTorch-Transformers has a unified API
# for 7 transformer architectures and 30 pretrained weights.
#          Model          | Tokenizer          | Pretrained weights shortcut
MODELS = [(XLNetModel,      XLNetTokenizer,     'xlnet-base-cased')]

# Let's encode some text in a sequence of hidden-states using each model:
for model_class, tokenizer_class, pretrained_weights in MODELS:
    # Load pretrained model/tokenizer
    tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
    model = model_class.from_pretrained(pretrained_weights)

```

with the code above I got following errors: 

```
Traceback (most recent call last):
  File "/Users/xx/xxx/xxx/test.py", line 424, in <module>
    model = model_class.from_pretrained(pretrained_weights)
  File "/Users/yxu132/pyflow3.6/lib/python3.6/site-packages/pytorch_transformers/modeling_utils.py", line 536, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/Users/yxu132/pyflow3.6/lib/python3.6/site-packages/pytorch_transformers/modeling_xlnet.py", line 731, in __init__
    self.word_embedding = nn.Embedding(config.n_token, config.d_model)
  File "/Users/xx/pyflow3.6/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 97, in __init__
    self.weight = Parameter(torch.Tensor(num_embeddings, embedding_dim))
RuntimeError: Trying to create tensor with negative dimension -1: [-1, 768]
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:  1.2.0
- Platform:   MacOS
- Python version:  Python3.6
- PyTorch version (GPU?):  Torch 1.5 (with and without GPU)


Comments:
Hi, I can't reproduce this on `master`, could you upgrade your library to the latest version and let me know if you face the same issue?

I think torch-1.5.0 and pytorch-transformers-1.2.0 are the latest versions, no?
I upgraded to python3.7 and tried the above again and still get the same issue. 

That's because `pytorch-transformers` became `transformers` in September!

Thanks! It works now. Sorry for asking such a dum question...


No worries, gald you could make it work!

Cool! Problem solved. Will close this issue. 
Nice weekend!

> That's because `pytorch-transformers` became `transformers` in September!

what does that mean? Sorry, I didn't get your point and I have the same issue.

> > That's because `pytorch-transformers` became `transformers` in September!
> 
> what does that mean? Sorry, I didn't get your point and I have the same issue.

Instead of install and use pytorch-transformers, install and use transformers. For examples,

> from transformers import *
Why classifier fine-tuning don't save best model based on the evaluation on dev dataset
Author: nghuyong
Pull request: no
Labels: name: wontfix

I want to use bert to train a classify model, I use the example [run_classifier.py].
But I find that the model will continue to train on train dataset until the max_epoch, without doing the evaluation on the dev dataset and save the best model according to the metric on the dev dataset.
So, the final saved model, it just the last epoch, but this saved model will not be the best model on the dev dataset!
Also, I suggest adding a args --predict and only make predictions.
This work helps me a lot! Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


were you able to fix this problem. If yes can you please tell how
How to init a subclass of BertForTokenClassification
Author: haorannlp
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I want to build a subclass of BertForTokenClassification and also want to use the weights of pretrained model
`
class SeqLabelClassifier(BertForTokenClassification):

    def __init__(self,  pretrained_model_name, config):
        super(SeqLabelClassifier,self).__init__(config)
        self.lstm=nn.LSTM(...)

config = BertConfig()
model = SeqLabelClassifier(pretrained_model_name, config)
model = model.from_pretrained(args.pretrained_model_name, config=config)
`
But I get this error

>   File "/home/haoran/anaconda3/envs/nsd/lib/python3.8/site-packages/transformers/modeling_utils.py", line 466, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() missing 1 required positional argument: 'config'
How to correctly pass the args? 

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pytorch 1.5.0
Author: LysandreJik
Pull request: yes

PyTorch 1.5.0 doesn't allow specifying a standard deviation of 0 in normal distributions. We were testing that our models initialized with a normal distribution with a mean and a standard deviation of 0 had all their parameters initialized to 0. 

This allows us to verify that all weights in the model are indeed initialized according to the specified weights initializations.

We now specify a tiny value (1e10) and round to the 9th decimal so that all these values are set to 0.

closes #3947 
closes #3872

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3973?src=pr&el=h1) Report
> Merging [#3973](https://codecov.io/gh/huggingface/transformers/pull/3973?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/4e817ff41885063e08bb3bcd63e5adfd835b9911&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3973/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3973?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3973      +/-   ##
==========================================
- Coverage   78.44%   78.43%   -0.02%     
==========================================
  Files         111      111              
  Lines       18518    18518              
==========================================
- Hits        14527    14525       -2     
- Misses       3991     3993       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3973?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3973/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `68.49% <0.00%> (-0.37%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3973/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.59% <0.00%> (-0.17%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3973?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3973?src=pr&el=footer). Last update [4e817ff...4bee514](https://codecov.io/gh/huggingface/transformers/pull/3973?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


~~I've seen one error with 1.5 and the new trainer interface (using ner) - I'll raise an issue for that :)~~
[Roberta] fix hard wired pad token id
Author: patrickvonplaten
Pull request: yes

This might break backward compatibility for users that have uploaded a roberta model and use a different pad token than 1 in the configs.



Comments:
I checked all models on `https://huggingface.co/models?search=roberta` and all of them have the pad_token_id set to `1` which was used as a default before. So this is good to merge for me.