noamaanMulla-03/transformers-issues huggingface dataset:
Why does huggingface bert pooler hack make mixed precission training stable?
Author: krishansubudhi
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

## Details

Huggigface BERT implementation has a hack to remove the pooler from optimizer.

https://github.com/huggingface/transformers/blob/b832d5bb8a6dfc5965015b828e577677eace601e/examples/run_squad.py#L927
```
# hack to remove pooler, which is not used
# thus it produce None grad that break apex
param_optimizer = [n for n in param_optimizer if 'pooler' not in n[0]]
```
We are trying to run pretraining on huggingface bert models. The code always diverges later during the training if this pooler hack is not applied. Everytime the reason is apex loss scaler becomes zero. 
After using the above hack, there is no divergence issue seen.

The pooler layer is a FFN with tanh activation
```
class BertPooler(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.activation = nn.Tanh()

    def forward(self, hidden_states):
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token.
        first_token_tensor = hidden_states[:, 0]
        pooled_output = self.dense(first_token_tensor)
        pooled_output = self.activation(pooled_output)
        return pooled_output
```
I even tried replacing the tanh acrivation with GELU and adding layer norm in the pooler layer. But the loss scaler became zero even faster.

My question is why this pooler hack solves numeric instability?


**https://stackoverflow.com/questions/60743907/why-does-huggingface-bert-pooler-hack-make-mixed-precission-training-stable**: 

Comments:
Any update on this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
how does masked_lm_labels  work ?
Author: mahdirezaey
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
Hi all
from hugging face{https://huggingface.co/transformers/model_doc/bert.html#bertformaskedlm}

in this code :
{
from transformers import BertTokenizer, BertForMaskedLM
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)

loss, prediction_scores = outputs[:2]
}

what happens by {outputs = model(input_ids, masked_lm_labels=input_ids)} ?

it will automatically makes [mask] 15% of all the tokens in each sentence of each bs , and calculates loss just for them ?
@thomwolf @tholor

No I don't think so, you need to mask the tokens and then pass them to model, look here (e.g. evaluate) for an example
https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py

@blackcat84 , thanks it helped a lot

@blackcat84 

and one more thing ,
does any function in those scripts , concatenate the short lines to each other ?
in order not to be enforced to pad each line so much

It's been a while so I might be wrong but I think you are correct, I don't remember in which function though. A simple way to be sure about it is to pass a dummy input to the script/function and check it by yourself

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
add TinyBERT?
Author: Dicksonchin93
Pull request: no
Labels: name: wontfix

# 🌟 New model addition

## Model description
TinyBERT is a smaller version of the Base BERT model, it uses transformer distillation (a type of knowledge distillation) to transfer the plenty of knowledge encoded in a large “teacher” BERT to a small “student” TinyBERT. is empirically effective and achieves more than 96% the performance of teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT is also significantly better than state-of-the-art baselines on BERT distillation, with only ∼28% parameters and ∼31% inference time of them. Here I have feature request to add the pretrained weights of TinyBERT after general learning from https://github.com/huawei-noah/Pretrained-Language-Model and model for both TF2.0 and pytorch. I think the transformer distillation method should be introduced too. 
https://arxiv.org/pdf/1909.10351.pdf

<!-- Important information -->

## Open source status

* [x] the model implementation is available: (give details)
https://github.com/huawei-noah/Pretrained-Language-Model, only pytorch is available in my knowledge at the moment
https://github.com/koursaros-ai/nboost
* [x] the model weights are available: (give details)
https://github.com/huawei-noah/Pretrained-Language-Model
* [x] who are the authors: (mention them, if possible by @gh-username)
https://github.com/huawei-noah/Pretrained-Language-Model  @jacobrxz https://github.com/jacobrxz


Comments:
tinybert just a normal bert with smaller parameters. I am not sure whether huggingface team will create new object called `TinyBert`. I think you can simply contact `huawei-noah` first  to get permission to upload tinybert using your personal account.

Or you could ask them if they would create an [org account](https://huggingface.co/organizations) and upload TinyBert there.

I'll also ping them as it would be really great (cc @jacobrxz)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Unused function in squad metrics
Author: Santosh-Gupta
Pull request: no
Labels: name: wontfix

In the squad metrics file, I noticed an unused function `find_all_best_thresh_v2`

https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py#L167

It looks to be pertaining to the squad v2.0 dataset type, which has 'impossible' questions for contexts. 

It looks like it was meant to be used something like

```
    if no_answer_probs:
        if version_2_with_negative:
            find_all_best_thresh_v2(evaluation, preds, exact, f1, no_answer_probs, qas_id_to_has_answer)
        else:
             find_all_best_thresh(evaluation, preds, exact, f1, no_answer_probs, qas_id_to_has_answer)
```

here

https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py#L236

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error in Longformer attention mask using apex mixed precision
Author: wfangtw
Pull request: no
Labels: name: PyTorch

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Longformer

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Install latest transformers (2.10.0) and apex (0.1)
2. Code: 
```
import torch
from transformers import LongformerTokenizer, LongformerModel, LongformerConfig
from apex import amp


tokenizer = LongformerTokenizer.from_pretrained('longformer-base-4096')
config = LongformerConfig.from_pretrained('longformer-base-4096')
model = LongformerModel.from_pretrained('longformer-base-4096').cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

model, optimizer = amp.initialize(model, optimizer, opt_level='O1')

# toy input example
inputs = torch.randint(config.vocab_size, (1, 1024)).cuda() # randomly select tokens with sequence length 1024
mask = torch.ones(1, 1024).cuda() # set mask for every token to local attention
mask[0] = 2. # global attention for first token

outputs = model(inputs, attention_mask=mask)
```



<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Error message:
```
File "/home/miniconda3/envs/test_transformer/lib/python3.8/site-packages/transformers/modeling_longformer.py", line 374, in
forward
    attn[extra_attention_mask_nonzeros[::-1]] = nonzero_selected_attn.view(
RuntimeError: expected dtype Half but got dtype Float
```
`attn` is half precision but is assigned a tensor that is casted into single precision `.type_as(hidden_states)` in line 376 .

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.10.0
- Platform: Linux-4.15.0-65-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Comments:
Good catch! I guess that soon you'd want to move over PyTorch's built-in AMP which takes care of this automatically (I _think_), but for the time being your suggestion is a good fix. You can submit a PR if you want!

I think this is solved with the PR #4574 no?
Error ImportError: cannot import name 'MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING' from 'transformers' (C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\transformers\__init__.py)
Author: kennysmith12
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

I am doing the tutorial ["Quick tour of the fine-tuning/usage scripts"](https://github.com/huggingface/transformers#quick-tour-of-the-fine-tuningusage-scripts)

I downloaded the Glue dataset.
When I try to run this command from pytorch
```
python ./examples/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name MRPC \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir C:/Git/RemoteDGX/MRPC/glue_data/MRPC \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/MRPC/

```
I am getting this error:
ImportError: cannot import name 'MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING' from 'transformers' (C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\transformers\__init__.py)




: 

## To reproduce

Steps to reproduce the behavior:

1.download the Glue database 
2. execute the script run_glue.py

this is the stacktrace:
```
2020-03-25 14:09:19.698135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Traceback (most recent call last):
  File "C:/Git/RemoteDGX/transformers/examples/run_glue.py", line 32, in <module>
    from transformers import (
ImportError: cannot import name 'MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING' from 'transformers' (C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\transformers\__init__.py)
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: windows
- Python version: 3.7
- PyTorch version 1.4.0 without GPU:





Comments:
You're not running the latest version of transformers.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
"Reformer: The Efficient Transformer" looks awesome. I'd love to see it in the library.
Author: Alan-Lee123
Pull request: no
Labels: name: wontfix

# 🌟New model addition

## Model description
Efficient Transformer with locality-sensitive hashing and reversible layers
https://openreview.net/forum?id=rkgNKkHtvB
<!-- Important information -->

## Open Source status

* [. ] the model implementation is available: (give details)
There is an implementation from google
https://github.com/google/trax/blob/master/trax/models/research/reformer.py
* [ ] the model weights are available: (give details)
* [. ] who are the authors: (mention them)
Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
I have started to refactor the original source code in Pytorch if you'd like to help I'd greatly appreciate it! [https://github.com/zbloss/reformer](https://github.com/zbloss/reformer)

I have a working implementation at https://github.com/lucidrains/reformer-pytorch !

Any update on adding this to the library?

They published in their blog about it https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html?m=1

Hence my interest in a huggingface implementation :) 

Looking forward to see this model in the transformers lib :)

Don't think we should rush this one. The reformer paper is pretty tricky to implement in a clean way, plus there aren't any pre-trained models that use it yet. Just one person's opinion, though.

The implementation by @lucidrains seems to work https://github.com/lucidrains/reformer-pytorch ; it'd be cool if it was included in the transformers library. It seems strange to me that no pretrained Reformer has been uploaded since the paper was released, any ideas why? is it possible that it doesn't work in practice as stated by the authors in the paper? Anyone who has trained a Reformer on their own and have tried it to solve a real problem?
Thank you very much in advance

Same here, curious to know why. Thank you!

https://github.com/google/trax/blob/master/trax/models/reformer/machine_translation.ipynb

There should be an pretrained model now.
Would be very happy to see Reformer model in this project.

+1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Closed by @patrickvonplaten 

has this been done ?
Masking of special tokens in masked LM finetuning.
Author: oadams
Pull request: no
Labels: name: wontfix

## 🐛 Bug

roBERTa throws repeated warnings about the absence of special tokens in masked LM fine-tuning with `run_lm_finetuning.py`:

```
 WARNING - transformers.modeling_roberta -   A sequence with no special tokens has been passed to the RoBERTa model. This model requires special tokens in order to work. Please specify add_special_tokens=True in your encoding.
```

This made it look like there was a preprocessing problem, but it appears as though the random masking of tokens applies to the special tokens as well, both for BERT and roBERTa training with that script. It's not clear from the original papers if that's meant to happen, but I assumed it's not. The wording in section 3.3.1 of the BERT paper suggests they might not: They "mask 15% of all _wordpiece_ tokens at random". Their implementation would probably shed light but I just wanted to check with you, since masking of special tokens will affect the representation of [CLS]. Is this masking meant to happen?

Note that BERT does not throw such a warning, but the masking of special tokens also applies to that model.

Model I am using (Bert, XLNet....): BERT and roBERTa

Language I am using the model on (English, Chinese....): English WikiText 2 data.

The problem arise when using:
* [x] the official example scripts: run_lm_finetuning.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: language model fine-tuning with run_lm_finetuning.py
* [] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

Download the wikitext-2 data, and then run:

```
python run_lm_finetuning.py --output_dir=models/roberta_wikitext --model_type=roberta --model_name_or_path=roberta-base --do_train --train_data_file=/Users/oadams/corpora/wikitext-2/wiki.train.tokens --mlm
```

This is basically what's recommended [in the examples](https://huggingface.co/transformers/examples.html)

## Expected behavior

Warning-free training, or a warning that's easy to interpret for the user.

## Environment

* OS: MacOS and Ubuntu.
* Python version: 3.7.4
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 2.0
* Using GPU ? No.
* Distributed of parallel setup ? No
* Any other relevant information:

Comments:
The same issue here, using 4 RTX 2080Ti with Ubuntu 18.04.

This issue exists as the `mask_tokens` function will sometimes replace `
` with a random word. Not sure whether `
` should be masked. A workaround would be adding a line 
```
masked_indices[:, 0] = 0  # tokenizer.bos_token_id
```
right below 
```
masked_indices = torch.bernoulli(torch.full(labels.shape, args.mlm_probability)).bool()
```
https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L111.



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


So the [CLS] and [SEP] are being masked during the training or not?
Create README.md
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4684?src=pr&el=h1) Report
> Merging [#4684](https://codecov.io/gh/huggingface/transformers/pull/4684?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0866669e751bef636fa693b704a28c1fea9a17f3&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4684/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4684?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4684   +/-   ##
=======================================
  Coverage   77.14%   77.15%           
=======================================
  Files         128      128           
  Lines       21070    21070           
=======================================
+ Hits        16255    16256    +1     
+ Misses       4815     4814    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4684?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4684/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.83% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4684?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4684?src=pr&el=footer). Last update [0866669...e12a141](https://codecov.io/gh/huggingface/transformers/pull/4684?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
End-to-end object detection with Transformers
Author: parmarsuraj99
Pull request: no
Labels: name: New model

# 🚀 Feature request

A modular Transformer Encoder-Decoder block that can be attached to ConvNets for many tasks.

## Motivation
As shown in the recent paper [End-to-end Object Detection with Transformers](https://ai.facebook.com/research/publications/end-to-end-object-detection-with-transformers) which used Transformer for Object Detection. [https://github.com/facebookresearch/detr](https://github.com/facebookresearch/detr)
## Your contribution


Comments:
I might be wrong, but I think the focus of this library is on NLP which at most is multimodal. Also including object detection transformers may fall out of the scope of this project.
Model evaluated at each checkpoint, but results not in checkpoint file
Author: CMobley7
Pull request: no

# ❓ Questions & Help

I start `run_language_modeling.py` and `run_glue.py` with `--do_eval` and `--evaluate_during_training` arguments. While it checkpoints and evaluates the model at each saving step, the evaluation results are not output to the checkpoint folder but merely the terminal at each saving step, as well as logging step. However, the final model is evaluated and its results placed in the appropriate location. I'd like the performance of each checkpoint. Am I doing something wrong? Is there an additional argument I must specify?

Comments:
What is the proper way to get an `eval.txt` file for each checkpoint?

No, this is not a built-in feature. I'd suggest you install from source and modify the code directly. 

The code in this repo is meant to be optimized for "hackability", feel free to open a new issue if needed.