noamaanMulla-03/transformers-issues huggingface dataset:
Create README.md for xlnet_large_squad
Author: ahotrod
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2942?src=pr&el=h1) Report
> Merging [#2942](https://codecov.io/gh/huggingface/transformers/pull/2942?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/53ce3854a16ad2a715bc6ac8af3e30c18b5a1d11?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2942/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2942?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #2942   +/-   ##
======================================
  Coverage    76.1%   76.1%           
======================================
  Files          98      98           
  Lines       15946   15946           
======================================
  Hits        12136   12136           
  Misses       3810    3810
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2942?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2942/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `89.93% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2942/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `95.71% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2942/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.15% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2942/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `94.27% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2942/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `81.54% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2942?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2942?src=pr&el=footer). Last update [53ce385...cfa068f](https://codecov.io/gh/huggingface/transformers/pull/2942?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
documentation for TF models mentions non-existent methods
Author: btel
Pull request: no
Labels: name: Documentation

Documentation of `TFPreTrainedModel.from_pretrained` method mentions the `.model()` and `.eval()` methods that are not defined for tensorflow models:

> The model is set in evaluation mode by default using ``model.eval()`` (Dropout modules are deactivated)
> To train the model, you should first set it back in training mode with ``model.train()``

https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_utils.py#L195


Comments:
Regarding attention size returned by the model
Author: divyag11
Pull request: no
Labels: name: TensorFlow

name: Core: Modeling

hi, while doing output_attentions - True in the huggingface model.it return attention of size:(no. of heads,seq length,seq length)
can we configure it to return only the attention of the last 2 heads of the model.
please let me know.


Comments:
Hi,
which model do you use? Can't you simply remove the output of the other heads?

I am using TFDistilbertmodelforsequenceclassification.
You are right,i can remove other head attention,but while using tfserving,it's taking a lot time ,since the output attentions has huge dimension.
So,that's why ,i was asking if i could get just last 2 head attention instead of getting all head attentions while hitting tfserving

Unfortunately, we don't have a way to do that right now.
Convert BERT to RoBERTa
Author: etetteh
Pull request: no
Labels: name: Core: Modeling

name: Usage

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

Given that RoBERTa outperformed BERT on several tasks, yet having a slight architecture modification, I want to know if it is possible to convert a BERT pretrained model to RoBERTa. 

I am working with a BERT pretrained model on a domain specific corpora, given that I don't have the resources to train RoBERTa from scratch I want to know if I can convert the model to RoBERTa. If yes, how do I go about it?

Comments:
The differences between the BERT and RoBERTa model are the following:

- Different pre-training (larger batch size for RoBERTa, no NSP, no token type ids ...)
- Different tokenizer

The model architecture is exactly the same. The only real difference after the pre-training is the difference in tokenization. Since you're working with a BERT model that was pre-trained, you  unfortunately won't be able to change the tokenizer now from a WordPiece (BERT) to a Byte-level BPE (RoBERTa).
Added CamembertForQuestionAnswering
Author: maximeilluin
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2746?src=pr&el=h1) Report
> Merging [#2746](https://codecov.io/gh/huggingface/transformers/pull/2746?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2184f87003c18ad8a172ecab9a821626522cf8e7?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2746/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2746?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #2746      +/-   ##
=========================================
+ Coverage    75.1%   75.1%   +<.01%     
=========================================
  Files          93      93              
  Lines       15249   15253       +4     
=========================================
+ Hits        11452   11456       +4     
  Misses       3797    3797
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2746?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/2746/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `98.87% <ø> (ø)` | :arrow_up: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/2746/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `29.18% <ø> (ø)` | :arrow_up: |
| [src/transformers/modeling\_camembert.py](https://codecov.io/gh/huggingface/transformers/pull/2746/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jYW1lbWJlcnQucHk=) | `100% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2746?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2746?src=pr&el=footer). Last update [2184f87...74c277a](https://codecov.io/gh/huggingface/transformers/pull/2746?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@julien-c Could you please review ;)
CTRL tokenizer has no special tokens to indicate EOS
Author: erickrf
Pull request: no

## 🐛 Bug

The `generate` method from `PreTrainedModel` by default uses index 0 as EOS. This is a problem with CTRL, because its tokenizer has the word `the` mapped to this id.

Actually the CTRL has no special tokens besides UNK:

```
tokenizer = CTRLTokenizer.from_pretrained('ctrl')
tokenizer.special_tokens_map                                                                                                                                                                             
# {'unk_token': '<unk>'}

tokenizer.convert_ids_to_tokens([0])                                                                                                                                                                     
# ['the']
```

I believe the CTRL tokenizer should have some special token to use as EOS or PAD, as the other models do.


Comments:
Conda version is not the latest
Author: kamyarghajar
Pull request: no
Labels: name: wontfix

## 🚀 Feature

The conda package in conda forge channel (v2.1.1) is not the latest released version (v2.2.2) so the ALBERT model is missing from the package.

## Motivation

In conda environment we need the latest packages containing the ALBERT model.

## Additional context

Also, for the ALBERT model in its code it has printing comments for each 12 layers. It shouldn't be there I guess in a production ready version.


Comments:
Do you mean lines 66-67 and 153 in the [modeling_albert.py](https://github.com/huggingface/transformers/blob/master/transformers/modeling_albert.py) script?

> ##  Feature
> The conda package in conda forge channel (v2.1.1) is not the latest released version (v2.2.2) so the ALBERT model is missing from the package.
> 
> ## Motivation
> In conda environment we need the latest packages containing the ALBERT model.
> 
> ## Additional context
> Also, for the ALBERT model in its code it has printing comments for each 12 layers. It shouldn't be there I guess in a production ready version.

@TheEdoardo93 in the last pip package in line 289 there was `print("Layer index", layer_index)` that is removed right now.

The last pip package (2.2.2) should not be outputting the layers. Could you please tell me what is the output in your console when running the following snippet?

```py
from transformers import AlbertModel, __version__
import torch

print(__version__)
model = AlbertModel.from_pretrained("albert-base-v1")
output = model(torch.tensor([[1,2,3]]))
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BERT model breaks during FP16 Apex training on the latest update (2.5.0) - due to gelu function
Author: Laksh1997
Pull request: no
Labels: name: PyTorch

name: Should Fix

# 🐛 Bug

BERT breaks during FP16 training due to the gelu function.

```
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 407, in forward

hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__

result = self.forward(*input, **kwargs)

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 379, in forward

intermediate_output = self.intermediate(attention_output)

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__

result = self.forward(*input, **kwargs)

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 332, in forward

hidden_states = self.intermediate_act_fn(hidden_states)

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py", line 1125, in gelu

return torch._C._nn.gelu(input)

RuntimeError: "GeluCUDAKernelImpl" not implemented for 'Half'
```

## Information

The reason this is happening is because in `modeling_bert.py`, in 2.4.1 we had:
```
def gelu(x):
    """ Original Implementation of the gelu activation function in Google Bert repo when initially created.
        For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
        0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
        Also see https://arxiv.org/abs/1606.08415
    """
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))

...

ACT2FN = {"gelu": gelu, "relu": torch.nn.functional.relu, "swish": swish, "gelu_new": gelu_new, "mish": mish}
```
whereas in 2.5.0 we have:
```
ACT2FN = {"gelu": gelu, "relu": torch.nn.functional.relu, "swish": swish, "gelu_new": gelu_new, "mish": mish}
```
where `gelu` now comes from `activations.py` as:
`gelu = getattr(F, "gelu", _gelu_python)` on line 21.


## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?): 1.2.0 CUDA 10.0
- Tensorflow version (GPU?): 2.5.0
- Using GPU in script?: V100
- Using distributed or parallel set-up in script?: No, but using Apex FP16 training


Comments:
Bias in `BertLMPredictionHead ` is added twice
Author: jzbjyb
Pull request: no

# 🐛 Bug

According to this PR #2521, a link is created between the linear layer bias and the model attribute bias in `BertLMPredictionHead`. In the `__init__`:
```python
self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
self.bias = nn.Parameter(torch.zeros(config.vocab_size))
self.decoder.bias = self.bias  # here is the problem
```
in the `forward`:
```
hidden_states = self.decoder(hidden_states) + self.bias
```
I am afraid this will cause the `self.decoder` (which is a `nn.Linear`) to have bias and as a result the bias is added twice in the `forward` function.

## To reproduce

Steps to reproduce the behavior:

For version 2.4.0, where the PR is merged and thus has this bug:
```python
from transformers import *
model = BertForMaskedLM.from_pretrained('bert-base-cased')
print(model.cls.predictions.bias)
# tensor([-0.1788, -0.1758, -0.1752,  ..., -0.3448, -0.3574, -0.3483], requires_grad=True)
print(model(torch.tensor([[0]])))
# (tensor([[[-12.2630, -12.2798, -12.1221,  ..., -10.2729, -10.8859, -11.1733]]], grad_fn=<AddBackward0>),)
```
For version 2.3.0, which is before the PR being merged and thus no bug:
```python
from transformers import *
model = BertForMaskedLM.from_pretrained('bert-base-cased')
print(model.cls.predictions.bias)
# tensor([-0.1788, -0.1758, -0.1752,  ..., -0.3448, -0.3574, -0.3483], requires_grad=True)
print(model(torch.tensor([[0]])))
# (tensor([[[-12.0842, -12.1040, -11.9469,  ...,  -9.9281, -10.5285, -10.8251]]], grad_fn=<AddBackward0>),)
```
Comparing the above output, you can clearly see that for version 2.4.0, the bias is added twice.

## Environment info
- `transformers` version: 2.4.0
- Platform: Ubuntu 
- Python version: 3.7
- PyTorch version (GPU?): 1.0.1 (with GPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
This is a known issue (cf. https://github.com/huggingface/transformers/pull/2928). It will be fixed in another (cleaner) PR, though.
Migrating from `pytorch-pretrained-bert` to `pytorch-transformers` issue regarding model() output
Author: ArashAskary
Pull request: no

I'm having trouble migrating my code from `pytorch_pretrained_bert` to `pytorch_transformers`. I'm attempting to run a cosine similarity exercise. I want to extract text embeddings values of the second-to-last of the 12 hidden embedding layer. 

```

import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel
#from pytorch_transofmers import BertTokenizer, BertModel
import pandas as pd
import numpy as np

model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# This is done by default in the pytorch_transformers
model.eval() 

input_query = "This is my test input query text"
marked_text = "[CLS] " + input_query + " [SEP]"
tokenized_text = tokenizer.tokenize(marked_text)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [1] * len(tokenized_text)
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
with torch.no_grad():
    encoded_layers, _ = model(tokens_tensor, segments_tensors)
    sentence_embedding = torch.mean(encoded_layers[10], 1)

```
Using the pytorch_pretrained_bert works perfectly fine with the above code. My `encoded_layers` object is a list of 12 hidden layer tensors, allowing me to pick and reduce the 11th layer by taking an average, resulting in `sentence_embedding` object I can run cosine similarities against.

However, when I migrate my code to the `pytorch_transformers` library, the resulting `encoded_layers` object is no longer the full list of 12 hidden layers, but a single torch tensor object of shape `torch.Size([1, 7, 768])`, which results in the following error when I attempt to create the `sentence_embedding` object:

```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-23-7f877a7d2f9c> in <module>
      9         encoded_layers, _ = model(tokens_tensor, segments_tensors)
---> 10         sentence_embedding = torch.mean(test[10], 1)
     11 

IndexError: index 10 is out of bounds for dimension 0 with size 7
```

The migration documentation (https://huggingface.co/transformers/migration.html) states that I should take the first element of the `encoded_layers` object as a replacement but that does not provide me with access to the second to last hidden layer of embeddings. 

How can I access it?

Thank you!

Comments:
You need to tell the model that you wish to get all the hidden states

```python
model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
```

Then, you'll find your expected output as the third item in the output tuple:
```python
 encoded_layers = model(tokens_tensor, segments_tensors)[2]
```

IIRC those layers now also include the embeddings (so 13 items in total), so you might need to update the index to get the second last layer. Might be better to use a negative index to be sure (-2).
End-Task Distillation with DistilBERT
Author: shreydesai
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

The DistilBERT paper notes the IMDB and SQuAD results were obtained "with a second step of distillation during fine-tuning". What does this involve exactly and how can it be performed with the DistilBERT model in this repo?

Comments:
Hello @shreydesai,
You should have a look at [run_squad_w_distillation.py](https://github.com/huggingface/transformers/blob/master/examples/distillation/run_squad_w_distillation.py) which is the script used in the experiment you are mentioning.
Victor

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
XLNet fine-tuning speed (Multi-label classification)
Author: yunju63
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi!

I'm wondering whether my speed of fine-tuning is normal.

Training is taking **2~3hours per epoch**.

I am using [fast-bert](https://github.com/kaushaltrivedi/fast-bert) to train multi-label classifier, with
- model = xlnet-base-cased
- max sequence length = 512 tokens
- mem states using = no
- gpu = V100 * 4
- training data amount = 110000 ~ 120000
- validation data amount = 30000 ~ 40000
- evaluation = per each epoch, per every 100 steps
- apex, f16 used

Thank you so much!






Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update modelcard of bert-base-german-cased
Author: tholor
Pull request: yes
Labels: name: model card

Add image

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2951?src=pr&el=h1) Report
> Merging [#2951](https://codecov.io/gh/huggingface/transformers/pull/2951?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/3e98f27e4a9bb0ac3d0fe24b94d30da42cdae8a7?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2951/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2951?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #2951   +/-   ##
======================================
  Coverage    76.1%   76.1%           
======================================
  Files          98      98           
  Lines       15946   15946           
======================================
  Hits        12136   12136           
  Misses       3810    3810
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2951?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2951?src=pr&el=footer). Last update [3e98f27...d3f1583](https://codecov.io/gh/huggingface/transformers/pull/2951?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


👍