noamaanMulla-03/transformers-issues huggingface dataset:
Fixed typo in docs README
Author: dveselov
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1398?src=pr&el=h1) Report
> Merging [#1398](https://codecov.io/gh/huggingface/transformers/pull/1398?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/391db836ab7ed2ca61c51a7cf1b135b6ab92be58?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1398/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1398?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1398   +/-   ##
=======================================
  Coverage   84.72%   84.72%           
=======================================
  Files          84       84           
  Lines       12591    12591           
=======================================
  Hits        10668    10668           
  Misses       1923     1923
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1398?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1398?src=pr&el=footer). Last update [391db83...cd69bc9](https://codecov.io/gh/huggingface/transformers/pull/1398?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


👍
DistilBert for Tensorflow doesn't work
Author: p-christ
Pull request: no

Model:  TFDistilBertForSequenceClassification
Language: English 
Task: multi-label classification
Environment: google colab

When trying to use TF Distil Bert I get the below error after I have loaded the model and try to run model.fit() : 

> TypeError: in converted code:
>     relative to /usr/local/lib/python3.6/dist-packages:
> 
>     transformers/modeling_tf_distilbert.py:680 call  *
>         distilbert_output = self.distilbert(inputs, **kwargs)
>     tensorflow_core/python/keras/engine/base_layer.py:842 __call__
>         outputs = call_fn(cast_inputs, *args, **kwargs)
>     transformers/modeling_tf_distilbert.py:447 call  *
>         tfmr_output = self.transformer([embedding_output, attention_mask, head_mask], training=training)
>     tensorflow_core/python/keras/engine/base_layer.py:891 __call__
>         outputs = self.call(cast_inputs, *args, **kwargs)
>     transformers/modeling_tf_distilbert.py:382 call
>         layer_outputs = layer_module([hidden_state, attn_mask, head_mask[i]], training=training)
>     tensorflow_core/python/keras/engine/base_layer.py:891 __call__
>         outputs = self.call(cast_inputs, *args, **kwargs)
>     transformers/modeling_tf_distilbert.py:324 call
>         sa_output = self.attention([x, x, x, attn_mask, head_mask], training=training)
>     tensorflow_core/python/keras/engine/base_layer.py:891 __call__
>         outputs = self.call(cast_inputs, *args, **kwargs)
>     transformers/modeling_tf_distilbert.py:229 call
>         assert 2 <= len(tf.shape(mask)) <= 3
>     tensorflow_core/python/framework/ops.py:741 __len__
>         "shape information.".format(self.name))
> 
>     TypeError: len is not well defined for symbolic Tensors. (tf_distil_bert_for_sequence_classification_1/distilbert/transformer/layer_._0/attention/Shape_2:0) Please call `x.shape` rather than `len(x)` for shape information.


The exact same procedure works if I use TF Bert but not Distil Bert.  Does anyone know how to get around this problem?

Comments:
I have been experiencing the same issue #1378.

Fixed on master with 23b7138, thanks.
Will be in this week's new release 2.1

thanks a lot
Issue with `decode` in the presence of special tokens
Author: harkous
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT-2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce


Steps to reproduce the behavior:

Run the following:

```bash
from transformers.tokenization_gpt2 import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.add_special_tokens({"sep_token": "[SEP]"})
# this works, outputting "[SEP]"
tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(tokenizer.encode("[SEP]")))
# this fails
tokenizer.decode(tokenizer.encode("[SEP]"))
```

The last command gives this error:
```
miniconda3/envs/deepnlg/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 937, in decode
    text = text.replace(self._cls_token, self._sep_token)
TypeError: replace() argument 1 must be str, not None
```
<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

The expectation is that [SEP] is output from the `decode` function.

## Environment

* OS: OSX
* Python version: 3.7
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): Master (2dc8cb87341223e86220516951bb4ad84f880b4a)
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

Comments:
Can't reproduce this on master now. Seems to be fixed.

Thanks a lot. It seems to be fixed. Now I get `'[SEP]'` and `' [SEP]'` consecutively with the first and the second command above. So we can close this issue.
Plan for Albert?
Author: rush86999
Pull request: no

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->

I think Albert is popular enough to not say anything more. The link to the paper is below.
https://arxiv.org/pdf/1909.11942v1.pdf

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->


Comments:
Duplicate of #1370
XLNet - Finetuning - Layer-wise LR decay
Author: yukioichida
Pull request: no

## ❓ Questions & Help
I'm trying to finetuning a XLNet using run_glue.py, but i haven't seen any references about **Layer-wise lr decay**, that were commented by the authors in the paper. 

- Where can I set this parameter on finetuning optimizer?
- The *linear learning rate decay* commented in the paper is related to Warmup Scheduler ?(considering that after warmup_steps is reached, the lr rate begins to decay) 

References:
(https://arxiv.org/pdf/1906.08237.pdf - page 16)


Comments:
No this means the layer rate is smaller deeper in the network, what is called "discriminative learning" in ULMFiT.
Check our NAACL Tutorial on Transfer Learning for more details, in particular, Hands-on n°5 slide 163 here: https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/edit?ts=5c8d09e7#slide=id.g5888218f39_54_89 

Thanks @thomwolf for the answer.
pretrained model(s) in onnx format
Author: WilliamTambellini
Pull request: no
Labels: name: Help wanted

name: wontfix

hi, would you assist/help in order to export/convert at least 1 model into the onnx format ?
https://onnx.ai
kind

Comments:
Hi @WilliamTambellini, have you tried to follow the standard ONNX procedure for converting a PyTorch model?
The model in this repo are just regular PyTorch models.

Hello Thomas, I ve not yet tried, just seen : 
https://github.com/onnx/models/issues/130
https://stackoverflow.com/questions/54220042/how-do-you-generate-an-onnx-representation-of-a-pytorch-bert-pretrained-neural-n
Will try, tks. 

Hi, when I try to export a TokenClassification model to a ONNX model, I encounter `RuntimeError: ONNX export failed: Couldn't export operator aten::erf`, does that mean some part of BERT model layers not supported by ONNX?
I think that problem comes from the definition of GELU function, which is `x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))`. Should I try to use other way to calculate this function or wait for ONNX to support this opertator? 

@geekboood update your pytorch version to latest and the problem will most likely go away.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


for anyone who is looking for the answer:
torch=1.1.0 
python=3.6

`torch.onnx.export(model, (input_ids, segment_ids, input_mask), "bert.onnx", verbose=False)`

works well for me

> for anyone who is looking for the answer:
> torch=1.1.0
> python=3.6
> 
> `torch.onnx.export(model, (input_ids, segment_ids, input_mask), "bert.onnx", verbose=False)`
> 
> works well for me

Hi, thanks for the answer. Do you get good results when using the exported model for inference in another framework? I exported a BertForQuestionAnswering model to ONNX without errors, but I'm getting wrong predictions when using onnxruntime or a second export to TF Serving and I can't figure out why!

Not sure if this is still an issue for you but in the BertForSequenceClassification model the parameters are in a different order

`torch.onnx.export(model, (input_ids, input_mask, segment_ids), "bert.onnx", verbose=False)`

works as intended

@chessgecko wow you're right, thanks! working now

cc @mfuntowicz :)
Checkpoint rotation
Author: jinoobaek-qz
Pull request: yes

By default, no change in existing behavior. However, if you pass in an argument with `save_total_limit` flag and a natural number as value, then, your machine might not run out of space when fine-tuning. Because, it will only keep the latest `save_total_limit` number of checkpoints and delete the older checkpoints.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1429?src=pr&el=h1) Report
> Merging [#1429](https://codecov.io/gh/huggingface/transformers/pull/1429?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8fcc6507ce9d0922ddb60f4a31d4b9a839de1270?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1429/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1429?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1429   +/-   ##
=======================================
  Coverage   84.72%   84.72%           
=======================================
  Files          84       84           
  Lines       12591    12591           
=======================================
  Hits        10668    10668           
  Misses       1923     1923
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1429?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1429?src=pr&el=footer). Last update [8fcc650...18c51b7](https://codecov.io/gh/huggingface/transformers/pull/1429?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


That's a nice addition, thanks!
`do_wordpiece_only` argument
Author: bkj
Pull request: no
Labels: name: wontfix

A `do_wordpiece_only` argument is referenced [here](https://github.com/huggingface/pytorch-transformers/blob/fec76a481d1ecfbf068d87735dd44ffc26158f6e/pytorch_transformers/tokenization_bert.py#L97) -- does that argument actually exist?  I'm not able to find it in the repo anywhere.

Related, is this expected behavior?
```python
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', never_split=['[unused0]'])
>>> tokenizer.tokenize('[CLS] [unused0] this is a [SEP] test')
['[', 'cl', '##s', ']', '[unused0]', 'this', 'is', 'a', '[', 'sep', ']', 'test']
>>> 
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> tokenizer.tokenize('[CLS] [unused0] this is a [SEP] test')
['[CLS]', '[', 'unused', '##0', ']', 'this', 'is', 'a', '[SEP]', 'test']
```

I want to be able to use the `[unused*]` tokens in `BertTokenizer`, but it seems like adding them to the `never_split` has some unexpected side effects.  Anyone have any ideas on how to set up the tokenizer to use the `[unused*]` tokens?  I'd prefer not to have to add the indices in a seperate step after the tokenization if possible.

__Edit:__ Seems like maybe you have to do
```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', never_split=['[UNK]', '[SEP]', '[PAD]', '[CLS]', '[MASK]', '[unused0]'])
```
and then behavior makes more sense -- is that right?

Thanks!

Comments:
Use _additional_special_tokens_  instead.

```
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', additional_special_tokens=['[unused0]'])
>>> tokenizer.tokenize('[CLS] [unused0] this is a [SEP] test')
['[CLS]', '[unused0]', 'this', 'is', 'a', '[SEP]', 'test']
```


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error while fine-tuning model for GPT2
Author: dasavisha
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using GPT2:

Language I am using the model on English:

The problem arise when using:
* [ ] the official example scripts: I run the run_lm_finetuning.py script from the examples folder

The tasks I am working on is:
* [ ] my own task or dataset: The WritingPrompts dataset.

## To Reproduce

Steps to reproduce the behavior:

1. Running the run_lm_generation.py on the WP dataset. 
 python run_lm_finetuning.py --output_dir=output_ft_gpt2 --model_type=gpt2 --model_name_or_path=gpt2 --do_train --train_data_file=../stories_data/writingPrompts/train.wp_target --do_eval --eval_data_file=../stories_data/writingPrompts/test.wp_target --block_size=128 --save_total_limit=100


Error: Traceback (most recent call last):
  File "run_lm_finetuning.py", line 538, in <module>
    main()
  File "run_lm_finetuning.py", line 485, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "run_lm_finetuning.py", line 97, in load_and_cache_examples
    dataset = TextDataset(tokenizer, file_path=args.eval_data_file if evaluate else args.train_data_file, block_size=args.block_size)
  File "run_lm_finetuning.py", line 80, in __init__
    self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i:i+block_size]))
AttributeError: 'GPT2Tokenizer' object has no attribute 'build_inputs_with_special_tokens'





Comments:
Hi! It seems like you have taken the example from the latest version but that your library is not up to date. Could you tell me the version of your `transformers` library ?

Hi! 
I am using the version 2.0.0.

If you´re using the version 2.0.0 you should use the [script that was used in this version](https://github.com/huggingface/transformers/blob/v2.0.0/examples/run_lm_finetuning.py). The current script works on version 2.1!

Thanks!
How much GPU memory is needed to run run_squad.py
Author: atinesh
Pull request: no

## ❓ Questions & Help

How much GPU memory is needed to run `run_squad.py`, I tried on `GTX 1050ti (4gb)` with the following setting and I am getting out of memory error

```
$ python3 examples/run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/ \
  --save_steps 1000
```

Comments:
With 4 GB you're bound to issues with a batch size of 12. You could figure out the total memory usage of the model + calculate the memory footprints of tensors to determine the biggest batch size that would fit on your GPU. 

Specifying a smaller batch size (like 1 or 2) would let you run the script, though.

> With 4 GB you're bound to issues with a batch size of 12. You could figure out the total memory usage of the model + calculate the memory footprints of tensors to determine the biggest batch size that would fit on your GPU.
> 
> Specifying a smaller batch size (like 1 or 2) would let you run the script, though.

Hello @LysandreJik I am able to run the code with `batch size = 1`
How can I get the transformers' parameters?
Author: YongtaoGE
Pull request: no

## ❓ Questions & Help
Hi, I am new to transformers. Does this library offer an interface to compute the total number of different model's parameters?


Comments:
The models we use inherit directly from `torch.nn.Module` for our pytorch models and `tf.keras.layers.Layer` for tensorflow modules. You can therefore get the total number of parameters as you would do with any other pytorch/tensorflow modules:

`sum(p.numel() for p in model.parameters() if p.requires_grad)` for pytorch and
`np.sum([np.prod(v.shape) for v in tf.trainable_variables])` for tensorflow, for example.

Got it!
Re-order XLNet attention head outputs for better perf
Author: slayton58
Pull request: yes

Significant performance boost over the original orderings. On an already somewhat optimised branch this gave me > 2x end-to-end throughput on a squad xlnet fine-tuning task (batch 8, seq-length 512, fp16, amp opt level = O2)

Justifying this is the contraction 
```
attn_vec = torch.einsum('bnij,jbnd->ibnd', attn_prob, v_head_h)
```

Given how `torch.einsum` and tensor contractions work, this is a batched gemm with batch dimension `bn` and gemm dimension `(i x j) * (j x d)`. Moving `bn` to be the first dimensions for the first input eliminates a sizable transpose that would otherwise need to be done.

Comments:
Remaining CI failures are valid, they look to assume a `ijbn` ordering for all attention-based things, which no longer holds.

I'm happy to add additional functionality to get these tests passing, but I'd like input on how you'd like that done (I'd lean to passing an optional `expected_attention_size` which is `[key_len, batch_size, num_heads]` by default, and checking that instead of assembling the expected sizes on-the-fly in the test(s))

Please ignore above :) test errors were due to a missed transpose on my part (if attention outputs are returned, they need to be transposed from `bnij` to `ijbn` ordering to keep the interface from attention <-> the rest of the code the same as before.)

This is a great work, thanks a lot @slayton58.

Can you confirm this has no noticeable impact on downstream performances (in terms of evaluation metrics), for instance on your SQuAD tests?

@thomwolf I have been testing against the config from https://github.com/huggingface/transformers/issues/947#issue-476001056 with seq-length=512, and obtained a consistent f1 score of 83 across both `ijbn` and `bnij` attention head orderings (also across fp32/fp16 O1/fp16 O2)

When there's a PR issued for the changes in https://github.com/huggingface/transformers/issues/947#issuecomment-535989890 I'd be happy to go ahead and repro those numbers with this change if you'd like the additional security.

Awesome, ok let's merge this then.
XLnet sentence vector
Author: luv4me
Pull request: no
Labels: name: wontfix

how can i get the XLnet sentence vector by pytorch-transformers. I use the sample but I only get the word vector. it drives me crazy

Comments:
You can train the model on a downstream task to get a sentence vector related to your task or you can get a sentence vector by averaging or max-pooling the output sequence of token hidden-states.

Try doing:

```Python
model = model_class.from_pretrained(pretrained_weights,
                                    output_hidden_states=True,
                                    output_attentions=True)

sequence_summary = SequenceSummary(model.config)

es = torch.tensor([tokenizer.encode(s)])
# sentence embedding
t = sequence_summary(model(es)[0])

A simple strategy is to take the concatentation of the last hidden state, the mean-pooling, and the max-pooling (tends to be a reasonably good baseline pooling strategy, e.g. [ULMFit](https://arxiv.org/pdf/1801.06146.pdf))

@cpcdoy hey, and I have a question why the sentence is not stationary.

So, we should pool over all hidden states, and not just use the hidden state corresponding to `[CLS]`?

I would go with @rishibommasani's solution for a general "semantic" sentence embeddings or fine-tuning and using the `[CLS]` for a task-specific sentence embedding.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
problem when calling resize_token_embeddings
Author: LeeJuly30
Pull request: no
Labels: name: wontfix

When calling resize_token_embeddings, the model actually only modifies its embedding and decoder weight, whlie the decoder bias is unchanged. So whenever the forward function is called, the following error will be raised.
```python
RuntimeError: The size of tensor a (21215) must match the size of tensor b (21128) at non-singleton dimension 2
```

Comments:
Which model were you resizing?

I'm working on chinese BertForPreTraining model

This is strange because Bert's LM head has no bias...

Would need to have a more complete error message to be able to understand.

```python
class BertLMPredictionHead(nn.Module):
    def __init__(self, config):
        super(BertLMPredictionHead, self).__init__()
        self.transform = BertPredictionHeadTransform(config)

        # The output weights are the same as the input embeddings, but there is
        # an output-only bias for each token.
        self.decoder = nn.Linear(config.hidden_size,
                                 config.vocab_size,
                                 bias=False)

        self.bias = nn.Parameter(torch.zeros(config.vocab_size))

    def forward(self, hidden_states):
        hidden_states = self.transform(hidden_states)
        hidden_states = self.decoder(hidden_states) + self.bias
        return hidden_states
```
```python
self.bias = nn.Parameter(torch.zeros(config.vocab_size))
```
For example, we load the pretrained model whose vocab size is 23189, and i add 1000 tokens and call resize_token_embeddings. Since the decoder weight is actually embedding weight, they are reshaped to (24189, hidden_size), but the bias is still 23189.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add new BERT models for German (cased and uncased)
Author: stefan-it
Pull request: yes

Hi,

this PR adds new BERT models for German (both cased and uncased) from @dbmdz.

Details can be found in [this repository](https://github.com/dbmdz/german-bert).

Tasks:

* [x] Models are stored on S3, only permissions need to be adjusted by @julien-c 

Comments:
Great, ok all the models should be public. Merging this now.
Awesome work @stefan-it!