noamaanMulla-03/transformers-issues huggingface dataset:
[CI] Don't test apex
Author: sshleifer
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6021?src=pr&el=h1) Report
> Merging [#6021](https://codecov.io/gh/huggingface/transformers/pull/6021?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/3996041d0ae23ce23dfb8a343e6344f2f8d54c16&el=desc) will **increase** coverage by `0.39%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6021/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6021?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6021      +/-   ##
==========================================
+ Coverage   78.29%   78.68%   +0.39%     
==========================================
  Files         146      146              
  Lines       26249    26249              
==========================================
+ Hits        20552    20655     +103     
+ Misses       5697     5594     -103     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6021?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/6021/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `43.98% <0.00%> (-49.38%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6021/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.21% <0.00%> (ø)` | |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/6021/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.25% <0.00%> (+1.29%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/6021/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `95.18% <0.00%> (+74.91%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6021?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6021?src=pr&el=footer). Last update [3996041...14292a2](https://codecov.io/gh/huggingface/transformers/pull/6021?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Avoid abort due to missing paths in case of '--save_total_limit' argument
Author: TJKlein
Pull request: yes
Labels: name: wontfix

Checkpoint path will be deleted when using --save_total_limit. torch.save() would not be able to store and abort.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Scaling text classification / reusing models
Author: timsuchanek
Pull request: no
Labels: name: wontfix

If I have a system, where I want to train many text classifiers for many users, how could I go about it with the transformers library in a scalable way?
Right now I would have to run let's say a 10min training per user on a RTX 2080 ti for Albert for the dataset I have. That doesn't scale if I have thousands of users.

If I understand correctly, in the sequence classification models, the whole transformer model is being trained, so the backpropagation happens through the whole network.

However, if I now want to reuse the model for another user, maybe just passing in a bit more labeled data to customize a base classifier, how could I go about that?

It seems to me, that I would have to basically "freeze" the whole "Bert" model, no matter which one I would use, and then only train a thin layer on top.

One possibility I see would be KNN using sentence transformers, I already asked in the repo there https://github.com/UKPLab/sentence-transformers/issues/209


Maybe someone here has an idea which approach would make sense for such a situation.

Thanks!

Comments:
You can pretty easily "freeze" parameters you don't want to backpropagate against, in PyTorch: 

```python
for param in parameters:
    param.requires_grad = False
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Dealing with class imbalance 
Author: al-yakubovich
Pull request: no
Labels: name: wontfix

Are there any built-in methods for dealing with class imbalance in BERT?

Comments:
Did you find anything on this? Been digging around for the same, seems like `from_pretrained` used to allow `weight` as an arg?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GPT2 perplexity rolling/striding way for evaluating a document.
Author: sb1992
Pull request: no
Labels: name: wontfix

As I understand GPT2 uses TextDataset as a loader and it produces the example list in block sizes. So say we have a sentence "**we are in a climate crisis**" and have block size 3. So this will produce the example list as 

`ex = [["we","are","in"],["a","climate","crisis"]]`

So in such a scenario when calculating overall perplexity for the document the word "a" has no previous context and "climate" only has "a" as context. What I would want would ideally the context to be a in rolling/striding way. So  I edited the text loader to produce a list like:

`ex = [["we","are","in"],["are","in","a"],["in,"a","climate],["a","climate","crisis"]]`

Now, if I calculate perplexity for this ex list surely a lot of words will be counted multiple times as they are in the lists in multiple times. But in in a way for every instance from 2nd onwards (ex[1] above) i Would only want to consider last word for perplexity/loss calculation.  So my query is how to tackle it, should I use attention mask so that say in the case of 

`ex[1] =["are","in","a"] ` 
with mask of [0,0,1] for loss calculation (and hence perplexity) it only takes "a" into account but for getting the context of "a' will also take previous 2 words in account?

Any help on the best way to approach this problem will be much appreciated.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Get BERT sentence encoding
Author: orenkobo
Pull request: no
Labels: name: wontfix

I am trying to access to encoding of sentences in the vaious layers in pre-trained BERT model.
So it should be something like this:

```
sentence = 'We bought a new car'
bert_encoder = load_encoder('bert-base-uncased')
enc = bert_encoder.encode(sentence)
enc.get_layer[0] #this is the first layer
enc.get_layer[-1] #this is the last layer

```
What is the best way to do it?
Thanks!

Comments:
I believe you can't do like that, you have to run the model just as is with all the necessary inputs(pertaining to the sentence) as mentioned in the docs : https://huggingface.co/transformers/model_doc/bert.html#transformers.BertModel
and then add the configuration : `config.output_hidden_states=True` for getting the embeddings  from each intermediate encoding layers. 

@Sriharsha-hatwar Thanks, do you have a code sample maybe?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BERT Tokenization problem when the input string has a "." in the string, like floating number
Author: wenhuchen
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using: Tokenizer
* [ ] the official example scripts: (give details below) N/A
* [ ] my own modified scripts: (give details below) N/A

The tasks I am working on is: Any
* [ ] an official GLUE/SQUaD task: (give the name) N/A
* [ ] my own task or dataset: (give details below) N/A

## To reproduce

Steps to reproduce the behavior:

1.  Load any BERT tokenizer
2. Tokenize something with a "." in between
3. Decode these ids, you will find it mismatch

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
x = tokenizer.encode('AN.C', add_special_tokens=False)
z = tokenizer.decode(x)
```
It prints:
```
AN. C
```

## Expected behavior
```
AN.C
```
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Ubuntu
- Python version: 3.6
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?): GPU
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to change transformers model embedding layer weights
Author: acmilannesta
Pull request: no
Labels: name: wontfix

I trained my own tokenizer and added new words. Now I need to change the embedding size from the pretrained model. What I do is like this:

```
import transformers as tfm
import tensorflow as tf

backbone = tfm.TFRobertaModel.from_pretrained(PRETRAINED_PATH, output_hidden_states=True)
add_emb = tf.random.uniform(shape=(new_vocab_size, 768), minval=-1., maxval=1.)
new_emb = tf.concat((backbone.roberta.embeddings.word_embeddings, add_emb), 0)
backbone.roberta.weights[194] = new_emb 
```
However, the shape of embedding weight is still the original vocab size.

But if I do 

```
backbone = tfm.TFRobertaModel.from_pretrained(PRETRAINED_PATH, output_hidden_states=True)
add_emb = tf.random.uniform(shape=(new_vocab_size, 768), minval=-1., maxval=1.)
backbone.roberta.embeddings.word_embeddings= tf.concat((backbone.roberta.embeddings.word_embeddings, add_emb), 0)
```
Then the embedding weights will be removed from model ```trainable_weights``` and has only 198 elements instead of 199 in the original list.

Am I doing something wrong to change the embedding weights? Thanks!

The original stack overflow question is also posted:
https://stackoverflow.com/questions/61860156/how-to-change-transformers-model-embedding-layer-weights

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to extract the best candidate after token classification?
Author: renjithsasidharan
Pull request: no
Labels: name: wontfix

Let's assume the model predicts the following for an input sequence.
```
The O            
creation. O
date O
is O
27 B-DATE
Aug I-DATE
2020 I-DATE
and  O
update. O
date.  O
is  O
01-09-2020 B-DATE
```

How do you pick the best candidate for **creation date** from logist values? 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
🐛 Weird learning rate with TPU Trainer
Author: astariul
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): **ELECTRA**

Language I am using the model on (English, Chinese ...): **English**

The problem arises when using:
* [ ] the official example scripts: 
* [x] my own modified scripts: 

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: 
* [x] my own task or dataset: CNN/DM

## To reproduce

I use `run_glue.py` as example to build a training script for TPU, with the Trainer API. My task is sequence classification with CNN/DM dataset.

I initialized the Trainer with following optimizer / scheduler :

```python
optimizer = AdamW(optimizer_grouped_parameters, lr=training_args.learning_rate, eps=training_args.adam_epsilon)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=training_args.warmup_steps, num_training_steps=287113)

trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=cnn_dm["train"] if training_args.do_train else None,
        eval_dataset=cnn_dm["validation"],
        data_collator=MyCollator(),
        prediction_loss_only=True,
        optimizers=(optimizer, scheduler),
    )
```

Now, the training procedure is working : the code run fine on 8 TPU core.

**But the loss is not decreasing.**

After looking into Tensorboards logs, I found the learning rate to be very weird :

 
![image](https://user-images.githubusercontent.com/43774355/82189972-3e888800-992b-11ea-8887-6715500f388d.png)

A few points to note :
* I specified a learning rate of **1e-4** (with the command argument `--learning_rate 1e-4`), but as you can see, the maximum value for the learning rate is **3.5e-6**.
* The shape of learning rate is not what I expected : After warmup, learning rate is supposed to decrease linearly, but instead, it stays fixed.

I don't know why the learning rate is being like this. Any idea what I might be doing wrong ?

_I can't share my notebook, but this seems to be the exact same issue with the official script `run_glue.py`, as described in #4358_ 

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: **2.9.1**
- Platform: **Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic**
- Python version: **3.6.9**
- PyTorch version (GPU?): **1.6.0a0+176174a (False)**
- Tensorflow version (GPU?): **2.2.0 (False)**
- Using GPU in script?: **No**
- Using distributed or parallel set-up in script?: **Yes : `xla_spawn.py`**

@julien-c 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Get word seperator char for tokenization
Author: Ricocotam
Pull request: no
Labels: name: wontfix

name: High-Level feature

name: Core: Tokenization

# 🚀 Feature request
When using some of tokenization models (e.g. : CamemBERT) you can find the char used to separate words after some investigation ([start](https://github.com/huggingface/transformers/blob/a4c75f149269099a98613f51b76cd0b579a109ee/src/transformers/tokenization_camembert.py#L274), [first jump](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_camembert.py#L27), [last jump](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_xlnet.py#L43)). Making it a general rule seems interesting.

## Motivation

With this you can detect whether a token is a subword or the start of a word (and then use this info for masking). But this is inconsistent across models. After several hours I couldn't find this information for RoBERTa (and GPT2) in the code. We can obviously get this char experimentally but it is not robust to new models or even different versions of the model. 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I don't think you should delete the issue since it's kinda useful to have and not having it may be embarassing. But do as you want !

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error with batch_encode_plus
Author: Nithin-Holla
Pull request: no

# 🐛 Bug

## Information
When using the tokenizer method `batch_encode_plus` on the new `transformers` version, I run into an error

Model I am using (Bert, XLNet ...): BertTokenizer

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Here is a sample script to reproduce the error:
```python
import torch
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = [['sample text 1'], ['sample text 2']]
encode = tokenizer.batch_encode_plus(text)
```

The stack trace is:
```
Traceback (most recent call last):
  File "bug.py", line 8, in <module>
    encode = tokenizer.batch_encode_plus(text)
  File "/home/nithinh/debug/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1832, in batch_encode_plus
    **kwargs,
  File "/home/nithinh/debug/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 534, in _batch_encode_plus
    ids, pair_ids = ids_or_pair_ids
ValueError: not enough values to unpack (expected 2, got 1)
```
## Expected behavior

Successfully tokenize without errors. The script runs fine with `transformers` v2.11.0.

## Environment info
     
- `transformers` version: 3.0.2
- Platform: Linux-4.19.0-9-amd64-x86_64-with-debian-10.4
- Python version: 3.6.3
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Hi, if want to encode as sentence pair then you should do it like this
```python3
[ ["sent1", "sent2"] ]
```
else just pass the examples in a single list
```python3
["example1", "example2"]
```

Also, consider switching to the new tokenizer API. You can find the docs [here](https://huggingface.co/transformers/preprocessing.html)

@patil-suraj I didn't intend to encode it as a sentence pair, but rather a batch of single sentences of size 2.

Then you can just pass a list of examples. No need for nested list

Okay, thanks!
ProjectedAdaptiveLogSoftmax.log_prob raises Exception
Author: gasteigerjo
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Transformer-XL

Language I am using the model on (English, Chinese ...): WikiText-103 (English)

The problem arises when using:
* [x] the official example scripts: `run_transfo_xl.py` (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official task: WikiText-103
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Fix `run_transfo_xl.py` by removing the unused `work_dir` argument (line 52) and changing `lm_labels=target` to `labels=target` (line 108).
2. Add `logits = self.crit.log_prob(pred_hid.flatten(0, 1))` right before the model output, e.g. in line 919 of `modeling_transfo_xl.py`.
3. Run `run_transfo_xl.py`.
4. Look at error message:
```
The size of tensor a (1280) must match the size of tensor b (20000) at non-singleton dimension 1
  File ".../transformers/src/transformers/modeling_transfo_xl_utilities.py", line 246, in log_prob
    logprob_i = head_logprob[:, -i] + tail_logprob_i
  File ".../transformers/src/transformers/modeling_transfo_xl.py", line 920, in forward
    logits = self.crit.log_prob(pred_hid.flatten(0, 1))
  File ".../run_transfo_xl.py", line 107, in evaluate
    ret = model(data, labels=target, mems=mems)
  File ".../run_transfo_xl.py", line 124, in main
    test_loss = evaluate(te_iter)
  File ".../run_transfo_xl.py", line 143, in <module>
    main()
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
`log_prob` should return the log probabilities instead of raising an Exception.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: Master (commit 384f0eb)
- Platform: Ubuntu
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (CUDA 10.1, CuDNN 7.6.3)
- Tensorflow version (GPU?): -
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pretrained Transformer-XL gives unreasonable result on WikiText-103
Author: gasteigerjo
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Transformer-XL (`transfo-xl-wt103`)

Language I am using the model on (English, Chinese ...): WikiText-103 (English)

The problem arises when using:
* [x] the official example scripts: `run_transfo_xl.py` (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official task: WikiText-103 (not GLUE/SQUaD)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Fix `run_transfo_xl.py` by removing the unused `work_dir` argument (line 52) and changing `lm_labels=target` to `labels=target` (line 108).
2. Run `run_transfo_xl.py`.
3. Observe the result: `test loss 10.20 | test ppl 26951.114`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

A reasonable result around the order of PPL=18.3, as reported in the paper. I know that the result will not be exactly the same, but something is definitely wrong here.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: Both 2.9.1 and Master (commit 384f0eb)
- Platform: Ubuntu
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (CUDA 10.1, CuDNN 7.6.3)
- Tensorflow version (GPU?): -
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.