noamaanMulla-03/transformers-issues huggingface dataset:
How to use pretrained model for inference?
Author: andrster
Pull request: no
Labels: name: wontfix

I have a bert model finetuned for pytorch and have trouble actually using it. Like how can I get to model("<some sentance>") outputting results for the tokens? 

Comments:
Hello @andrster 
Which model are you trying ? for ex, for QA, token classification, sentence classification etc.
Please elaborate more.  

Sorry, token classification 

@andrster have you checked out the pipelines section of the README?

doesn't have one on NER

@andrster 
ner pipeline is available in Transformers. Check here https://huggingface.co/transformers/usage.html#named-entity-recognition

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Model is running on special characters and word pieces for token classification
Author: andrster
Pull request: no
Labels: name: wontfix

['last', 'completed', 'interactions', 'on', 'wed', '##nes', '##day']
and it will return for 9 labels, but should really on return 5

Comments:
Hi @andrster , it's not clear what you mean here. Can you please provide more explanation

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Albert pretraining loss not decreasing
Author: 008karan
Pull request: no
Labels: name: wontfix


I am training Albert from scratch using run_language_modeling.py.
Doing training on 8 V100.p3dn8x.
Launching with this parameters.

```
python transformers/examples/language-modeling/test.py   --train_data_file x.txt  --output_dir albert_model --model_type albert --mlm --config_name test --tokenizer_name test --do_train --line_by_line --learning_rate 0.00088 --num_train_epochs 3 --save_total_limit 50 --save_steps 5000 --per_gpu_train_batch_size 150 --seed 42 --overwrite_output_dir --max_steps 200000 --fp16
```
![image](https://user-images.githubusercontent.com/18630864/83558269-65041100-a530-11ea-965b-f24a76550a28.png)

loss is not decreasing. The above plot contains training with and without warmup steps training curve. loss is stuck on `7.27` in both case.
Also, weird thing is while I am setting the number of epochs 3 but in training its showing 9
```
was: ModuleNotFoundError("No module named 'amp_C'",)
06/02/2020 14:45:14 - INFO - transformers.trainer -   ***** Running training *****
06/02/2020 14:45:14 - INFO - transformers.trainer -     Num examples = 28236463
06/02/2020 14:45:14 - INFO - transformers.trainer -     Num Epochs = 9
06/02/2020 14:45:14 - INFO - transformers.trainer -     Instantaneous batch size per device = 150
06/02/2020 14:45:14 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 1200
06/02/2020 14:45:14 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/02/2020 14:45:14 - INFO - transformers.trainer -     Total optimization steps = 200000
``` 
Can anyone suggest what could be reasons for such behavior?


Comments:
Not certain, but looks like maybe nvidia apex was not installed correctly?

"was: ModuleNotFoundError("No module named 'amp_C'",)"

thats warning. 
Have followed `pip install -v --no-cache-dir ./` for apex installation. 
Changing LR to 5e-5 reducing the loss.

@008karan, setting the loss to 5e-5 led your model to convergence?

@LysandreJik loss is decreasing as of now 

> Also, weird thing is while I am setting the number of epochs 3 but in training its showing 9

can you comment on this?

I'll have a look.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can I save a word embedding from BERT and used again later for computational purpose(as BERT takes much more time). Just like in Glove. If yes then How? and is this good idea to do so?
Author: shubhamk16
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
As per my understanding, you can! 

If you read the BERT paper by Devlin et.al., you can see that two suggested ways to extract said word embeddings would be to concatenate the last four hidden layers (9 to 12), generating a 4*768=3072 sized embedding for each token. Alternatively you can also sum or average out the last 4 layers to generate vectors of size 768.

You can also instead prefer storing sentence embeddings, then the CLS token serves as averaged representation of the sentence embedding (not a very good representation though, unless the model was fine-tuned well on the data or the LM was pretrained on similar data, and even then the sentence embeddings through that CLS token could be substandard). Although, averaging out the second to last hidden layer (i.e. averaging out token embeddings for the sentence at the 11th hidden layer) seems to generate decent sentence embeddings (of size 768, the same as one token, since its averaged and not concatenated). 

Hope this helps!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
question about tokenizer
Author: yuimo
Pull request: no
Labels: name: wontfix

hi, is there any difference between initializing BertTokenizer and load it using from_pretrained, 
such as:
1、tokenizer = BertTokenizer(bert-base-uncased-vocab)
2、tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

i want to pretrain for some new language by myself, so i need to create the vocabulary, and how could i integrate it to the transformers's tokenizer
thanks a lot!

Comments:
This notebook might help you, it shows how you can train a LM from scratch on a new language.
https://github.com/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb

@patil-suraj thanks for your reply. i have read this notebook. but i want to re-use the functions in the transfomers such as functions like encode_plus, batch_encode_plus
so my question is :
after train a tokenizer by myself, how could i integrate it to the transformers's tokenizer to use functions in it?
thanks again

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
lightning_base: new clarg: lr_scheduler=polynomial_decay
Author: sshleifer
Pull request: no

lr_scheduler should be a string with many options, including polynomial_decay, like this [command](https://github.com/pytorch/fairseq/blob/master/examples/mbart/README.md#finetune-on-en-ro)

Comments:
@stas00 as discussed on slack

got it

PR for stage 1 to support the existing schedulers: https://github.com/huggingface/transformers/pull/6232
once this is happy and merged, I will work on importing new ones.

poly: https://github.com/huggingface/transformers/pull/6361

@sshleifer, so now that get_polynomial_decay_schedule_with_warmup is done (about to be merged) - do we need any others?

Not that I know of. We can close this once the associated PR  ismerged.
seq2seq examples require pytest
Author: dmlap
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-5.4.0-42-generic-x86_64-with-glibc2.29
- Python version: 3.8.2
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: n/a
- Using distributed or parallel set-up in script?: n/a

### Who can help
 examples/seq2seq: @sshleifer
documentation: @sgugger

## To reproduce

Steps to reproduce the behavior:

1. Create a new virtual environment and set it up to run the examples tests. Do _not_ install `pytest` and `pytest-xdist`.
2. Run the tests with `unittest` as [described in the docs](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#tests)

## Expected behavior
The examples tests pass. Actual behavior:

```sh
======================================================================
ERROR: seq2seq.test_bash_script (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: seq2seq.test_bash_script
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/dmlap/projects/transformers/examples/seq2seq/test_bash_script.py", line 8, in <module>
    import pytest
ModuleNotFoundError: No module named 'pytest'


======================================================================
ERROR: seq2seq.test_seq2seq_examples (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: seq2seq.test_seq2seq_examples
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/dmlap/projects/transformers/examples/seq2seq/test_seq2seq_examples.py", line 10, in <module>
    import pytest
ModuleNotFoundError: No module named 'pytest'


----------------------------------------------------------------------
Ran 16 tests in 179.454s

FAILED (errors=2)
```

Perhaps the documentation should be updated to require `pytest`?

Comments:
yes, great catch. Will update!
Bug in optimization_tf   create_optimizer 
Author: JunnYu
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

WHEN I am using optimization_tf(create_optimizer):
Problems with learning rate schedule
## To reproduce
```
from transformers.optimization_tf import create_optimizer
import matplotlib.pyplot as plt
%matplotlib inline
opt = create_optimizer(init_lr=5e-5, num_train_steps=100, num_warmup_steps=50)
lr = opt.learning_rate
results = [lr(i).numpy() for i in range(101)]
print(results[49:51])
plt.plot(results)
plt.show()
```
output  [4.9e-05, 2.5e-05, 2.45e-05]
## Expected output
the max lr should be 5e-05
expected output [4.9e-05, 5e-05, 4.9e-05]


```
class WarmUp(tf.keras.optimizers.schedules.LearningRateSchedule):
    def __call__(self, step):
        with tf.name_scope(self.name or "WarmUp") as name:
            # Implements polynomial warmup. i.e., if global_step < warmup_steps, the
            # learning rate will be `global_step/num_warmup_steps * init_lr`.
            global_step_float = tf.cast(step, tf.float32)
            warmup_steps_float = tf.cast(self.warmup_steps, tf.float32)
            warmup_percent_done = global_step_float / warmup_steps_float
            warmup_learning_rate = self.initial_learning_rate * tf.math.pow(warmup_percent_done, self.power)
            return tf.cond(
                global_step_float < warmup_steps_float,
                lambda: warmup_learning_rate,
                lambda: self.decay_schedule_fn(step),
                name=name,
            )
```
change:
lambda: self.decay_schedule_fn(step) =>lambda: self.decay_schedule_fn(step-warmup_steps_float)

```
def create_optimizer(init_lr, num_train_steps, num_warmup_steps):
    """Creates an optimizer with learning rate schedule."""
    # Implements linear decay of the learning rate.
    learning_rate_fn = tf.keras.optimizers.schedules.PolynomialDecay(
        initial_learning_rate=init_lr, decay_steps=num_train_steps, end_learning_rate=0.0
    )
    if num_warmup_steps :
        learning_rate_fn = WarmUp(
            initial_learning_rate=init_lr, decay_schedule_fn=learning_rate_fn, warmup_steps=num_warmup_steps
        )
```
change:
PolynomialDecay  decay_steps  should be  num_train_steps-num_warmup_steps


Comments:
I have raised https://github.com/huggingface/transformers/pull/4940, waiting for approval.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Write With Transformer Request:
Author: BigSalmon2
Pull request: no
Labels: name: wontfix

name: Write With Transformer

# 🚀 Feature request

I understand if this is unattainable for reason of money, but I was wondering if you could replace three autocomplete suggestions with five.

Comments:
This wouldn't change the unit economics of providing this service, but I'm curious, why don't you press tab twice instead?

The previous suggestions disappear after I press tab. 

It is easier to pick an autocompletion that is suitable, the more options I have to compare. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Adding PaddingDataCollator
Author: sgugger
Pull request: yes

New version of #6398 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6442?src=pr&el=h1) Report
> Merging [#6442](https://codecov.io/gh/huggingface/transformers/pull/6442?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/96c3329f19f28e47eab7f9f20ed3504619e16722&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `52.94%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6442/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6442?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6442      +/-   ##
==========================================
- Coverage   79.95%   79.93%   -0.02%     
==========================================
  Files         153      153              
  Lines       27932    27947      +15     
==========================================
+ Hits        22332    22339       +7     
- Misses       5600     5608       +8     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6442?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6442/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.27% <ø> (ø)` | |
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/6442/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `90.90% <52.94%> (-5.68%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6442?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6442?src=pr&el=footer). Last update [96c3329...a153ed4](https://codecov.io/gh/huggingface/transformers/pull/6442?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[DO NOT SUBMIT] Run TPU examples for PR commits.
Author: zcain117
Pull request: yes

Trying out the CircleCI flow. I'll delete this PR after testing.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6448?src=pr&el=h1) Report
> Merging [#6448](https://codecov.io/gh/huggingface/transformers/pull/6448?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/bc820476a5c72060f810f825298befd5ec85da4d&el=desc) will **increase** coverage by `0.09%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6448/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6448?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6448      +/-   ##
==========================================
+ Coverage   79.98%   80.08%   +0.09%     
==========================================
  Files         153      153              
  Lines       28005    28005              
==========================================
+ Hits        22401    22429      +28     
+ Misses       5604     5576      -28     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6448?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6448/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.47% <0.00%> (-32.95%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6448/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+7.26%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6448/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `94.72% <0.00%> (+22.87%)` | :arrow_up: |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/6448/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `87.50% <0.00%> (+58.65%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6448?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6448?src=pr&el=footer). Last update [bc82047...6567722](https://codecov.io/gh/huggingface/transformers/pull/6448?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


The TPU test succeeded: https://app.circleci.com/pipelines/github/huggingface/transformers/10480/workflows/c669aea7-b861-4b1b-b90c-d8c8b50e60dc/jobs/72547

I'll delete this PR now
ImportError: cannot import name 'DataCollatorForPermutationLanguageModeling'
Author: krannnn
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on English


The problem arises when using:
* [ ] the official example scripts: [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py).


The tasks I am working on is:
* [ ] Continue Training XLNet on domain-specific dataset / finetuning XLNet LM

## To reproduce

Steps to reproduce the behavior:

1. Install transformers 3.0
2. run the following command as mentioned in the readme file from examples :

```
python run_language_modeling.py \
    --output_dir=output \
    --model_type=xlnet \
    --model_name_or_path=xlnet-base-cased \
    --do_train \
    --train_data_file=$TRAIN_FILE
```
Error message : 
```
2020-07-20 10:47:29.463663: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "run_language_modeling.py", line 29, in <module>
    from transformers import (
ImportError: cannot import name 'DataCollatorForPermutationLanguageModeling'
```

## Expected behavior
I expect not to have this import error since I'm using the latest release of the library
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0
- Platform: Google Colab
- Python version: Python 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101
- Tensorflow version (GPU?): 2.2.0

- Using GPU in script?: Tesla K80
- Using distributed or parallel set-up in script?: No


Comments:
Hi @krannnn , `DataCollatorForPermutationLanguageModeling` is added after 3.0, you will need to install from source if you want to run examples

Hi @patil-suraj , out of curiosity, how do you install it? What do you mean by you will need to install from source?
TrainingArguments are ignored?!
Author: NebelAI
Pull request: no

## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
Trainer: @sgugger

## Information

Hey, I'm using `01_how-to-train.ipynb` to get a feeling for the object oriented way of training a bert model from scratch. Until now I've been using the scripts offered by offical bert repository. My target is to train all of my future Transformer models with your Huggingface interface (from scratch and of course fine tuning too).

I used `max_steps = 500_000` but it gets completely ignored. After training is started the output says:
```
Iteration: 11639/422095 [1:52:03<70:16:42, 1.62it/s]
Epoch 0/2 [00:00<?, ?it/s]
```
**Two epochs and 422095 iterations seems wrong!?** Official docs say _"max_steps = the total number of training steps to perform"_ Am I misinterpreting something?

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset:
  * line by line dataset
  * training a bert language model from scratch (generating vocab, setting a config, ...)

## To reproduce

Use colab "01_how-to-train.ipynb" (https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
) and change TrainingArguments to the following:
```
training_args = TrainingArguments(
    output_dir="./smallBERTa",
    overwrite_output_dir=True,
    do_train=True,
    warmup_steps=5000,
    max_steps=500000,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
)
```
Yes, I am passing the `training_args` to the `Trainer()` object.


## Expected behavior

I'm expecting to get 500.000 global training steps and just one epoch.


Comments:
I don't see what's wrong? There are 422,095 iterations in one of your epochs, so to get to 500,000 you'll do one full epoch and the beginning of the second epoch. Training will stop after it has reached 500,000.

You can't have just one epoch since you need more iterations to reach the `max_steps` you have given. That argument overrides the number of epochs. 

Then I can definitely say: I misinterpreted the logs. There was something like "global steps" when using BERT's pretrain script and the value was identical with the previously set "max_steps" parameter. Now I get it ... Thanks for clearing it up.