noamaanMulla-03/transformers-issues huggingface dataset:
QUESTION: How do I know what type of positional encoding to input during fine-tuning or pretrained BERT?
Author: abhisheknovoic
Pull request: no

Hello HuggingFace team,

I am familiarizing myself with the HuggingFace tutorials and understand the functionality of the various methods. However, I have a general question for example when using models like BERT.

Considering that I am doing sentiment classification and I want to fine-tune the whole BERT based on pre-trained weights, how do I know what should be the positional encoding as input during the `forward ( )` method? 

I know it has a default value of None, but doesn't it mean that during fine-tuning I need to input the same value that it was originally trained on in the first place? If so, how do I know what it was trained on during its original training from scratch? Is there a documentation for that somewhere?

Following to this, if I am freezing the weights of BERT for sentiment classification, I again have the same question on what should be my positional encoding input to the `forward ( )` method.

Please clarify this. Thanks for your time!



Comments:
Hi! If you're new to the library, I heavily recommend taking a look at the [glossary (position IDs in this case)](https://huggingface.co/transformers/glossary.html#position-ids), which explains how to use such inputs.

If you ignore the `position_ids`, then they're always automatically generated to be the same as the model's pre-training scheme. If you're fine-tuning and wish to keep the same position embeddings, then you don't need to pass them to the model.

THanks @LysandreJik , your second statement pretty much answered it completely! Thanks

@LysandreJik Hi, does that mean the position embeddings won't get updated if `position_ids` are not passed to the model (or with the default value of `None`)? Would you point me to the related lines in the code that implements this logic? Thanks!

This means that the position IDs will be generated on the fly, and the position embeddings will be exactly the same than during the pre-training. You can check the code [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L192-L214).
Albert Tokenizer hangs
Author: LeonieWeissweiler
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# 🐛 Bug

## Information
I am following the language modeling tutorial to train a LM on a simple wikipedia corpus from scratch. I am trying to use Albert instead of Roberta. As I couldn't find information on how to train an Albert Tokenizer from scratch, I'm loading the albert-base-v2 tokenizer. The Dataset creation doesn't work, it hangs for ages and when I stop it, I can see that it is always stuck in tokenization_albert.py, line 193:  

```python           
outputs = "".join([c for c in outputs if not unicodedata.combining(c)])
```

A week ago, it crashed consistently in this line due to large RAM allocations, but I can't reproduce that behaviour right now. 

## To reproduce

Steps to reproduce the behavior:
```python
from transformers import AlbertTokenizer
tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
from transformers import TextDataset
train_set = TextDataset(
    tokenizer=tokenizer,
    file_path="./drive/My Drive/datasets/simplewiki_train.txt",
    block_size=128,
)
```
## Expected behavior

I expected the tokenizer to run through in less than an hour for 100MB input. 

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.10.0
- Platform: Colab
- Python version: 3.6

Is anyone else experiencing this? I read in another issue that Albert should work with run_language_modeling out of the box. 


Comments:
Does this also happen when you use the slow tokenizers?

```python
tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2", use_fast=False)
```

Thanks for the suggestion @BramVanroy . I just tried this and the tokenizer has now been running for 70 minutes, so I think that's a yes, it also happens when I use slow mode.

cc @mfuntowicz AlbertTokenizer seems to hang in fast mode but not in slow mode

Hi, there is no fast mode for `AlbertTokenizer`, it's a SentencePiece based tokenizer which is not currently supported by `tokenizers`.

Do you think you can find more information about where the tokenizer actually hang?
Can you reproduce the behavior with a shorter input?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Does BERT public an embedding file like glove.840B.300d.txt? 
Author: 14H034160212
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
Hi, I have seen BERT public the pre-training language model like BERT-based-uncased. Does BERT public an embedding file like glove.840B.300d.txt? Many thanks.


Comments:
BERT is a fine-tuning based model. As far as I'm aware, there are no 'embedding files' as such. You could generate embedding relevant to your case, by providing the model with words (see documentation) and generate outputs.

> BERT is a fine-tuning based model. As far as I'm aware, there are no 'embedding files' as such. You could generate embedding relevant to your case, by providing the model with words (see documentation) and generate outputs.

Many thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ZeroDivisionError with Reformer
Author: eliasjacob
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.0
- Platform: Linux-5.4.0-42-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): **Reformer**

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below) (well, actually, not my own, but @patrickvonplaten 's) 

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Execute @patrickvonplaten 's notebook available at https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Reformer_For_Masked_LM.ipynb 
2. I've tried to run it on google colab and works fine. The problem appears when I try to run on my machine.
3. I've tried it with two different clean virtual environments (python 3.6 and 3.7), but they've both failed.
4. I haven't change the dataset, nor any model config/training args.
4. After calling trainer.train() I get the following error

```
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-13-02431faf649a> in <module>
      8 
      9 # train
---> 10 trainer.train()

/data/venv36/lib/python3.6/site-packages/transformers/trainer.py in train(self, model_path)
    394             t_total = self.args.max_steps
    395             num_train_epochs = (
--> 396                 self.args.max_steps // (len(train_dataloader) // self.args.gradient_accumulation_steps) + 1
    397             )
    398         else:

ZeroDivisionError: integer division or modulo by zero

```
## Expected behavior

The model should begin to train

Thanks!

Comments:
Hey @eliasjacob, the problem is probably that `self.args.gradient_accumaltion_steps` is set to a value > then `(len(train_dataloader)`

You were right (although I am unsure why the same notebook yielded different results in collab). Thank you very much!
Unable to install Transformers Master version
Author: AnkitVarshney02
Pull request: no

@patil-suraj @joeddav 
Is there some other way to install master version of transformers?
I tried using the URL to install the master version but it installed v3.0.2

_Originally posted by @AnkitVarshney02 in https://github.com/huggingface/transformers/issues/6752#issuecomment-681716615_

Comments:
Answered in #6752
Failing ONNX Slow Test
Author: sshleifer
Pull request: no

```
___________________ OnnxExportTestCase.test_quantize_pytorch ___________________
[gw0] linux -- Python 3.7.6 /home/hf/actions-runner_transformers/_work/transformers/transformers/.env/bin/python

self = <tests.test_onnx.OnnxExportTestCase testMethod=test_quantize_pytorch>

    @require_torch
    @slow
    def test_quantize_pytorch(self):
        for model in OnnxExportTestCase.MODEL_TO_TEST:
            path = self._test_export(model, "pt", 12)
            quantized_path = quantize(path)
    
            # Ensure the actual quantized model is not bigger than the original one
>           if quantized_path.stat().st_size >= Path(path).stat().st_size:
E           AttributeError: 'NoneType' object has no attribute 'stat'

tests/test_onnx.py:76: AttributeError
=============================== warnings summary ===============================

```

https://github.com/huggingface/transformers/runs/996031882?check_suite_focus=true

Comments:
@sshleifer Is it still a failing test? I can have a look

Still failing! See https://github.com/huggingface/transformers/runs/1024137280?check_suite_focus=true

On it 💪 

attempt at: #6716
ValueError: Unrecognized model identifier in facebook/bart-large-cnn.
Author: SagarPalyal
Pull request: no

## Environment info
     
- `transformers` version: 2.2.2
- Platform: windows 10
- Python version: 3.7.8
- PyTorch version (GPU?): 1.6.0+cpu
- Tensorflow version (GPU?):
- Using GPU in script?:No
- Using distributed or parallel set-up in script?:No


 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Model Cards: @julien-c
 Summarization: @sshleifer 
 examples/distillation: @VictorSanh
 Bart: @sshleifer
documentation: @sgugger
 -->

## Information

I am using facebook/bart-large-cnn summarization model as mentioned in hugging face website using transformers:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
model = AutoModel.from_pretrained("facebook/bart-large-cnn")

After running above lines of code I am getting below error message:
>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "E:\workspace\BERTExtractiveSummarizer\lib\site-packages\transformers\tokenization_auto.py", line 148, in from_pretrained
    "'xlm', 'roberta', 'distilbert,' 'camembert', 'ctrl', 'albert'".format(pretrained_model_name_or_path))
ValueError: Unrecognized model identifier in facebook/bart-large-cnn. Should contains one of 'bert', 'openai-gpt', 'gpt2', 'transfo-xl', 'xlnet', 'xlm', 'roberta', 'distilbert,' 'camembert', 'ctrl', 'albert'


Please let me know how to resolve this error. Thanks in advance.

PS: Same Error is coming for using "sshleifer/distilbart-cnn-12-6" model as well.


Comments:
Please only tag one or two people.
I think `pip install transformers --upgrade` will fix your problem.
[testing] replace hardcoded paths to allow running tests from anywhere
Author: stas00
Pull request: yes

Currently, some tests can be run only from the root of the project, since they hardcode relative paths.

With this PR, it's now possible to run tests from the root of the project, from inside `tests`, or really anywhere.

the gist of the change:
```
- PATH_SAMPLE_TEXT = "./tests/fixtures/sample_text.txt"
+ PATH_SAMPLE_TEXT = f"{get_tests_dir()}/fixtures/sample_text.txt"
```
now can do:
```
pytest tests/test_trainer.py
```
and:
```
cd tests
pytest ./test_trainer.py
```
or even:
```
cd examples
pytest ../tests/test_trainer.py
```
i.e. you no longer need to go to the root of the project to run tests

(I'm trying to figure out the codecov issue, running multiple tests, so it's easier to run tests from within `tests` dir)

p.s. CI failures are unrelated.


Comments:
CI failure is unrelated to this PR

so dope!
[style] a new `black` version is out there (20.8)
Author: stas00
Pull request: yes

it now seems to respect `--line-length 119` in the `make style/quality` command

so a big reformat - no code change

These are just the changes after running `make style`.


Comments:
weird, I thought I had a newer version, but it was old for some reason. updating to `pip install black==20.8b1` resolved this.
Unexpected behavior encoding token_type_ids in GPT models
Author: sarahwie
Pull request: no

# 🐛 Bug

## Information

When `token_type_ids` are passed into the `GPT2Model` and subclasses, they're encoded using the `nn.Embedding` lookup table as the vocabulary ([line](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_gpt2.py#L477)). Because the token type ids output by `encode_plus()` are `{0,1}`, this seems to create unexpected behavior by using the same embedding vectors to represent both word-tokens 0 and 1 in the vocabulary and the very distinct token type ids.

## Expected Behavior

Either:
1. instead of returning `token_type_ids` consisting of indices 0 and 1, extend the vocabulary by 2 and use those two ids as the `token_type_ids` in `encode_plus()`, or
2. addition of a separate `nn.Embedding` matrix for token_type_ids [here](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_gpt2.py#L352) (such as [in BERTModel](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_bert.py#L153))
3. discourage or throw a warning when `token_type_ids` are passed to GPTModel instances


Comments:
Fix tensor label type inference in default collator
Author: joeddav
Pull request: yes

Quick fix to `default_data_collator` allowing it to recognize the correct type of PT tensor label inputs rather than always casting them to float (#5060)

Comments:
What's the perf impact of a try/catch?

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5250?src=pr&el=h1) Report
> Merging [#5250](https://codecov.io/gh/huggingface/transformers/pull/5250?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/49f6e7a3c6729025e0d412ee19786c71811a6390&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5250/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5250?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5250   +/-   ##
=======================================
  Coverage   77.96%   77.96%           
=======================================
  Files         138      138           
  Lines       23886    23887    +1     
=======================================
+ Hits        18622    18623    +1     
  Misses       5264     5264           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5250?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/5250/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `98.36% <100.00%> (+0.02%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5250/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `76.42% <0.00%> (-0.39%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5250/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.26% <0.00%> (+0.12%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5250?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5250?src=pr&el=footer). Last update [49f6e7a...513dcb5](https://codecov.io/gh/huggingface/transformers/pull/5250?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Is it supposed to be slower? I ran a quick profile iterating over IMDb train set and saw no difference, but can manually check for `torch.tensor` instead if that's preferred.
bart-large-mnli-yahoo-answers model card
Author: joeddav
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6133?src=pr&el=h1) Report
> Merging [#6133](https://codecov.io/gh/huggingface/transformers/pull/6133?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6c002853a68906a5b1c2dd2ebb416770f1fc322b&el=desc) will **increase** coverage by `0.08%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6133/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6133?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6133      +/-   ##
==========================================
+ Coverage   77.77%   77.86%   +0.08%     
==========================================
  Files         146      146              
  Lines       26326    26326              
==========================================
+ Hits        20476    20499      +23     
+ Misses       5850     5827      -23     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6133?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6133/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `65.42% <0.00%> (-29.91%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6133/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `80.95% <0.00%> (-5.27%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6133/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.20% <0.00%> (-0.29%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6133/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `98.79% <0.00%> (+33.89%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6133?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6133?src=pr&el=footer). Last update [6c00285...5c85f49](https://codecov.io/gh/huggingface/transformers/pull/6133?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Feel free to merge whenever ready
Bart: make decoder_input_ids correctly if labels specified.
Author: sshleifer
Pull request: no
Labels: name: seq2seq

should call shift_tokens_right(labels), like T5.

cc @patil-suraj: does that make sense?

Comments:
Yes, definitely. Better to handle it in the model. Since lots of recent issues were due to incorrect or not shifting labels

But then `prepare_seq2seq_batch` should return `labels` instead of `decoder_input_ids`

yep. I'll take a stab.
Pegasus finetuning diary
Author: sshleifer
Pull request: no

Best score so far .2065 Rouge, much worse than paper. Generations appear to start lower case/be missing words at the beginning.



Clues:
- adding `<pad>` as prefix (like for generation) makes loss nan for at least 1000 steps (I killed it). 
- Without prefix, loss is nan for 5 steps, them improves.
- distillation with teacher produces huge hidden state MSE losses. This is probably unrelated and caused by the same large activations that break fp16.

Suspects:
- different causal mask than tf?
- tf doesn't shift labels or add a decoder prefix token. We shift labels but don't add a prefix token. there is a suraj issue where this appears to be suboptimal for t5 (which also has no bos token).
- bug in label smoothing


Best run:

![image](https://user-images.githubusercontent.com/6045025/89748679-5c7d0900-da92-11ea-9505-59ee413193d9.png)

![image](https://user-images.githubusercontent.com/6045025/89748681-5dae3600-da92-11ea-94bd-5043c586bd9c.png)


Comments:
freeze_embeds didn't matter (before mask fix)
mask fix explanation: 
we need `decoder_start_token_id=pad_token_id` to avoid the first word issue, but `decoder_padding_mask` should NOT tell the model to ignore that decoder_start_token_id, else you get nan.

This fix makes 1 example per batch not have eos in decoder_input_ids (t5,bart=same problem).
But maybe that can explain batch_size=1 truncation.

Original repo uses adafactor.

pegasus finetuning Running on fork branch has rouge2 23 with full beam search after 1.5 epochs
https://app.wandb.ai/sshleifer/transformers_fork-examples_seq2seq/runs/3cz2fe87?workspace=user-sshleifer

XSUM Metrics from today:
Models train on hack-pegasus-batches branch.
```
finetune: {'rouge1': 45.6515, 'rouge2': 22.9858, 'rougeL': 37.7569, 'n_obs': 11333, 'runtime': 4175.217807531357, 'seconds_per_sample': 0.3684}
dpx8 {'rouge1': 45.9739, 'rouge2': 23.1417, 'rougeL': 38.1625, 'n_obs': 11333, 'runtime': 2207.9071719646454, 'seconds_per_sample': 0.1948}
dpx4 {'rouge1': 43.0961, 'rouge2': 20.1954, 'rougeL': 35.5679, 'n_obs': 11333, 'runtime': 1813.8934507369995, 'seconds_per_sample': 0.1601}	
```

(10% chance 1st two rows are flipped)

Adafactor saves a lot of memory. All of those runs use adafactor.