noamaanMulla-03/transformers-issues huggingface dataset:
Advantage of BertAdam over Adam?
Author: BramVanroy
Pull request: no
Labels: name: Discussion

I have implemented BERT, taking the output of [CLS] and feeding that to a linear layer on top to do regression. I froze the embedding layers of BERT, though. I was using the standard Adam optimizer and did not run into any issues.

When and/or why should one use BERTAdam? And, in a set-up like mine, would you use BERTAdam for BERT, and regular Adam for the rest of the whole model?

Comments:
Some explanation is given here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/694e2117f33d752ae89542e70b84533c52cb9142/README.md#optimizers

`BertAdam` is a `torch.optimizer` adapted to be closer to the optimizer used in the TensorFlow implementation of Bert. The differences with PyTorch `Adam` optimizer are the following:

* `BertAdam` implements weight decay fix,
* `BertAdam` doesn't compensate for bias as in the regular `Adam` optimizer.


@stefan-it  Thanks for the link. These improvements are not the same as the suggested AdamW improvements, I assume?

Yes they are the same. `BertAdam` implements AdamW and in addition doesn't compensate for the bias (I don't know why the Google team decided to do that but that's what they did).

In most case we have been using standard Adam with good performances (example by using NVIDIA's apex fusedAdam as optimizer) so you probably shouldn't worry too much about the differences between the two. We've incorporated `BertAdam`mostly to be able to exactly reproduce the behavior of the TensorFlow implementation.

@thomwolf Thanks, that explanation really helps. I have been using standard Adam with good results and BertAdam didn't improve that. So in my particular case it may not have been useful.

Closing this, as my question has been answered.

For reference for future visits, recent research suggests that the omission of the bias compensation in BERTAdam is one of the sources of instability in finetuning:

* [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://www.lsv.uni-saarland.de/wp-content/publications/2020/On_the_Stability_of_Fine-tuning_BERT_preprint.pdf)
* [Revisiting Few-sample BERT Fine-tuning](https://arxiv.org/abs/2006.05987)
Installation of Transformers without Sacremoses
Author: zanderkent
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi HuggingFace Team!

I was checking the dependencies of this library, and I found that sacremoses does not have an accepted licence type for my system. The setup.py file says that it's needed for XLM. If I don't plan on using XLM would I be able to modify the setup.py and remove the sacremoses requirement?

Thanks!
Zander


Comments:
I commented sacramoses it out in the setup.py and installed it, everything worked as designed! As long as I don't use XLM

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Having optional GPL code in a widely used package is an issue.  If fixing it is as simple as commenting it out in the setup, couldn't there be a way to make that available through some variant, so as not to taint other open source packages?

`sacremoses` seems to have been licensed under MIT since https://github.com/alvations/sacremoses/pull/92 though?
Update run_language_modeling.py to handle writes on networked filesystem better
Author: Genius1237
Pull request: yes
Labels: name: wontfix

In the case of multi-node distributed training, reads and writes typically happen to a common networked filesystem. 

In the current version of the `run_language_modeling.py` script, processes that have `local_rank` as 0 perform the writes to disk (tensorboard, dataset cache and model checkpointing). In the case of multi-node distributed training, there ends up being one process per node having `local_rank` as 0, hence multiple processes try writing to the filesystem at one point, resulting on errors being thrown depending on the filesystem.

This pull request updates the script such that only the process having a `global_rank` of 0 does the writing. `global_rank` isn't a variable directly accessible in the script, it is obtained by calling `torch.distributed.get_rank()`.

I've tested the script in 4 different cases and they work without any error in these cases: multi-node training with DDP, single-node training with DDP, single-node training with DP and single gpu training.

Comments:
For some reference, check out https://github.com/pytorch/pytorch/issues/12042 and https://github.com/facebookresearch/maskrcnn-benchmark/pull/40. These address the same issue.

Also, one of the checks that failed, `check_code_quality`, would fail for the existing version of the script as well. There's a check for a line length of 119, and there are already many lines exceeding that.

I did think about the other scripts. Are those already setup with `DistributedDataParallel` cause one would theorize that those tasks aren't that heavy and wouldn't benefit much from running across multiple GPUs. 

Also, I have one or 2 more fixes along the lines of this one for distributed training. I was wondering if I should rename this PR and add those in, or create a new one for each of those fixes. One of them is about loading checkpoints (of the optimizer and scheduler) while resuming training.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Closing this as `run_language_modeling.py` is now based on the trainer. Thank you for your contribution!!
Fix mlm
Author: Santosh-Gupta
Pull request: yes
Labels: name: wontfix

The way the texts is being split up into blocks right now

```
tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))

for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i : i + block_size]))
```

Results in double [CLS] tokens at the beginning, since special tokens are being added at `tokenizer.convert_tokens_to_ids` And at `tokenizer.build_inputs_with_special_tokens`. Somehow, double [SEP] tokens are not occuring. 

The following eliminates the double [CLS] token. 

```
tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text)[1:-2])

for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i : i + block_size]))
```

Comments:
Couldn't we achieve the same result by specifying `add_special_tokens=False` instead of that? This isn't robust to different models, as GPT-2 (which doesn't have special tokens) would get some tokens removed.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Closing this as `run_language_modeling.py` is now based on the trainer. Thanks for your contribution!!
question-answering pipeline error : too many values to unpack (expected 2)
Author: wwwehr
Pull request: yes
Labels: name: wontfix

name: Core: Pipeline

For some `question-answering` models the pipleline encounters extra tuples from the `.model()` call, where we must have exactly 2.

> This stub results in the following error:
```python
from transformers.pipelines import pipeline

model_name = "mrm8488/bert-uncased-finetuned-qnli"

nlp = pipeline("question-answering", model=model_name, tokenizer=model_name)
QA_input = {
    "question": "Why is model conversion important?",
    "context": "The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.",
}
res = nlp(QA_input)
```

> error message
```bash
Traceback (most recent call last):
  File "test3.py", line 10, in <module>
    res = nlp(QA_input)
  File "/opt/conda/lib/python3.7/site-packages/transformers/pipelines.py", line 1010, in __call__
    start, end = self.model(**fw_args)
ValueError: too many values to unpack (expected 2)
```

> env
```text
- `transformers` version: 2.8.0
- Platform: Linux-4.9.184-linuxkit-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: nope
- Using distributed or parallel set-up in script?: nope
```

Comments:
hmmm, `mrm8488/bert-uncased-finetuned-qnli` is a sequence classification model, not a QA model.

You probably get warnings while loading it in a QA Pipeline.

Does this happen with other (QA) models?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fix #2549
Author: Guitaricet
Pull request: yes
Labels: name: wontfix

closes #2549
proposed solution for unsupported operand type error in tokenizer.batch_encode_plus

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


=(

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, this should have been fixed with the release of `v3.0.0`. Thanks a lot for your contribution!
Model doc failed
Author: zheyuye
Pull request: no

As title, model doc failed in page https://huggingface.co/transformers/model_summary.html, e.g. https://huggingface.co/model_doc/distilbert

Comments:
This has been fixed in master, https://huggingface.co/transformers/master/model_summary.html will have the proper links.

This has now been fixed for the stable version as well!
Should check that torch TPU is available
Author: LysandreJik
Pull request: yes

fix #5634

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5636?src=pr&el=h1) Report
> Merging [#5636](https://codecov.io/gh/huggingface/transformers/pull/5636?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b9d8af07e66764bbf4213e1ce443fcdfa927ca46&el=desc) will **not change** coverage.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5636/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5636?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5636   +/-   ##
=======================================
  Coverage   77.66%   77.66%           
=======================================
  Files         145      145           
  Lines       25364    25364           
=======================================
  Hits        19700    19700           
  Misses       5664     5664           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5636?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/5636/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `93.60% <ø> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5636/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `86.91% <100.00%> (ø)` | |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5636/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.70% <0.00%> (-2.26%)` | :arrow_down: |
| [src/transformers/tokenization\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5636/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtX3JvYmVydGEucHk=) | `95.23% <0.00%> (+10.71%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5636?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5636?src=pr&el=footer). Last update [3cc23ee...a09ad90](https://codecov.io/gh/huggingface/transformers/pull/5636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
T5 has no module  ```torch_xla``` when using T5 fine-tuned on SQUADv2
Author: mrm8488
Pull request: no

# 🐛 Bug

## Information

I get this error:
``` ModuleNotFoundError: No module named 'torch_xla'```
Full error message:

```
      2 
      3 tokenizer = AutoTokenizer.from_pretrained("mrm8488/t5-base-finetuned-squadv2")
----> 4 model = AutoModelWithLMHead.from_pretrained("mrm8488/t5-base-finetuned-squadv2")
      5 
      6 def get_answer(question, context):

1 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    796 
    797         if hasattr(config, "xla_device") and config.xla_device:
--> 798             import torch_xla.core.xla_model as xm
    799 
    800             model = xm.send_cpu_data_to_device(model, xm.xla_device())

ModuleNotFoundError: No module named 'torch_xla'
```

## To reproduce
```python
from transformers import AutoModelWithLMHead, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mrm8488/t5-base-finetuned-squadv2")
model = AutoModelWithLMHead.from_pretrained("mrm8488/t5-base-finetuned-squadv2")

def get_answer(question, context):
  input_text = "question: %s  context: %s 
" % (question, context)
  features = tokenizer([input_text], return_tensors='pt')

  output = model.generate(input_ids=features['input_ids'], 
               attention_mask=features['attention_mask'])

  return tokenizer.decode(output[0])

context = "Manuel have created RuPERTa-base with the support of HF-Transformers and Google"
question = "Who has supported Manuel?"

get_answer(question, context)
```
I used this example code a few weeks ago and had no problem...

Comments:
Indeed, it seems this model was once trained or initialized on TPU. Thanks for letting us know, I'm patching it in #5636.

Yes, it was trained on TPU

It should be fixed on master now, can you try pulling from master and running your code?

I already did it and it works!!!
Thank you!!
Fixed TextGenerationPipeline on torch + GPU
Author: TevenLeScao
Pull request: yes

Fixes #5622 .

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5629?src=pr&el=h1) Report
> Merging [#5629](https://codecov.io/gh/huggingface/transformers/pull/5629?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/fa5423b1695cd24856bcff47214172e0f540d924&el=desc) will **decrease** coverage by `0.21%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5629/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5629?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5629      +/-   ##
==========================================
- Coverage   77.79%   77.58%   -0.22%     
==========================================
  Files         145      145              
  Lines       25355    25357       +2     
==========================================
- Hits        19726    19672      -54     
- Misses       5629     5685      +56     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5629?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/5629/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `76.24% <100.00%> (+0.08%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5629/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5629/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxuZXQucHk=) | `66.66% <0.00%> (-23.43%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5629/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.95% <0.00%> (-2.01%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5629/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `93.36% <0.00%> (+49.37%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5629?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5629?src=pr&el=footer). Last update [fa5423b...dfeeffa](https://codecov.io/gh/huggingface/transformers/pull/5629?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@LysandreJik if you wanna take a look, it's just a short bugfix (essentially adding the `if self.framework == "pt", generated_sequence = generated_sequence.cpu()` line)

LGTM!
TextGenerationPipeline breaks when used with device=0
Author: TevenLeScao
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): model-agnostic (breaks with GPT2 and XLNet)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
[x] my own modified scripts: (give details below)

The tasks I am working on is:
[x] my own task or dataset: plain old language generation

## To reproduce

Steps to reproduce the behavior:

```
#!/usr/bin/env python3
import random
from transformers import pipeline, XLNetLMHeadModel
import torch
import time
random.seed(0)
torch.manual_seed(0)

generator = pipeline("text-generation", model="xlnet-base-cased", tokenizer="xlnet-base-cased", device=0)
output_to_check = generator("Today is a beautiful day and I, ", offset=offset, do_sample=True, top_k=50, max_len=100)
```

## Expected behavior

What should happen : text generation
What actually happens : 

```
Traceback (most recent call last):
  File "/home/teven/dev_transformers/perso/transformers/generation_script.py", line 15, in <module>
    output_to_check = generator("Today is a beautiful day and I, ", offset=offset, do_sample=True, top_k=50, max_len=100)
  File "/home/teven/dev_transformers/perso/transformers/src/transformers/pipelines.py", line 692, in __call__
    generated_sequence = generated_sequence.numpy().tolist()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
```

Just missing a conversion before the `.numpy()` call

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-62-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
If that's not the case, we should make sure that the pipelines run on GPU in the GPU CI. (fast and slow), to catch things like this.
Is it possible to have high perplexity of some individual sentences, as compared to overall testing corpus perplexity?
Author: states786
Pull request: no
Labels: name: wontfix

Hi,

I have calculated an overall perplexity of my testing corpus after finetuning GPT-2 pretrained model
Similarly, I have also calculated perplexity of individual sentences, after extracting it from my testing corpus.

**I notice that some individual sentence perplexity is even higher than the overall perplexity.** 

What will be the reason for it? Will it be possible? Have I done something wrong in my calculation? 


Kindly share your opinion.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Create README.md - Model card for sentence-transformers/bert-base-nli-mean-tokens
Author: nreimers
Pull request: yes
Labels: name: model card

Model card for https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens

Comments:
Thanks for sharing! Note that we don't currently have automated deployment on ExBERT (cc @bhoov)

➡️   [model page](https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens)
T5 fp16 overflow in forward (T5DenseReluDense)
Author: lior1990
Pull request: no

# 🐛 Bug
Using `AutoModelWithLMHead.from_pretrained("t5-base")` for fine-tuning, after 34 iterations I get nan loss from the forward method.
After debugging it, I found that the source of the nan is due to an overflow that happens in `T5DenseReluDense`, when running `h = self.wo(h)`.  The result of this forward is a tensor that has `inf` in one of its values, which later on causes the nan loss. 
I looked into this calculation with fp32 and I saw that his `inf` is caused due to a value of 66246.3906, which is over the maximum value of 65504 in fp16.

This issue only happens with fp16 (opt_level="O1"), with opt_level="O0" everything is fine.

## Information

Model I am using (Bert, XLNet ...): T5

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
I don't have step by step instructions, because I will need to upload my entire dataset for that.
I have a pickle for the vector `h` and the weights of `self.wo` that causes the overflow in `T5DenseReluDense`, I can upload it if it might help.

## Expected behavior

get a numeric loss

## Environment info
    
- `transformers` version:  3.0.2
- Platform: Linux-5.3.0-1030-aws-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Comments:
See: #4586