noamaanMulla-03/transformers-issues huggingface dataset:
Roberta for NER task
Author: sl-victormazzeo
Pull request: no

## ❓ Questions & Help

Hello, is there a way to use Roberta model for NER task? Is there a script somewhere? 

Thank you.

Comments:
Hi @militu you should take a look at this long thread discussing NER for BERT (should be the same for RoBERTa): https://github.com/huggingface/pytorch-transformers/issues/64

But there is not RobertaForTokenClassification and TFRobertaForTokenClassification like BertForTokenClassification and TFBertForTokenClassification. 

Not yet indeed, do you want to submit a PR copying these models from Bert?

@thomwolf is this something the team is open to reviewing? I can open a PR that (ambitiously?) adds both `RobertaForTokenClassification` and `TFRobertaForTokenClassification` in the next few days/week.

Yes, sure (though I won't commit to a specific delay for reviewing hahaha).

Adding `RobertaForTokenClassification` and `TFRobertaForTokenClassification` should be very simple and basically kept as a copy-past from Bert similar models.

The most important here is actually to finish the PR adding token-to-string-character mappings (#1274 by @michaelrglass) so we can translate NER labels to token labels for training. Though I think there may also be a simpler way to do that by modifying RoBERTa/GPT-2 tokenizers to accept tokenized word but this require some knowledge of the internal functioning of GPT-2 tokenizer.

Created a PR and looking for feedback. Hoping to jump on the `run_ner.py` script as well tonight/tomorrow.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This was done and probably should be closed.
HuggingFace transformers documentation webpage is blank?
Author: h56cho
Pull request: no

Hello,

Is HuggingFace updating their transformers documentation site (https://huggingface.co/transformers/)?

I looked there to get some information about the HugginFace GPT-2, but for some reason all the contents of the website are gone.

Thank you,



Comments:
Having the same issue.

May be related to this:
https://twitter.com/Thom_Wolf/status/1208365367493636096


Are you still having an issue or is it fixed (if it is, please close the issue)?

Hello,
I am still having the same issue.


Ok, we are in the process of fixing it. Thanks for the report

It works now, thank you for the help
training BERT from scratch for native language PT-BR? Without init weight
Author: calusbr
Pull request: no
Labels: name: wontfix

I would like to train BERT from scratch for a textual base in PT-BR (8GB data). Is it possible to use the run_lm_finetuning.py code to perform this process without using the multi-language bert model?

I already have a vocab.txt for the PT-BR base and I don't want to load initial weights.

Is there any script or tutorial to perform this process step by step?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Should the option to run on TPU in run_glue.py use some sort of xla data parallelizer ? 
Author: Santosh-Gupta
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

The xla API ( https://github.com/pytorch/xla/blob/master/API_GUIDE.md ) and the TPU colab examples ( https://github.com/pytorch/xla/tree/master/contrib/colab ) each parallelize their data, either using a `torch_xla.distributed.parallel_loader.c` object or a `torch_xla.distributed.data_parallel.DataParallel` object (which uses `ParallelLoader`).

The `run_glue.py` example (https://github.com/huggingface/transformers/blob/master/examples/run_glue.py ) doesn't do this. I am wondering if there was a reason not to use xla's data parallelizers.

Comments:
Indeed, it would be great to improve the current TPU script to include better optimization, such as using the TPU DataParallel from Pytorch. We haven't gotten to it yet and we'll probably do so soon.

We'd be very happy to welcome a PR too! :)

Sounds good, I'm trying to figure it out. The part I'm stuck on is that the colab TPU examples use a multi-threading approach, but the official API recommends using multi-processing over multi-threading, so I am wonder if the TPUs require a multi-threading approach. 

Once I figure this out I think I can update the code. I asked a question on the xla repo about this
https://github.com/pytorch/xla/issues/1217



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fix error due to wrong argument name to Tensor.scatter()
Author: ShnitzelKiller
Pull request: yes

The named argument is called "source", not "src" in the out of place version for some reason, despite it being called "src" in the in-place version of the same Pytorch function. This causes an error.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2276?src=pr&el=h1) Report
> Merging [#2276](https://codecov.io/gh/huggingface/transformers/pull/2276?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ce50305e5b8c8748b81b0c8f5539a337b6a995b9?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2276/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2276?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2276   +/-   ##
=======================================
  Coverage   74.45%   74.45%           
=======================================
  Files          85       85           
  Lines       14610    14610           
=======================================
  Hits        10878    10878           
  Misses       3732     3732
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2276?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2276/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `63.91% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2276?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2276?src=pr&el=footer). Last update [ce50305...398bb03](https://codecov.io/gh/huggingface/transformers/pull/2276?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks a lot for catching that @ShnitzelKiller!

I just realized that despite the official (current) documentation saying that the argument name is "source", after upgrading my pytorch version, this code is what throws an error saying the argument name is "src"! I should probably notify you to revert this pull request then.
Maybe some parameters are error in document for distributed training ?
Author: ljch2018
Pull request: no

Based on [Distributed training document](https://huggingface.co/transformers/examples.html#id1) , one can use `bert-base-cased` model to fine-tune MR model and reaches very high score.

> Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD1.0:

```
python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
    --model_type bert \
    --model_name_or_path bert-base-cased \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/wwm_uncased_finetuned_squad/ \
    --per_gpu_train_batch_size 24 \
    --gradient_accumulation_steps 12
```

```
f1 = 93.15
exact_match = 86.91
```

**But based on [google bert repo](https://github.com/google-research/bert#squad-11) , the model `bert-base-cased` perfermance is**
```
{"f1": 88.41249612335034, "exact_match": 81.2488174077578}
```

Maybe the right pretrained model is `bert-large-uncased` ? 
Thanks~

Comments:
A question about BERT position embedding.
Author: DrDavidS
Pull request: no

## ❓ Questions & Help

I noticed that in Transformers, it turn position_ids into position_embedding through `nn.Embedding(config.max_position_embeddings, config.hidden_size)`

Here `config.max_position_embeddings` is 512 ,and `config.hidden_size` is 768.

So, when I input a sentence shorter than 512 ,such as "Today is a nice day", will this sentence's position embedding still be 512?  

Or just as long as position_ids? Here is 7 with [CLS] and [SEP].


Comments:
The sentence "Today is a nice day" will already be padded to 512 because of the tokenization process, so the input will remain 512 in the embedding layer

> The sentence "Today is a nice day" will already be padded to 512 because of the tokenization process, so the input will remain 512 in the embedding layer

Thanks a lot!
Tuning BERT on our own data set for multi-class classification problem
Author: rshah1990
Pull request: no
Labels: name: wontfix

I want to tune pre-trained BERT for multi-class classification with **6 million class, 30 million rows & highly imbalance data set.**  
Can we tune BERT in batch of classes? 
For example, I will take 15 classes (last layer will have only 15 neuron) and train my BERT model & in next batch use that trained model to train batch of next 15 classes, I just want to understand cons of this process.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Compatibility between DistilBert and Bert models
Author: pvgladkov
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I have a regular classification task for sentences in russian language. 
I used to train `BertForSequenceClassification` with pretrained Bert from [DeepPavlov](http://docs.deeppavlov.ai/en/master/features/models/bert.html) [RuBERT](http://files.deeppavlov.ai/deeppavlov_data/bert/rubert_cased_L-12_H-768_A-12_v2.tar.gz) (using PyTorch). Then I switched to `DistilBertForSequenceClassification`, but still using pretrained RuBert (because there is no pretrained DistilBert with russian language). And it worked. 

Then after [this changing](https://github.com/huggingface/transformers/pull/1203/commits/465870c33fe4ade66863ca0edfe13616f9d24da5#diff-9dc1f6db4a89dbf13c19d02a9f27093dL178) there is impossible to load `DistilBertConfig` from `BertConfig` config.json. `DistilBertConfig` uses property decorator for compatibility between `DistilBertConfig` and `BertConfig`, that's why using setattr() causes error `AttributeError: can't set attribute`.

So my question is following: Is it a bug or feature? Is it OK to load DistilBert from pretrained Bert or not? Or maybe the best way for me is to distill RuBERT by myself using your script for distillation?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Penalize high confident false negative classifications?
Author: callenilsson
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
I added the line `logits = torch.nn.functional.softmax(logits)` to convert binary classifications to a confidence score between 0.0 - 1.0. However, the predictions are very harsh being really close to either 0.0 or 1.0 and not somewhere in between. Is there a way to penalize the model from being so categorical? I especially want to minimize high confidence scores on false negatives.

Comments:
The softmax function specifically uses exponentiation to exacerbate the differences in scores (to get the soft 'max'). You can normalize scores by other means than a softmax.

Related to your title: using a log loss will penalize wrong predictions with high confidence more (e.g. BCE).

Related read is the section entitled "Don’t Mistake Class Probabilities for Confidence" here: https://www.inovex.de/blog/uncertainty-quantification-deep-learning/

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
revert erroneous fix #2276
Author: ShnitzelKiller
Pull request: yes

I based #2276 on having an error pop up on an older pytorch version, and also on the erroneous (current!) documentation for pytorch.Tensor.scatter():

> `scatter(dim, index, source)` → Tensor
> 
>     Out-of-place version of torch.Tensor.scatter_()
> 
> `scatter_(dim, index, src)` → Tensor
> ...

The argument was called `source`, inconsistently, in the version I was using, but somewhere along the way it went back to being `src` without the docs changing, which caused this confusion...

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2310?src=pr&el=h1) Report
> Merging [#2310](https://codecov.io/gh/huggingface/transformers/pull/2310?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/81db12c3ba0c2067f43c4a63edf5e45f54161042?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2310/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2310?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2310   +/-   ##
=======================================
  Coverage   73.54%   73.54%           
=======================================
  Files          87       87           
  Lines       14789    14789           
=======================================
  Hits        10876    10876           
  Misses       3913     3913
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2310?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2310/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `63.91% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2310?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2310?src=pr&el=footer). Last update [81db12c...e1844d9](https://codecov.io/gh/huggingface/transformers/pull/2310?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, probably best to just use positional arguments here (instead of keywords) then, don't you think?

Great, thanks!
the num_labels in run_squad
Author: zhujun5164
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
In the run_squad, I have not find any code to define the num_labels. In the modeling_utils, the num_labels have been default as 2, but in the Question & Answer task it may predict the start_position and end_position in the inputs. Is there the code have missing consider the reset of num_labels, or 2 is the right setting of num_labels for the task?
<!-- A clear and concise description of the question. -->


Comments:
Hey @zhujun5164, 2 is the right setting of num_labels for the task. If you look at the model they use (say Bert is BertForQuestionAnswering), you'll see that they get two outputs for each position which is from the num_labels = 2. The two outputs correspond to the start_logits position and the end_logits position.

```
        outputs = self.bert(input_ids,
                            attention_mask=attention_mask,
                            token_type_ids=token_type_ids,
                            position_ids=position_ids, 
                            head_mask=head_mask)

        sequence_output = outputs[0]

        logits = self.qa_outputs(sequence_output)
        start_logits, end_logits = logits.split(1, dim=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)
```

Does that make sense?

thanks @cformosa, and i find that in the untils_squad.py(341-357) the start_position and end_position have defined as a position number in the sentence (may be in token by word piece), it means that the run_squad predict the position number of start_position and end_position. The split(1, dim = -1) in your copy code is split half of the data in the last dim, so it easier to make me misunderstand it have predict one-hot of the start_position and end_position.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Correct tokenization for special and added tokens
Author: vitaliyradchenko
Pull request: yes

When a tokenizer is being loaded with `PreTrainedTokenizer._from_pretrained`, it should set `added_tokens` and `all_special_tokens` to `unique_added_tokens_encoder`. 
If we don't do it, it will corrupt the tokenization. 

Example:
```
import transformers
tokenizer = transformers.BertTokenizer.from_pretrained("bert-base-uncased")
tokenizer.tokenize("[CLS] token should not be splitted.")
# correct output
# ['[CLS]', 'token', 'should', 'not', 'be', 'split', '##ted', '.']
# incorrect output
# ['[', '[UNK]', ']', 'token', 'should', 'not', 'be', 'split', '##ted', '.']
```


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2312?src=pr&el=h1) Report
> Merging [#2312](https://codecov.io/gh/huggingface/transformers/pull/2312?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cea04a244351a7c5bce44e1cfc01abde0ceb60fd?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2312/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2312?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2312      +/-   ##
==========================================
+ Coverage   73.54%   73.54%   +<.01%     
==========================================
  Files          87       87              
  Lines       14789    14791       +2     
==========================================
+ Hits        10876    10878       +2     
  Misses       3913     3913
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2312?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2312/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `92.11% <100%> (+0.03%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2312/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `63.91% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2312?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2312?src=pr&el=footer). Last update [cea04a2...b262577](https://codecov.io/gh/huggingface/transformers/pull/2312?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This is awesome, thanks a lot @vitaliyradchenko
Bug: Tokenization of Special Tokens
Author: eladsegal
Pull request: no

## 🐛 Bug

The commit https://github.com/huggingface/transformers/commit/deceb001616995199a6a5dca866ffec95c3ebe74 introduces a bug in the tokenization of special tokens when using `from_pretrained` to initialize the tokenizer.

```
from transformers import AutoTokenizer

bert_tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
bert_text = "[CLS] An example for [MASK] change. [SEP]"
bert_tokenizer.tokenize(bert_text)
# Before: ['[CLS]', 'an', 'example', 'for', '[MASK]', 'change', '.', '[SEP]']
# After: ['[', '[UNK]', ']', 'an', 'example', 'for', '[', '[UNK]', ']', 'change', '.', '[', '[UNK]', ']']

roberta_tokenizer = AutoTokenizer.from_pretrained('roberta-base')
roberta_text = "
 An example for <mask> change. 
"
roberta_tokenizer.tokenize(roberta_text)
# Before: ['
', 'An', 'Ġexample', 'Ġfor', '<mask>', 'change', '.', '
']
# After: ['<', 's', '>', 'ĠAn', 'Ġexample', 'Ġfor', 'Ġ<', 'mask', '>', 'Ġchange', '.', 'Ġ</', 's', '>']
````

Fixed by https://github.com/huggingface/transformers/pull/2312.

Comments:
fix bug in prepare inputs for language generation for xlm for effective batch_size > 1
Author: patrickvonplaten
Pull request: yes

if multiple sentence are to be generated the masked tokens to be appended have to equal the effective batch size

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2289?src=pr&el=h1) Report
> Merging [#2289](https://codecov.io/gh/huggingface/transformers/pull/2289?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/81db12c3ba0c2067f43c4a63edf5e45f54161042?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2289/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2289?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2289      +/-   ##
==========================================
- Coverage   73.54%   73.52%   -0.02%     
==========================================
  Files          87       87              
  Lines       14789    14792       +3     
==========================================
  Hits        10876    10876              
- Misses       3913     3916       +3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2289?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/2289/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG0ucHk=) | `86.23% <0%> (-0.23%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2289/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.26% <0%> (-0.25%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2289?src=pr&el=footer). Last update [81db12c...f18ac4c](https://codecov.io/gh/huggingface/transformers/pull/2289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Indeed, thanks @patrickvonplaten