noamaanMulla-03/transformers-issues huggingface dataset:
Serializing XLMRobertaTokenizer
Author: brandenchan
Pull request: yes

I am currently trying to use the XLMRobertaTokenizer in a multiprocessor setting. To do this, the XLMRobertaTokenizer needs to be serializable. Currently XLMRobertaTokenizer is not serializable while other tokenizers such as AlbertTokenizer are.

This PR adds the __getstate__ and __setstate__ methods to XLMRobertaTokenizer so that it can be serialized.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2414?src=pr&el=h1) Report
> Merging [#2414](https://codecov.io/gh/huggingface/transformers/pull/2414?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0ffc8eaf53542092271a208a52e881668e753e72?src=pr&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `16.66%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2414/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2414?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #2414      +/-   ##
=========================================
- Coverage   73.24%   73.2%   -0.05%     
=========================================
  Files          87      87              
  Lines       14989   15000      +11     
=========================================
+ Hits        10979   10980       +1     
- Misses       4010    4020      +10
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2414?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2414/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtX3JvYmVydGEucHk=) | `32.91% <16.66%> (-3.86%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2414?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2414?src=pr&el=footer). Last update [0ffc8ea...1d332a7](https://codecov.io/gh/huggingface/transformers/pull/2414?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Is anything blocking from merging this? :)
Would help us a lot with parallelizing the preprocessing!

Awesome! Thanks @LysandreJik  :)
mems output in XLNet
Author: drasros
Pull request: no

## 🐛 Bug

Hi, 
I am trying to use memory of the last forward pass (mems arg) with XLNet. I am getting a tuple of None as mems output instead of a tuple of tensor. The same code with TransformerXL is running fine. Am I doing anything wrong or is this a bug ? Below is a short code snippet to reproduce the error. 
Many thanks,
A

```python
import torch
from pytorch_transformers import TransfoXLTokenizer, TransfoXLModel, XLNetTokenizer, XLNetModel

text = ['This is the first sentence. ', 'And this is another one']

# transformer-XL
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
model = TransfoXLModel.from_pretrained('transfo-xl-wt103')
mems = None
for i in range(2):
    input_ids = torch.tensor(tokenizer.encode(text[i])).unsqueeze(0) 
    outputs = model(input_ids, mems=mems)
    mems = outputs[1]
# RUNS OK

# XLNet
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
model = XLNetModel.from_pretrained('xlnet-large-cased')
mems = None
for i in range(2):
    input_ids = torch.tensor(tokenizer.encode(text[i])).unsqueeze(0)
    outputs = model(input_ids, mems=mems)
    mems = outputs[1]
# We get tuple of None in first model output, second forward crashes.
# File "/home/asors/anaconda3/envs/psco/lib/python3.7/site-packages/pytorch_transformers/modeling_xlnet.py", line 858, in forward
#     mlen = mems[0].shape[0] if mems is not None else 0
```

<!-- Important information -->

Model I am using: XLNet
Language I am using the model on (English, Chinese....):

* OS: Ubuntu
* Python version: 3.7
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.1.0

Comments:
Indeed I see, we have to add:
- an explanation in XLM's docstring that you should set the model configuration `mem_len` parameter if you want to use the memory (the answer to your main question). You can do for instance `model = XLNetModel.from_pretrained('xlnet-large-cased', mem_len=1024)` if you want a max memory of 1024 tokens. By default the model doesn't use memory (`mem_len = None`)
- a check in the code to avoid the error you are reporting.

Thanks for your reply and the new doc!
Gpt2/xl Broken on "Write With Transformer" site
Author: crackerjam
Pull request: no

If you navigate to the [GPT-2 section of the Write With Transformer site](https://transformer.huggingface.co/doc/gpt2-large), select gpt2/xl, and try to generate text, the process will not generate anything.

Comments:
Same problem. And it seems not fixed yet.

It should be fixed now. Thank you for raising this issue.
Adding scibert in the list of pre-trained models?
Author: aCampello
Pull request: no

# 🌟New model addition

## Model description

Would it be possible/is it in the pipeline to add SCIBERT as one of the pre-trained models for Bert? Could be as simple as adding it to the `BERT_PRETRAINED_MODEL_ARCHIVE_MAP`.

## Open Source status

Scibert is available on its own repository (https://github.com/allenai/scibert). The advantage of adding it to the map is that we don't need to download it ad-hoc every time we want to use it, it would be cached in the same repository, etc.

* [x] the model implementation is available: 
* [x] the model weights are available: 
* [x] who are the authors: Iz Beltagy and Kyle Lo and Arman Cohan authored the ALlenAI

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
It would be nice if AllenAI uploaded their models to [the user hub](https://huggingface.co/models). That would allow you to simply load the models like `.from_pretrained('allenai/scibert-scivocab-uncased')`. Perhaps you can open an issue on their repository and ask whether that is possible. It might be too much work/maintenance for them, though.

Right, I missed that 2.2.2 update for model sharing (https://huggingface.co/transformers/model_sharing.html). So you're right, probably the best thing is for them to upload their model.

Might be best to close this issue here and keep everything in the issue that you created over at AllenAI.

Yeah, makes sense. Thanks.
Conversion of [Model]ForSequenceClassification logits to probabilities
Author: ghost
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I noticed that whenever I would convert logits coming from the model to probabilities using the following equation:

probability = e^logit/(1 + e^logit)

The probabilities for my two classes do not add up to 1. Why is this?


Comments:
Logits are the classification scores before softmax.
I don't see why the results of your equation should add up to 1.
You may try to use the softmax function directly.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Prediction on NER Tensorflow 2
Author: imayachita
Pull request: no

Hi,
I tried running the implementation of NER on Tensorflow 2. I have a problem doing the prediction. Seems like the label to index are off. Here is some examples:

```
SOCCER B-ORG
- B-ORG
JAPAN B-MISC
GET B-ORG
LUCKY B-MISC
WIN B-ORG
, B-ORG
CHINA B-MISC
IN B-ORG
SURPRISE O
DEFEAT B-ORG
. B-ORG

Nadim B-MISC
Ladki B-PER

AL-AIN I-PER
, B-ORG
United I-PER
Arab I-MISC
Emirates I-MISC
1996-12-06 B-ORG

Japan I-PER
began B-ORG
the B-ORG
defence B-ORG
of B-ORG
their B-ORG
Asian O
Cup I-ORG
title B-ORG
with B-ORG
a B-ORG
lucky B-ORG
2-1 B-ORG
win B-ORG
against B-ORG
Syria I-PER
in B-ORG
a B-ORG
Group B-ORG
C I-ORG
championship B-ORG
match B-ORG
on B-ORG
Friday B-ORG
. B-ORG
```

Anyone has the same problem? Thanks!

Comments:
Yes, the output is wrong. 
I think the run_tf_ner.py script has a bug where the labels are off by 1. 
And the off-by-1 prediction result is sent for evaluation `metrics.classification_report(y_true, y_pred, digits=4)` therefore the evaluation result is wrong too. 

Thanks @HuiyingLi! The workaround works.
[CamemBert] About SentencePiece training
Author: loretoparisi
Pull request: no

## ❓ Questions & Help

According to the paper, SentencePiece uses a vocabulary of size of 32k subword tokens, learned on 107 sentences sampled from the pretraining dataset. How the sampling was performed? The chosen size of the vocabulary (32K subwords token) is related to the pretraining dataset in some way? Or it is an arbitrary choice?

Thank you.

Comments:
ping author @louismartin :)

Hi @loretoparisi, 
We sampled 10**7 lines randomly from the pretraining corpus.
The size of the vocabulary was chosen to somewhat match the original BERT paper which used a 30k wordpiece vocabulary, so yes it's mostly arbitrary. 

@louismartin thanks a lot for the details. I was wondering about the 32k size if this could have been biased by the language...w

Yes maybe there is a more adapted vocabulary size, we did not investigate that :)

Can I close the issue now?

Adding a reference to https://github.com/google/sentencepiece/issues/415

@louismartin: 
> We sampled 10**7 lines randomly from the pretraining corpus.

May I ask how did you come up with that number? I'm trying to figure out how many lines I should select to train a model. Assuming I have access to 1 billion rows of ngrams with mean length of 7 words; I'm not sure how many random lines/ngrams would be enough to train a tokenizer with fixed vocab of size 50k?

I think this is more of a resource allocation question. How much time or compute do you want to allocate to training your tokenizer? Alternate phrasing: why wouldn't you train your tokenizer on the full corpus?

@julien-c Well sure, that's a valid point. In theory I can train a tokenizer on full corpus by setting the fixed size for the vocabulary. It will just take more and more time (& possibly more compute resources) with increased size of dataset. I was wondering if there's any correlation between quality of fixed threshold vocabulary generated from increasing size of training dataset. I can see that this may be task dependent and requires iterative experiments. Is there any paper that I can look into regarding this? Thanks! 

PS: did you check out [`tokenizers`](https://github.com/huggingface/tokenizers)? It is pretty fast 😄 

I've trained a byte-level BPE on 10 GB of text in ~15 minutes.

@julien-c are you suggesting that, thanks to 🤗 amazing library `tokenizer` we could potentially train the sentence piece tokenizer without setting up a boundary?
This means that, potentially, current models could improve a lot: from 32K subwords tokens to let's say 1M, what will happen?

Those are two different things: size of vocab, and size of corpus that you train your tokenizer on.

@julien-c that's true, and it also seems to have no clear relation in terms of final overall accuracy. Let's say we take as metrics the PPL, and we consider a fixed corpus size and vary the vocab size in batches of 8K, like: 8K, 16K, 32K, 64K, 128K, until we are closer to the whole size of the non unique tokens vocabulary. What will be the related PPL for each training?

(PPL or BLEU, or other...)
How to fine-tune BERT on a large training dataset?
Author: wcgan
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi,

I want to fine-tune BERT on a large training dataset. With around 1.5million training examples, this is currently consuming around 60GB of RAM. Is there any way to reduce the RAM usage or load the training examples in parts? Thanks!.


Comments:
I think you should check out pytorch dataloaders https://pytorch.org/docs/stable/data.html
Also, gradient accumulation is helpful.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Wrong definition of the `logging_steps` parameter at the `run_lm_finetuning.py`
Author: iedmrc
Pull request: no
Labels: name: wontfix

Hi,
`logging_steps` parameter defined as `"Log every X updates steps."` but it's been affected by `gradient_accumulation_steps` as follows:

https://github.com/huggingface/transformers/blob/f3386d938348628c91457fc7d8650c223317a053/examples/run_lm_finetuning.py#L243-L254

Therefore, if you, e.g., set logging_steps=1000 and  gradient_accumulation_steps=5, it'll log in every 5000 steps. That affects `evaluate_during_training` in a not intended way.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
improve final answer extraction in utils_squad.py
Author: adai183
Pull request: no
Labels: name: wontfix

Shouldn't ` get_final_text` use the specific optionally pre-trained tokenizer instead of generically using `BasicTokenizer` ?

[examples/utils_squad.py L911](https://github.com/huggingface/transformers/blob/6596e3d56626c921b3920e313866b7412633b91a/examples/utils_squad.py#L911)

Comments:
Any update?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Classify entities - run_ner script 
Author: ghost
Pull request: no
Labels: name: wontfix

Hello all, 
I am just wondering what extra input to the "BertForTokenClassification"  if I want to classify the entities TO (PER, LOC ..) . Note that the entities are given in advance. I used run_ner script but it extracts the entities and classify them (the extraction is not needed). I did not get how the script(or the input) can be modified for my task? any idea? 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is the usage of scheduler described in README correct?
Author: tamuhey
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/transformers/blame/master/README.md#L546

I think it's not here to do `scheduler.step` but in epoch loop.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Distilling GPT2 with gives OOM
Author: snaik2016
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Distilling GPT2 gives OOM what is the best way to fit both teacher student in single GPU and train?
Tried reducing batch size but that itself results into an error.

  File "train.py", line 285, in main
    distiller.train()
  File "trabsformersexamples\distillation\distiller.py", line 340, in train
    self.step(input_ids=token_ids, attention_mask=attn_mask, lm_labels=lm_labels)
  File "trabsformersexamples\distillation\distiller.py", line 378, in step
    s_logits, _, s_hidden_states = self.student(input_ids=input_ids, attention_mask=None)            # (bs, seq_length, voc_size)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "conda\conda\envs\pytorch\lib\site-packages\transformers\modeling_gpt2.py", line 549, in forward
    inputs_embeds=inputs_embeds)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "conda\conda\envs\pytorch\lib\site-packages\transformers\modeling_gpt2.py", line 439, in forward
    inputs_embeds = self.wte(input_ids)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\modules\sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.IntTensor instead (while checking arguments for embedding)




Comments:
Do you need pretrain distilgpt2 from scratch? You can consider just finetuning it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix token_type_ids for XLM-R
Author: MaksymDel
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2653?src=pr&el=h1) Report
> Merging [#2653](https://codecov.io/gh/huggingface/transformers/pull/2653?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/babd41e7fa07bdd764f8fe91c33469046ab7dbd1?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2653/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2653?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2653   +/-   ##
=======================================
  Coverage   74.58%   74.58%           
=======================================
  Files          87       87           
  Lines       14892    14892           
=======================================
  Hits        11107    11107           
  Misses       3785     3785
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2653?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2653/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtX3JvYmVydGEucHk=) | `32.91% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2653?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2653?src=pr&el=footer). Last update [babd41e...56133cd](https://codecov.io/gh/huggingface/transformers/pull/2653?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix AutoModelForQuestionAnswering for Roberta
Author: tholor
Pull request: yes

When using `AutoModelForQuestionAnswering()` to load a Roberta model, we are currently instantiating a `BertForQuestionAnswering` class. This is happening because `RobertaConfig` is an instance of `BertConfig` (due to inheritance) and there's no other mapping for Roberta in here:

https://github.com/huggingface/transformers/blob/bac51fba3a6b96f02f482e9a352601242b200e47/src/transformers/modeling_auto.py#L176-L184




Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2655?src=pr&el=h1) Report
> Merging [#2655](https://codecov.io/gh/huggingface/transformers/pull/2655?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/babd41e7fa07bdd764f8fe91c33469046ab7dbd1?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2655/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2655?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2655   +/-   ##
=======================================
  Coverage   74.58%   74.58%           
=======================================
  Files          87       87           
  Lines       14892    14892           
=======================================
  Hits        11107    11107           
  Misses       3785     3785
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2655?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2655/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `75.55% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2655?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2655?src=pr&el=footer). Last update [babd41e...213877a](https://codecov.io/gh/huggingface/transformers/pull/2655?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


can you run `make style` @tholor? Seems like I can't push to your fork or I'd have done it myself.

Thank you!

Actually I'll do it myself. Thanks!
Add AutoModelForPreTraining
Author: thomwolf
Pull request: yes

Add `AutoModelForPretraining` and `TFAutoModelForPretraining` classes which will load the full model used for pretraining (guarantee we should have all the pre-trained weights).

This class can be used for instance to convert between an original PyTorch and a TF2.0 models while being sure that all the pretrained weights are converted:

```python
# PyTorch => TF 2.0 (save TF 2.0 weights from PT weights)
tf_model = TFAutoModelForPretraining.from_pretrained('my-model', from_pt=True)
tf_model.save_pretrained()

# TF 2.0 => PyTorch (save PT weights from TF 2.0 weights)
pt_model = AutoModelForPretraining.from_pretrained('my-model', from_tf=True)
pt_model.save_pretrained()
```

Comments: