noamaanMulla-03/transformers-issues huggingface dataset:
ru language
Author: vvssttkk
Pull request: no

which pre-trained model can work for russian language? i want get only vectors

Comments:
of course i can use bert aka multi-language but they work very bad  in my mind for ru 

I think XLM models is better than mBERT. There are XLM models for 17 and 100 languages including ru.

cool
can u help me where i can download pre-trained xlm from ru because [here](https://huggingface.co/pytorch-transformers/pretrained_models.html) i can't find models for RU?

i think [here](https://github.com/facebookresearch/XLM) i can get pre-trained

@vtrokhymenko Yes, it's here

While wanting to understand how to convert a BERT Tensorflow model to one that works in pytorch-transformers, I stumbled upon RuBERT from DeepPavlov.

https://github.com/fredriko/bert-tensorflow-pytorch-spacy-conversion

or this 
```
import tensorflow as tf
  
from bert_dp.modeling import BertConfig, BertModel
from deeppavlov.models.preprocessors.bert_preprocessor import BertPreprocessor


bert_config = BertConfig.from_json_file('./rubert_cased_L-12_H-768_A-12_v1/bert_config.json')

input_ids = tf.placeholder(shape=(None, None), dtype=tf.int32)
input_mask = tf.placeholder(shape=(None, None), dtype=tf.int32)
token_type_ids = tf.placeholder(shape=(None, None), dtype=tf.int32)

bert = BertModel(config=bert_config,
                 is_training=False,
                 input_ids=input_ids,
                 input_mask=input_mask,
                 token_type_ids=token_type_ids,
                 use_one_hot_embeddings=False)

preprocessor = BertPreprocessor(vocab_file='./rubert_cased_L-12_H-768_A-12_v1/vocab.txt',
                                do_lower_case=False,
                                max_seq_length=512)

with tf.Session() as sess:

    # Load model
    tf.train.Saver().restore(sess, './rubert_cased_L-12_H-768_A-12_v1/bert_model.ckpt')

    # Get predictions
    features = preprocessor(["Bert z ulicy Sezamkowej"])[0]

    print(sess.run(bert.sequence_output, feed_dict={input_ids: [features.input_ids],
                                                    input_mask: [features.input_mask],
                                                    token_type_ids: [features.input_type_ids]}))

    features = preprocessor(["Это", "Берт", "с", "Улицы", "Сезам"])[0]

    print(sess.run(bert.sequence_output, feed_dict={input_ids: [features.input_ids],
                                                    input_mask: [features.input_mask],
                                                    token_type_ids: [features.input_type_ids]}))
```
Bert's keyword argument 'output_all_encoded_layers' does not exist anymore?
Author: fhamborg
Pull request: no

## 📚 Migration

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)

Details of the issue:

I was using the keyword argument `output_all_encoded_layers` before. Now the code throws an error, since it seems that this argument was removed. How can I still set `output_all_encoded_layers` to either True or False, e.g.:

```
context, _ = self.bert(context, output_all_encoded_layers=False)
```

## Checklist

- [X] I have read the migration guide in the readme.
- [X] I checked if a related official extension example runs on my machine.



Comments:
Hi! @thomwolf can correct me if I'm wrong, but I believe this keyword was changed to `output_hidden_states` in version 1.0.0.

I can confirm what @LysandreJik suggests. The output of the embeddings is now also included as the first element. 

Alright, thank you very much!
How to speedup BERT eval
Author: abaheti95
Pull request: no

## ❓ Questions & Help
Is there a simple way to speedup `.eval()` when using the BERT model
Specifically I am using `BertForSequenceClassification`. I have finetuned a the model separately on my own data and I am trying to get hidden representations after doing `model.eval()` as follows:
`last_hidden_layer, all_hidden_states = model(input_ids)`

However for each input it is taking about 2.3 seconds on `cpu` and 2.6 seconds on `gpu`. Is there a way where I can do faster than this?

Comments:
Did you try using DistilBERT? Inference should be ~ 60% faster

Turns out I wasn't using the using the gpu correctly. I moved the model and the inputs to gpu by doing `.to(device)` and it became 100x faster. Thanks for the suggestion.
No parameter which is presented in documentation
Author: CrafterKolyan
Pull request: no

Documentation web page: https://huggingface.co/pytorch-transformers/model_doc/gpt2.html#pytorch_transformers.GPT2Model
See `Inputs -> attention_mask`.

There is actually no parameter `attention_mask` in `GPT2Model.forward` method (see https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_gpt2.py#L473)

Of course trying to provide `attention_mask` parameter to model raises an exception:
> TypeError: forward() got an unexpected keyword argument 'attention_mask'

Please either add parameter `attention_mask` to `GPT2Model.forward` or delete it from documentation.

Same for https://huggingface.co/pytorch-transformers/model_doc/gpt2.html#pytorch_transformers.GPT2LMHeadModel
and for
https://huggingface.co/pytorch-transformers/model_doc/gpt2.html#pytorch_transformers.GPT2DoubleHeadsModel

I've created #1116 in case you want just delete it from documentation.

Comments:
Indeed, thanks for that!

Sorry, but I still get this error, and I can see the parameter on the forward functions at the source code. Am I doing something wrong ?

Thanks for this amazing contribution!

Can you open a new issue with details on the error?
`attention_mask` has been added to GPT2 now so it's not the same situation.

Sorry, my problem was that there was no attention_mask parameter on the forward function, but I can see it now. Thanks
Generate Variable Length Text With GPT2
Author: neild0
Pull request: no

This might be obviously explained in the documentation, but I've been browsing through the code for a while and can't seem to find a resolution, so thank you in advance for your help.

As demoed with Write with Transformers, it seems to generate variable length text suggestions. I was wondering how this would be possible with the transformers library given, and how it would be possible to interface the largest version of GPT2 to do so.

Thank you!

Comments:
Hi! In Write With Transformer, we use the context to predict the following token. We then add that token to the initial context, to generate the following one. This way we can generate long sequences according to a given token.

In that app we stop generating tokens once we have reached a given time, or once we have seen an end of sentence token. We, therefore, don't generate variable length text suggestions, we just adjust the batch according to end tokens identified in our results.

Thanks for the explanation!
Help loading BioBERT weights
Author: happypanda5
Pull request: no
Labels: name: wontfix

I have completed the following:

**1. Downloaded pretrained BioBERT weights from their current release**

**2. Convert TensorFlow checkpoints into Pytorch weights bin file using the following code**

import os

os.system( ' pytorch_pretrained_bert convert_tf_checkpoint_to_pytorch \
  "/content/biobert_v1.1_pubmed/model.ckpt.index" \
  "/content/biobert_v1.1_pubmed/bert_config.json" \
  "/content/biobert_pytorch.bin"            ' )

`
**3. I then tried to test whether I can load these weights. In order to do so, I tried the following code**

state_dict = torch.load( "/content/biobert_pytorch.bin" )
model.load_state_dict(state_dict)

**but I get the error**

> IncompatibleKeys(missing_keys=[], unexpected_keys=[])

***Please guide me***

Comments:
Not really providing a solution here, but have you considered https://github.com/allenai/scibert instead?
AllenAI provides PyTorch weights, and through tests they claim their model is superior https://arxiv.org/pdf/1903.10676.pdf on their suite of tasks. For that and for ease of use, it may be a valid alternative.

Hello, I am trying to figure out how to load the SciBert weights. I see that you can use

```
# Simple serialization for models and tokenizers
model.save_pretrained('./directory/to/save/')  # save
model = model_class.from_pretrained('./directory/to/save/')  # re-load
tokenizer.save_pretrained('./directory/to/save/')  # save
```

So my guess is to download them from here
https://github.com/allenai/scibert#pytorch-models

Untar them, then point to that directory

```
model = model_class.from_pretrained('DIRECTORY/TO/DOWNLOADED/UNZIPPED/SCIBERT/Pytorch.bin')  
```

Unless there is more to that, the part I am confused about is that SciBert also has it's own vocab and 'vocab.txt' file. I am wondering how to point to that file, and not the default one. 

Edit

Found the answer

https://github.com/huggingface/pytorch-transformers/issues/69#issuecomment-443215315

you can just do a direct path to it

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is it possible to modify the parameters in GPT-2?
Author: weiguowilliam
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I wonder whether it's possible to modify the parameters in GPT-2? Since we can not train GPT-2, modifying the parameters and observing the changes in results will be helpful. Thank you in advance!

Comments:
Hi! GPT-2, like all models in this library, directly inherit from pytorch's `nn.Module`, so you're free to finetune them or modify their parameters as you wish.

> Hi! GPT-2, like all models in this library, directly inherit from pytorch's `nn.Module`, so you're free to finetune them or modify their parameters as you wish.

Thank you for your help!
DistillBert Documentation Code Example fixes
Author: dharmendrach
Pull request: yes

Following code examples in the documentation are throwing errors:-

1. [DistilBertForQuestionAnswering](https://huggingface.co/transformers/model_doc/distilbert.html#transformers.DistilBertForQuestionAnswering)

```
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
start_positions = torch.tensor([1])
end_positions = torch.tensor([3])
outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions)
loss, start_scores, end_scores = outputs[:2]
```
> ValueError: not enough values to unpack (expected 3, got 2)


2. [TFDistilBertForMaskedLM](https://huggingface.co/transformers/model_doc/distilbert.html#transformers.TFDistilBertForMaskedLM)

```
import tensorflow as tf
from transformers import DistilBertTokenizer, TFDistilBertForMaskedLM

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = TFDistilBertForMaskedLM.from_pretrained('distilbert-base-uncased')
input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)
prediction_scores = outputs[0]

```
> TypeError: call() got an unexpected keyword argument 'masked_lm_labels'

3. [TFDistilBertForQuestionAnswering](https://huggingface.co/transformers/model_doc/distilbert.html#transformers.TFDistilBertForQuestionAnswering)

```
import tensorflow as tf
from transformers import BertTokenizer, TFDistilBertForQuestionAnswering

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = TFDistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased')
input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
start_positions = tf.constant([1])
end_positions = tf.constant([3])
outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions)
start_scores, end_scores = outputs[:2]
```
> TypeError: call() got an unexpected keyword argument 'start_positions'

The first issue is just list indexing issue. Second and Third are due to implementation difference between Tensorflow and Pytorch DistillBERT. Tensorflow implementation doesn't have loss calculation inside `call`, but We do in `forward` for Pytorch.  I have updated code examples in the docstring.

Let me know if you will be interested in a pull request for making same function API structure for Tensorflow implementation via adding loss calculation in `call function` similar to Pytorch.

This is my first issue. Let me know if you require any changes on pull request.

Regards 😃 
Dharmendra 

Comments:
Indeed, thanks for the PR @drc10723 !
Added ValueError for duplicates in list of added tokens
Author: danai-antoniou
Pull request: yes

Very small addition to raise an error if the list of tokens passed to `add_tokens` contains duplicates. This otherwise raises cryptic errors down the line. Happy to update it to `Warning` if someone believes there's any reason for duplicates to be allowed here.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1296?src=pr&el=h1) Report
> Merging [#1296](https://codecov.io/gh/huggingface/transformers/pull/1296?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/391db836ab7ed2ca61c51a7cf1b135b6ab92be58?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1296/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1296?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1296   +/-   ##
=======================================
  Coverage   84.72%   84.72%           
=======================================
  Files          84       84           
  Lines       12591    12591           
=======================================
  Hits        10668    10668           
  Misses       1923     1923
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1296?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1296/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `91.48% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1296?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1296?src=pr&el=footer). Last update [391db83...a951585](https://codecov.io/gh/huggingface/transformers/pull/1296?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok, great, thanks @danai-antoniou!
add support for file I/O
Author: rajarsheem
Pull request: yes

sometimes we need to process multiple prompts from a file and generate multiple sequences. also, writing results to a file would be less verbose and fast. 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1297?src=pr&el=h1) Report
> Merging [#1297](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1297?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/0d1dad6d5323cf627cb8d7ddd428856ab8475f6b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1297/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1297?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1297   +/-   ##
=======================================
  Coverage   80.77%   80.77%           
=======================================
  Files          57       57           
  Lines        8092     8092           
=======================================
  Hits         6536     6536           
  Misses       1556     1556
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1297?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1297?src=pr&el=footer). Last update [0d1dad6...2a11412](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1297?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi @rajarsheem ! Thank you for your PR. With the example scripts, we are really reaching for **simple scripts that showcase how the library works** and how it interacts with different elements of the Pytorch codebase (ex: distributed learning, gradient clipping, ...).

Using a text file as input may be useful in some cases, however, I don't feel like it really gives a deeper understanding of the library, as it is just a different way to obtain a context string. I don't think it would be particularly worth it in terms of the added complexity/deeper understanding of the lib ratio.

Please don't let that discourage you from opening other PRs.
Add option to use a 'stop token'
Author: enzoampil
Pull request: yes

This will be used to truncate the output text to everything till right before the 'stop token'. If the 'stop token' is not found, then the whole text will be returned based on the specified 'length'.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1313?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@ecc4f1b`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1313/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1313?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master    #1313   +/-   ##
=========================================
  Coverage          ?   84.72%           
=========================================
  Files             ?       84           
  Lines             ?    12591           
  Branches          ?        0           
=========================================
  Hits              ?    10668           
  Misses            ?     1923           
  Partials          ?        0
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1313?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1313?src=pr&el=footer). Last update [ecc4f1b...d3f24df](https://codecov.io/gh/huggingface/transformers/pull/1313?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks!
❓ Why the criterion of XLNet LMHeadModel use ignore_index = -1 ?
Author: astariul
Pull request: no

In the XLNetLMHeadModel, the criterion used to compute the loss uses `ignore_index=-1` :

https://github.com/huggingface/pytorch-transformers/blob/9f995b99d4c4067662c3bd4f1274315c0839deeb/pytorch_transformers/modeling_xlnet.py#L927-L931

**Why ?**
Isn't it supposed to ignore the padding index ID, i.e. 5 ?

Comments:
you should set padding and all the labels to be ignored to -1. In Bert/XLNet training, we usually only use 15% of the tokens as labels.
Wrong layer names for selecting parameters groups (run_openai_gpt.py)
Author: saareliad
Pull request: no
Labels: name: wontfix

Hi,
In this script [run_openai_gpt.py](https://github.com/huggingface/pytorch-transformers/blob/master/examples/single_model_scripts/run_openai_gpt.py)

Parameter names for selecting param groups are wrong.
Should be:

`  
    no_decay = no_decay = ['bias', 'ln_1.bias', 'ln_1.weight', 'ln_2.bias', 'ln_2.weight']
`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Printing Iteration every example problem
Author: AhmedBahaaElDinMohammed
Pull request: no
Labels: name: wontfix

```
Iteration:   0%|          | 1/250 [00:00<03:28,  1.19it/s]




Iteration:   1%|          | 2/250 [00:01<03:21,  1.23it/s]




Iteration:   1%|          | 3/250 [00:02<03:17,  1.25it/s]




Iteration:   2%|▏         | 4/250 [00:03<03:14,  1.27it/s]




Iteration:   2%|▏         | 5/250 [00:03<03:11,  1.28it/s]




Iteration:   2%|▏         | 6/250 [00:04<03:09,  1.29it/s]




Iteration:   3%|▎         | 7/250 [00:05<03:05,  1.31it/s]
```
I dont know what is the error that is causing this error , can somebody help ?
and this is the code 
``` 
train_iterator = trange(int(num_train_epochs), desc="Epoch", disable= local_rank not in [-1, 0])
    set_seed(42)
 
    Epochs = 0
    for _ in train_iterator:
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable= local_rank not in [-1, 0])
        #print('Check')
        Epochs = Epochs + 1
        for step, batch in enumerate(epoch_iterator):
            model.train()
```


Comments:
Are you running in jupyter? This might be an artifact of how `tqdm` is interacting with whatever shell you're running it in. If you don't want to print anything, you could simply drop the `tdqm` wrapper and just iterate over `train_dataloader`. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Avoid i/o in class __init__ methods
Author: honnibal
Pull request: no
Labels: name: wontfix

Working with model serialization and configs is pretty painful, and we went through a lot of design iterations on this for spaCy.

I think one thing that's definitely unideal in `pytorch_transformers` is that the tokenizers often expect file names in the `__init__` methods. This means that if you're holding the data in memory, you have to first write it to file in order to create the class.

I think it would be nicer to move the load-from-disk part into a method, that could be called after the `__init__`. This wouldn't really change the usage of the classes, since mostly people are using the `.from_pretrained()` class method, but it would make the classes a bit more flexible.

Comments:
Make sense to me. I'll include that in a coming PR.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.