noamaanMulla-03/transformers-issues huggingface dataset:
Using new pretrained model with it's own vocab.txt file. 
Author: Santosh-Gupta
Pull request: no
Labels: name: wontfix

I am trying to use SciBert pretrained weights, which has its own vocab , so it's own 'vocab.txt', file. 

I think it's fairly straight forward to point the `pytorch_model.bin` but I do not see any options to introduce a new vocab.txt file. 

Comments:
Found the answer

https://github.com/huggingface/pytorch-transformers/issues/69#issuecomment-443215315

you can just do a direct path to it

Can it work? I tried the solution but didn't work. I put the vocab.txt file under a certain path.

What error message did you get? Maybe try an absolute path to the file. 

i

> What error message did you get? Maybe try an absolute path to the file.

It only works when you store the vocab.txt in `/tmp` which is the default `cache_dir`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
if cutoffs=[], convert_transfo_xl_checkpoint_to_pytorch.py has a bug
Author: Pydataman
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): transformer_xl

Language I am using the model on (English, Chinese....): I train xl model base on own dataset

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: centos 7.3
* Python version: python3.6
* PyTorch version: torch1.1
* PyTorch Transformers version (or branch): 1.0.0
* Using GPU: yes
* Distributed of parallel setup ? no
* Any other relevant information: no 

## detail context

<!-- Add any other context about the problem here. -->

AttributeError: 'ProjectedAdaptiveLogSoftmax' object has no attribute  'cluster_weight'

I see 'ProjectedAdaptiveLogSoftmax' code, if len(cutoffs) - 1 > 0,  will have attribute  'cluster_weight'


Comments:
Hi!

Yes, you should indeed specify `cutoffs` in your `TransfoXLConfig` or the adaptive softmax won't be able to create its clusters. We should probably put a more explicit error.

Hello, @LysandreJik

The checkpoint of https://github.com/kimiyoung/transformer-xl doesn't have cutoff_N when adaptive softmax is not used. Does PyTorch-Transformers support TransfoXLConfig.adaptive = False? If supported, should it read checkpoint without explicit error? The content of checkpoint is like this without adaptive softmax.

```
transformer/adaptive_embed/lookup_table (DT_FLOAT) [32768,512]
transformer/adaptive_embed/lookup_table/Adam (DT_FLOAT) [32768,512]
transformer/adaptive_embed/lookup_table/Adam_1 (DT_FLOAT) [32768,512]
transformer/adaptive_softmax/bias (DT_FLOAT) [32768]
transformer/adaptive_softmax/bias/Adam (DT_FLOAT) [32768]
transformer/adaptive_softmax/bias/Adam_1 (DT_FLOAT) [32768]
transformer/layer_0/ff/LayerNorm/beta (DT_FLOAT) [512]
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix run_bertology.py
Author: adrianbg
Pull request: yes

Make imports and `args.overwrite_cache` match `run_glue.py`.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1765?src=pr&el=h1) Report
> Merging [#1765](https://codecov.io/gh/huggingface/transformers/pull/1765?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1c542df7e554a2014051dd09becf60f157fed524?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1765/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1765?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1765   +/-   ##
=======================================
  Coverage   84.03%   84.03%           
=======================================
  Files          94       94           
  Lines       14032    14032           
=======================================
  Hits        11792    11792           
  Misses       2240     2240
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1765?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1765?src=pr&el=footer). Last update [1c542df...c199d96](https://codecov.io/gh/huggingface/transformers/pull/1765?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


That looks great, thanks @adrianbg !
How to install previous versions of pytorch-transformers 
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I am using some codes requiring old versions of your work, do you mind telling me 
how to install old version of this repository?
thanks
Best
Julia 

Comments:
You can install an older version using the standard pypi procedure: `pip install pytorch-transformers==$VERSION`. The examples showcased on this repository probably won't work on older versions though.

HI
Thanks a lot, I need though a version which I think was called
pretrained_bert , thanks for your help.
Best
Julia

On Mon, Sep 2, 2019 at 5:25 PM Lysandre Debut <notifications@github.com>
wrote:

> You can install an older version using the standard pypi procedure: pip
> install pytorch-transformers==$VERSION. The examples showcased on this
> repository probably won't work on older versions though.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/issues/1177?email_source=notifications&email_token=AM3GZM62DOTECUG7OSEG57TQHUV5FA5CNFSM4IS35RG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5WDCZY#issuecomment-527184231>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AM3GZMYYXIZA76ZWLGLSTK3QHUV5FANCNFSM4IS35RGQ>
> .
>


I believe you can still install it with `pip install pytorch-pretrained-BERT==$VERSION`!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Extracting the output layer of HuggingFace GPT2DoubleHeadsModel
Author: h56cho
Pull request: no

Hello,

Suppose that I have two GPT2DoubleHeadsModel (let’s call it model A and B).
Is there any way that I can: 

1. take the hidden state of a given input at the n-th layer of the model A and feed it directly into the output layer of model B to compute output

AND

2. Take the output obtained from 1. and calculate the loss based on an appropriate label 

Some coding example would be a great help.

Thank you,


Comments:
Please do not open duplicate issues (#1774)
Fine-tuning (BERT & RoBERTa) base outperforms large
Author: wahlforss
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

On all datasets I have used so far, base always outperforms large after the fine-tune. This is true for both BERT and RoBERTa. 

Why is that? Am I doing something wrong? Does large require far more epochs to train or a different learning rate?

Comments:
Maybe you need more GPUs and a bigger batch size

I have 8 GPUs 2080 rtx, each with 10gb of data. But yeah I use a batch size of 4. However, I need the sequence length of 512 so it is impossible to increase the batch size.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Convert RoBERTa to TF checkpoint
Author: YoPatapon
Pull request: no
Labels: name: wontfix

Can we use the "convert_pytorch_checkpoint_to_tf" script to convert the RoBERTa checkpoint to the Tensorflow ckpt? Thanks.

Comments:
Hello, I believe that unfortunately the script currently only works for the `BertModel` base class. You would have to create a similar script for RoBERTa, it shouldn't be too different as both models have very similar architectures!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
LM fine tuning on top of a custom model
Author: ZhaofengWu
Pull request: no
Labels: name: Discussion

Currently the `finetune_on_pregenerated.py` script allows only fine-tuning on top of one of the five pretrained bert models. However I don't understand why there is such a restriction. I am trying to finetune a LM on top of a custom bert model (mt-dnn). Of course I can just remove the `choices`, but I am wondering if there is some rationale behind this, as none of the other example scripts contains this restriction.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/94980b529fad34f55c5d34c94bae2814db6773a6/examples/lm_finetuning/finetune_on_pregenerated.py#L126-L128

Comments:
Not really, I guess we could remove this restriction and just let people point to an arbitrary folder (as we do in the other scripts).
What do you think @Rocketknight1?

No, that's entirely my bad. We should allow an arbitrary folder!

Should I just remove the `choices` argument and allow any string there?

Yes, I would go for that!

Actually, if you have push access to the repo, do you want to just make that one-line change? I'd have to fork and submit a PR, which seems a bit unnecessary.

Indeed, I'll take care of it. I'm on the repo now anyway.

is this change applicable and working with run_lm_finetuning.py in 2.x?
Distributed device ordinal question
Author: Zhaofeng-Wu
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

In the following line, we set the device ordinal to be local rank. However, suppose we have four independent nodes each with only one GPU. Then the 4th node (rank 3) will execute this line with the device ordinal 3, but it really only has 1 GPU, so it'll be invalid to ask for the GPU 3. So won't that break things? Shouldn't it be `torch.device("cuda", 0)`?

https://github.com/huggingface/pytorch-transformers/blob/0287d264e913e10018a95a2723115dc9121e5fc6/examples/run_glue.py#L403

Comments:
No, `local_rank` is only the local rank on each node.

Ah I see, thanks! A tangential question, considering the distributed setting only, would it be the same if we simply call `.cuda()` for the model and tensors instead of passing around the device?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Learning word-pieces garble the predictions
Author: chikubee
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The tasks I am working on is:
* [ x] my own task or dataset: Sentiment Analysis on chatbot conversations

Word-pieces change the prediction completely, I fail to understand why.
If i do a token to token similarity also, by considering the token vector to be the average of wordpieces or the first token vector to see if they carry the same context, they don't.
If I had lemmatized the training set, these pieces would have not learnt anything really.
But that is not intuitive.

<img width="571" alt="Screenshot 2019-09-05 at 12 01 26 PM" src="https://user-images.githubusercontent.com/25073753/64317678-07a95680-cfd6-11e9-8849-f9c11f8531ae.png">
<img width="574" alt="Screenshot 2019-09-05 at 11 59 48 AM" src="https://user-images.githubusercontent.com/25073753/64317679-07a95680-cfd6-11e9-900a-96ec5d318b12.png">
<img width="594" alt="Screenshot 2019-09-05 at 11 59 53 AM" src="https://user-images.githubusercontent.com/25073753/64317680-07a95680-cfd6-11e9-9950-2957c284e6e8.png">

[CLS] token vector is fed to the classifier. Changes need to be done internally.
How can I handle such cases?
Any leads would be helpful.
Thanks in advance.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
convert_roberta_checkpoint_to_pytorch.py 514 max position?
Author: rush86999
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I m not sure if this is intentional but why is max position 514. I m assuming the origingal roberta model is 512 like bert or is this incorrect? This is the only place i find this reference as well.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to finetune DistilBERT on custom data?
Author: 008karan
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I want to build classifier by using DistillBERT. I would like to know how to finetune it on the custom dataset and build classifier on it.
Thansk!

Comments:
Hello @008karan,
There is the class `DistilBertForSequenceClassification` for classification tasks. Its used is really similar to `BertForSequenceClassification`: the main difference is that `DistilBertForSequenceClassification` does not need `token_type_ids` as inputs.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Type of model for each GLUE task
Author: ghost
Pull request: no

## ❓ Questions & Help

There are nine GLUE tasks, and I wanted to verify which BERT model type is best suited for each task. Can anyone confirm these matchings? I am not sure what to do for STS-B especially, and am unsure if BertForMultipleChoice is perhaps the correct option for MNLI.


Comments:
All models on GLUE should use BertForSequenceClassification (MNLI is 3 class, STS-B is 1 class).

As specified in the documentation, for `XxxForSequenceClassification` models:

```
If ``config.num_labels == 1`` a regression loss is computed (Mean-Square loss),
If ``config.num_labels > 1`` a classification loss is computed (Cross-Entropy).
```

So you can use `BertForSequenceClassification` for a regression task such as STS-B.
Fine-tuned RoBERTa models on CPU
Author: avostryakov
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Because of using FusedLayerNorm as BertLayerNorm from apex library after finetuning a saved model isn't possible to use on CPU with apex and Cuda installed. What will be the easiest way to run finetuned models on a server without a GPU? I see that the easiest way to just use the python version of BertLayerNorm during training but maybe there is another way.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Extracting First Hidden States
Author: brytjy
Pull request: no

## ❓ Questions & Help

For example, right now in order to extract the first hidden states of DistilBert model,

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertModel.from_pretrained('distilbert-base-uncased', **output_hidden_states=True**)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
**first_hidden_states = outputs[1][0]  # The first hidden-state**

However, I can only extract the first layer hidden states after running through the entire model; all 7 layers (which is unnecessary if I only want the first).

Is there a way I could use the model to only extract the hidden states of the first layer and stop there? I am looking to further optimize the inference time of DistilBert.

Thank you :)



Comments:
Using do_eval from run_glue.py uses the cached result
Author: wahlforss
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Using do_eval from run_glue.py uses the cached result. I want to evaluate my fine-tuned models and I can't find any guide on how to do so. Anybody that can point me in the right direction?

Comments:
If you simply remove the cached file or move it elsewhere, it won't be used by run_glue.py. That's what I have been doing.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Cut off sequences of length greater than max_length= 512 for roberta
Author: rush86999
Pull request: no
Labels: name: wontfix

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
Roberta uses max_length of 512 but text to tokenize is variable length. Is there an option to cut off source text to maximum length during tokenization process?
## Motivation
Most text are not fixed sizes but are still required for end goal of using roberta on new datasets. The roberta model will throw an error when it encounters a text size > 512. Any help is appreciated to allow tokenization and limit the size to maximum length for pretrained encoders. Alternative is to let the pretrained encoder adapt by increasing positional embedding size so an error is not thrown or use average pooling. I m not sure how this would be implemented and allow fine tuning.
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->
I can't seem to figure out how to circumvent the text length problem as new data text length can go as large as 2500 or more but roberta is only 512 max. 
## Additional context

<!-- Add any other context or screenshots about the feature request here. -->

Comments:
Hi! Indeed RoBERTa has a max length of 512. Why don't you slice your text?

I was hoping the tokenizer could take care of it as a functionality? The actual hope is not to throw an error but allow training with it by increasing the positional encoding as a way to allow training on the whole text length. Is this possible?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is pytorch-transformers useful for training from scratch on a custom dataset?
Author: Caselles
Pull request: no
Labels: name: wontfix

Hello, 

I'm looking into the great repo, and I'm wondering if there is a feature that could allow me to train a, let's say, gpt2 model on a custom dataset of sequences. 

Is it already provided in your codebase and features ? Otherwise I'll tinker with code on my own. 

Thanks in advance and again, great job for the repo which is super useful.

Comments:
This depends on the model you're interested in. For GPT2, for example, there's a class called `GPT2LMHeadModel` that you could use for pretraining with minimal modifications. For XLNet, the implementation in this repo is missing some key functionality (the permutation generation function and an analogue of the dataset record generator) which you'd have to implement yourself. For the BERT model in this repo, there appears to be a class explicitly designed for this (`BertForPreTraining`). 

Hi, we don't provide efficient scripts for training from scratch but you can have a look at what Microsoft did for instance: https://azure.microsoft.com/en-us/blog/microsoft-makes-it-easier-to-build-popular-language-representation-model-bert-at-large-scale/

They shared all the recipes they used for training a full-scale Bert based on this library. Kudos to them!

I'd like to see efficient scripts for training from scratch too please. The Azure repo looks interesting, but looks very Azure-specific, and also bert specific. Would be nice to have training scripts within the hugging face repo itself.

(In addition to being able to train standard BERT etc on proprietary data, it would also be nice to be able to easily experiment with training from scratch using variations of the standard BERT etc models, using the existing public datasets).

@hughperkins
I wrote this post when I modified code to run on (custom) IMDB dataset for BERT model: https://medium.com/dsnet/running-pytorch-transformers-on-custom-datasets-717fd9e10fe2
Not sure if this helps you.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> Hi, we don't provide efficient scripts for training from scratch but you can have a look at what Microsoft did for instance: https://azure.microsoft.com/en-us/blog/microsoft-makes-it-easier-to-build-popular-language-representation-model-bert-at-large-scale/
> 
> They shared all the recipes they used for training a full-scale Bert based on this library. Kudos to them!

@thomwolf Indeed this seems very Azure specific and not very helpful. What would be helpful is showing minimal scripts for training transformers, say GPT2, on custom datasets from scratch. Training from scratch is basic requisite functionality for this library to be used in fundamental research as opposed to tweaking / fine-tuning existing results.
Confused by GPT2DoubleHeadsModel example
Author: weiguowilliam
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Your library is really helpful. And I have 2 questions about the example of GPT2DoubleHeadsModel.

1. 
In source code of model, 

> choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]

Here the comment said that "number of choices: 2". So is it a single sample or two samples? What are the two label classes?

2.
What task is the GPT2DOUBLEHeadsModel trained on? I don't find the information from the document, issues or the original paper.

Please help me. Thank you in advance. 

Comments:
For 2: It's next word prediction: https://openai.com/blog/better-language-models/

> For 2: It's next word prediction: https://openai.com/blog/better-language-models/

Thank you for your reply. From my perspective, I believe the next word prediction is what the language model does, which means it is the "unsupervised" part in the first head. 

The evidence is that gpt-2 is based on gpt. Based on figure 1 in gpt paper, the left head is for "text prediction"(next word prediction), and the right head is for "task prediction".

Hi, the GPT2DoubleHeadsModel, as defined in [the documentation](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel), is: "_The GPT2 Model transformer with a language modeling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks. The two heads are two linear layers. The language modeling head has its weights tied to the input embeddings, the classification head takes as input the input of a specified classification token index in the input sequence)._".

You use a double head model when you want to have two separate losses. In this case the two linear layers compute two different losses: one of the heads is a language modeling loss, with a linear layer of size `hidden_size` x `vocab_size` and the other is a classification loss with a linear layer of size `hidden size` x `number_of_choices`.

You can read [this blog post](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313) by @thomwolf which uses GPT2DoubleHeadsModel with a previous version of this library (pytorch-pretrained-BERT).

@LysandreJik 
Hi, thank you for your help!