noamaanMulla-03/transformers-issues huggingface dataset:
Is HuggingFace TransfoXLLMHeadModels trainable from scratch?
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

Is HuggingFace TransfoXLLMHeadModels trainable from scratch? The documentation makes it look like it is possible to train the TransfoXLLMHeadModel from scratch, since (according to the documentation) the loss can be returned via TransfoXLLMHeadModel( ) as long as labels are provided (https://huggingface.co/transformers/model_doc/transformerxl.html#transfoxllmheadmodel) .

However, the code for TransfoXLLMHeadModels shown in the Github repository 
(https://github.com/huggingface/transformers/blob/master/transformers/modeling_transfo_xl.py#L780) seem to suggest that the loss is, in fact, not returned even when the labels are provided.

Is HuggingFace TransfoXLLMHeadModels trainable from scratch?

Thank you,


Comments:
Hi @h56cho,

The loss is actually returned if labels are present.

Check https://github.com/huggingface/transformers/blob/master/transformers/modeling_transfo_xl.py#L793

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Regression Loss
Author: jasonmusespresso
Pull request: no
Labels: name: wontfix

I want `BertForSequenceClassification` to compute regression loss. I load the pre-trained model through
```
config = BertConfig.from_pretrained('bert-base-uncased')
config.num_labels = 1  # Is it proper to set the num_label in this way?
model = BertForSequenceClassification(config)
```

How shall I change the num_labels properly?

Thank you!

Comments:
That is a good way to load the labels! 

Please be aware, however, that by loading the configuration this way you will not load the weights for your model associated with the `bert-base-uncased` checkpoint as you would have done had you used the `BertForSequenceClassification.from_pretrained(...)` method.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
The id of the word obtained by tokenizer.encode does not correspond to the id of the word in vocab.txt
Author: coder-duibai
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
hello，I encountered a problem，code show as below.
tokenizer=BertTokenizer.from_pretrained('bert-base-uncased')
input_ids=torch.LongTensor(tokenizer.encode("[CLS] the 2018 boss nationals [SEP]"))
print(input_ids)
The result is:tensor([  101,  1996,  2760,  5795, 10342,   102]) , but In the voab.txt, the id of "the" word is 8174, not 1996. Why is this the reason for the berttokenizer token dictionary error?

Comments:
Hi, in the `bert-base-uncased` vocab.txt file, the id of the token "the" is 1996. You can see it [inside the file, where "the" is on the row 1996](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt). You can also check it out directly in the tokenizer with the following command: 

```py
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
tokenizer.vocab.get("the")  # 1996
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
input token embedding issues
Author: abrhaleitela
Pull request: no
Labels: name: wontfix

1. During fine-tuning an XLNet, does the word embeddings kept constant (learned from the pre-trained model) or are they initialized randomly but learned contextually during fine-tuning itself? 

2. And how would I set my own word embeddings during fine-tuning (XLNetForSequenceClassification)? I have my own token embeddings that I want to feed for my XLNetForSequenceClassification and is it possible to keep this embeddings constant during fine-tuning? 
get_input_embeddings and set_input_embeddings functions does not seem to work with XLNetForSequenceClassification Class. 

Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
porting XLM-Roberta to tensorflow 2.0
Author: andompesta
Pull request: no

## ❓ Questions & Help

Yesterday I have ported XLM-Roberta from pytorch to tensorflow mainly following the instruction provided in [huggingface/from-tensorflow-to-pytorch](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28).
The final error is computed using the DUMMY_INPUT as input values for the large MaskedLM model and is evaluated on the final prediction_score output.  
I compute the error as:
```python
    max_absolute_diff = np.amax(np.abs(tf_model_out.numpy() - pt_model_out.detach().numpy()))
```
and the final output is 0.00027179718; which is lower than the suggested 1e^-3 bound.

According to your indication the error seems to be acceptable, given that XLM-Roberta is a huge model. However, I have experienced some huge output difference when I do not specify the position_ids. That is, the position_ids computed by the TFRobertaEmbeddings seems to be correct since they correctly take in consideration the presence of some pad tokens using the ``create_position_ids_from_input_ids`` function. Instead the PyTorch RobertaEmbeddings doesn't.

Moreover, I'm also wandering if it is possible to merge the interface of the TF models with the PyTorch models. Not sure if it is worth it, but by using the __call__ and call function provided by TF2.0 it is possible to obtain an equivalent interface between the 2 frameworks.
For example:
```python
class TFXLMRobertaForMaskedLM(TFXLMRobertaPreTrainedModel):
    def __call__(self, input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, **kwargs):
        inputs = (input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds)
        return super(TFXLMRobertaForMaskedLM, self).__call__(inputs, **kwargs)

    def call(self, inputs, **kwargs):
        outputs = self.xlm_roberta(*inputs, **kwargs)

        sequence_output = outputs[0]
        prediction_scores = self.lm_head(sequence_output)
        outputs = (prediction_scores,) + outputs[2:]  
        return outputs  # prediction_scores, (hidden_states), (attentions)
```
should be equivalent to the PyTorch implementation

Comments:
Hi @andompesta hard to say without looking at the code – did you check out this related PR by @jplu : #2443
GPT-2 finetuning with run_lm_finetuning.py script
Author: dkajtoch
Pull request: no

## ❓ Questions & Help
I tried to finetune gpt-2 model using `run_lm_finetuning.py` script with the following parameters:
```
python run_lm_finetuning.py \
    --train_data_file=text8.train \
    --output_dir=/content/gpt2 \
    --eval_data_file=text8.val \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --do_eval \
    --per_gpu_train_batch_size=32 \
    --per_gpu_eval_batch_size=32 \
    --gradient_accumulation_steps=1 \
    --num_train_epochs=3 \
    --warmup_steps=200 
``` 
and it throws memory error no matter what my machine type is. In extreme case I wanted to run this demo on Google Cloud with 32 cpus and 120GB RAM - not possible. It just eats the whole RAM and does not even make a single iteration. On the other hand, I was able to do finetuning from this project [gpt-2-simple](https://github.com/minimaxir/gpt-2-simple) on Google Colab with 124M model (CPU). What is going on? Am I doing something wrong?



Comments:
To me, you have set an extreme `batch_size`. Did you try it with e.g. `per_gpu_train_batch_size=1` and `per_gpu_eval_batch_size=1` ?

@iedmrc I finally managed to fine-tune it with `per_gpu_train_batch_size=1` and `gradient_accumulation_steps=32`. Indeed the batch size was the problem but I haven't realized it is such a big problem. Everything works fine now.
the BertModel have the class "BertForTokenClassification", why XLNetModel don't have the class
Author: zyxdSTU
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->


Comments:
By reading the code, I find the BertModel have the class "BertForTokenClassification", but XLNetModel
don't have the class "XLNetForTokenClassification". Why? Can someone explain this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update pipelines.py
Author: Perseus14
Pull request: yes
Labels: name: Core: Pipeline

Modified QA pipeline to consider all features for each example before generating topk answers. 
Current pipeline only takes one SquadExample, one SquadFeature, one start logit list, one end logit list to retrieve the answer, this is not correct as one SquadExample can produce multiple SquadFeatures.

Comments:
Hi @Perseus14, thanks for your contribution :).

I took the liberty to apply black formatting so that tests are happy.

Looks good to me 👍 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2459?src=pr&el=h1) Report
> Merging [#2459](https://codecov.io/gh/huggingface/transformers/pull/2459?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/16ce15ed4bd0865d24a94aa839a44cf0f400ef50?src=pr&el=desc) will **increase** coverage by `0.14%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2459/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2459?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2459      +/-   ##
==========================================
+ Coverage   73.24%   73.39%   +0.14%     
==========================================
  Files          87       87              
  Lines       15001    15005       +4     
==========================================
+ Hits        10988    11013      +25     
+ Misses       4013     3992      -21
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2459?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/2459/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `69.03% <100%> (+0.35%)` | :arrow_up: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2459/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `88% <0%> (+0.16%)` | :arrow_up: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/2459/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `35.97% <0%> (+6.6%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2459?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2459?src=pr&el=footer). Last update [16ce15e...0d6c17f](https://codecov.io/gh/huggingface/transformers/pull/2459?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok great, thanks @Perseus14 @mfuntowicz!
Fixed answer structure for QAPipeline
Author: Perseus14
Pull request: yes
Labels: name: Core: Pipeline

Updated answers list in QuestionAnswering pipeline to handle multiple (question, context) pair with (top-k >1) solutions

Comments:
Hi can you confirm this PR is a subset of #2459 and that we can close it now that #2459 is merged?

Sure, thanks!!
________________________________
From: Thomas Wolf <notifications@github.com>
Sent: Monday, January 13, 2020 8:33:47 PM
To: huggingface/transformers <transformers@noreply.github.com>
Cc: Rishabh Manoj (IMT2013035) <Rishabh.Manoj@iiitb.org>; Author <author@noreply.github.com>
Subject: Re: [huggingface/transformers] Fixed answer structure for QAPipeline (#2416)


Hi can you confirm this PR is a subset of #2459<https://github.com/huggingface/transformers/pull/2459> and that we can close it now that #2459<https://github.com/huggingface/transformers/pull/2459> is merged?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/pull/2416?email_source=notifications&email_token=ACAOU5V7GHKDDYFHRQHXQPLQ5R7FHA5CNFSM4KDEIJ72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIZBBAI#issuecomment-573706369>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACAOU5UZ5LYGXF2PHGZSCCLQ5R7FHANCNFSM4KDEIJ7Q>.
Perplexity for (not-stateful) Transformer - Why is it still fair to compare to RNN?
Author: hoangcuong2011
Pull request: no
Labels: name: wontfix

Greetings,

It is clear to me how to compute perplexity for RNNs as RNNs are a stateful model. 

In general, given a very long document, I believe we need to: 1. chunk the document into a sequence of chunks, and we compute the cross entropy loss for each of the chunks, before taking the average for all chunks and take exponential. 

Because RNNs are a stateful model, we can use what we have from the last RNN hidden state regarding to previous chunk to initialize hidden state for RNN that handles the next chunk. Because of this, I think RNN computes the "right" perplexity for a long document.

Meanwhile Transformer are not a stateful model. For this reason there is no such things as "transferring" hidden state from previous chunk to next chunk I guess. For this reason I think Transformers are not really computing the perplexity for a long document. It compute something different.

So why is it still fair to compare two models regarding to perplexity for long documents?
Plus, if I create a non-stateful RNN (e.g. as in Keras I can set stateful=False), will the perplexity the non-stateful model computes without "transferring" hidden state be the right perplexity we need? Is it comparable to the stateful RNN?

Thank you!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Import issues in run_squad_w_distillation
Author: graviraja
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using : DistilBert

Language I am using the model on : English

The problem arise when using:
* [x] the official example scripts: run_squad_w_distillation.py in examples/distillation


The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQuAD


## To Reproduce

Steps to reproduce the behavior:

1. python run_squad_w_distillation.py --help 


<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
It should show the input arguments required for the code to run

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: CentOS Linux
* Python version: 3.6.9
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU: yes
* Distributed or parallel setup: None
* Any other relevant information:

## Additional context
```python
Traceback (most recent call last):
  File "run_squad_w_distillation.py", line 51, in <module>
    from ..utils_squad import (
ValueError: attempted relative import beyond top-level package
```

There is no utils_squad or utils_squad_evaluate files present in the repo but are imported in run_squad_w_distillation.py file. How to solve this? 

Is there any release planned for distill version of squad2.0 like the one released on squad 1.1? 
<!-- Add any other context about the problem here. -->


Comments:
Option to upload a trained model from gpt-2-simple to use with Write With Transformer
Author: torakoneko
Pull request: no
Labels: name: wontfix

name: Write With Transformer

## 🚀 Feature

I would like to be able to upload model checkpoints created in GPT-2-simple to use with Write with Transformer.

## Motivation

It would be really fun and allow people to use their own checkpoints without having to get them approved or anything or make them public.

## Additional context

none

Comments:
That's on the long term horizon, but that'd be a cool feature, indeed. We are working on a way to let users of `🤗/transformers` upload their weights to share them with the community super easily.

Once we ship this, it would be doable to also host some of those on Write With Transformer. (with some *interesting* challenges on how to scale our infra to host lots of concurrent models, cc @LysandreJik  :)

I'm training GPT-2 on all the Harry Potter books and I'd really love to play with it in Write With Transformer if you guys wanted to put it up, lol. (I know it's a really really small dataset but it's just for fun)

@torakoneko No ETA yet, but things are progressing on the aforementioned roadmap.

@julien-c Would it be possible to run it locally or on something like RunwayML with one's own checkpoint?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BERT: Uncased vocabulary has 30,552 tokens whereas cased has 28,996 tokens. Why this difference?
Author: culcomp
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I was expecting cased vocabulary to be larger than uncased as it needs to include both tokens with capitalizations and without. But, to the contrary, I found the cased vocabulary to be smaller. May I know reason for this?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
pytorch_pretrained_bert giving different scores for BertForNextSentencePrediction 
Author: LiZhengArsenal
Pull request: no

## ❓ Questions & Help
from pytorch_transformers.modeling_bert import BertForNextSentencePrediction
from pytorch_transformers import BertTokenizer, BertConfig
import torch

#Load pretrained model from local
config = BertConfig.from_json_file('resources/bert_config.json')
token = BertTokenizer('resources/vocab.txt')
model = BertForNextSentencePrediction.from_pretrained('resources/pytorch_model.bin', config=config)
model.eval()
textA_ids = token.tokenize("How old are you?")
textB_ids = token.tokenize("The Eiffel Tower is in Paris")
text_ids = token.convert_tokens_to_ids(["[CLS]"] + textA_ids + ["[SEP]"] + textB_ids + ["[SEP]"])
segments_ids = [0]\*(len(textA_ids)+2) + [1]\*(len(textB_ids)+1)
text_inputs = torch.tensor([text_ids])
segments_inputs = torch.tensor([segments_ids])
with torch.no_grad():
    outputs = model(text_inputs, token_type_ids=segments_inputs)
print(outputs)

Outputs changed everytime when I ran the code. I followed many ways in other issues to solve the problem, but they didin't work.  And I've used local pretrained model for many other tasks, but never happened this thing before. So I don't think model caused this problem.

Version Information:
torch 1.1.0
pytorch-transformers 1.2.0 




Comments:
Error when running run_generation.py
Author: ailoverz
Pull request: no

I tried to run this code:
python ./examples/run_generation.py --model_type=gpt2 --length=20 --model_name_or_path=gpt2

However I am getting the error below:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\GPT2\\.cache\\torch\\transformers\\tmpy2recb0u' -> 'C:\\Users\\GPT2\\.cache\\torch\\transformers\\f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./examples/run_generation.py", line 237, in <module>
    main()
  File "./examples/run_generation.py", line 200, in main
    tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path)
  File "C:\gpt2\venv\lib\site-packages\transformers\tokenization_utils.py", line 309, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "C:\gpt2\venv\lib\site-packages\transformers\tokenization_utils.py", line 415, in _from_pretrained
    raise EnvironmentError(msg)
OSError: Couldn't reach server at '{}' to download vocabulary files.


How do I get over this hump? Thanks



Comments:
It seems to me that there is either another program that has a lock on the GPT-2 file or that you can't access our S3. Does the error still happen if you restart your machine?

yes. I restarted several times but the issue persist

Seems to be a file lock issue. Can't rename a file because it's being used.
See here:
https://github.com/huggingface/transformers/blob/f599623a99b808e3d5926d89cd13237457b9eeba/src/transformers/file_utils.py#L392
Related #2385

Ok this should be solved on master now that #2384 is merged
The method os.rename() in file_utils.py  make a permissionError
Author: heroazhe
Pull request: no

## ❓ Questions & Help
Is someone happen to this question?
<!-- A clear and concise description of the question. -->

![image](https://user-images.githubusercontent.com/22883367/71649688-af981c00-2d4b-11ea-811b-7fd9ebef4776.png)


Comments:
I have the same problem when downloading the pre-trained tokenizer. I also need help!

> I have the same problem when downloading the pre-trained tokenizer. I also need help!

Online download often occur different problems,so i download it first and use it locally.

Ok this should be solved on master now that #2384 is merged
update probabilitiy to probability, misspelled the word
Author: 7will10
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2507?src=pr&el=h1) Report
> Merging [#2507](https://codecov.io/gh/huggingface/transformers/pull/2507?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a3085020ed0d81d4903c50967687192e3101e770?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2507/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2507?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2507   +/-   ##
=======================================
  Coverage   73.24%   73.24%           
=======================================
  Files          87       87           
  Lines       15006    15006           
=======================================
  Hits        10991    10991           
  Misses       4015     4015
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2507?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2507/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2JlcnQucHk=) | `100% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2507?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2507?src=pr&el=footer). Last update [a308502...8eb4e0c](https://codecov.io/gh/huggingface/transformers/pull/2507?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, thanks for your PR but it has been superseded by #2492 !