noamaanMulla-03/transformers-issues huggingface dataset:
Fixing incorrect link in model docstring
Author: Rexhaif
Pull request: yes

The docstring contains a link to Salesforce/CTRL repo, while the model itself is Facebookresearch/mmbt. It may be the wrong copy\paste.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2260?src=pr&el=h1) Report
> Merging [#2260](https://codecov.io/gh/huggingface/transformers/pull/2260?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/645713e2cb8307e41febb2b7c9f6036f6645efce?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2260/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2260?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2260   +/-   ##
=======================================
  Coverage   78.35%   78.35%           
=======================================
  Files         133      133           
  Lines       19878    19878           
=======================================
  Hits        15576    15576           
  Misses       4302     4302
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2260?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_mmbt.py](https://codecov.io/gh/huggingface/transformers/pull/2260/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX21tYnQucHk=) | `18.25% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2260?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2260?src=pr&el=footer). Last update [645713e...b668a74](https://codecov.io/gh/huggingface/transformers/pull/2260?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks
Is it possible fine tune XLNet?
Author: tyu0912
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello, is there currently a script or recommended process to fine tune XLNet? Currently I am writing my own and using https://mccormickml.com/2019/09/19/XLNet-fine-tuning/ as an inspiration but overall I am still pretty new at this. I managed to get it to run but it seems to be taking forever so just thought I'd check. 

Thanks! Any help would be great.

<!-- A clear and concise description of the question. -->


Comments:
I have tried to fine-tune xlnet on squad1.0 and squad2.0, although it's a few points lower than original results, but it works fine. This #1803 is for fine-tuning on squad2.0, I'm not familiar with other datasets.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using multiple inputs for GPT-2
Author: giterdun345
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
A current project I am working on, to utilize a tuned approach in text generation, is creating a cover letter based off of a job description and some personal qualifications. From researching, I found the convAI blog that you used and thought to apply the concept with OpenAIGPTDoubleHeadsModel. I added special tokens to the vocabulary for delimiters and segment indicators to include word, position and segment embeddings. I have the persona ('some personal qualifications'), turned history into job description and the reply was "I am qualified for this position because...". From my understanding, the attention will help relate the job description and persona to the text generation.  Now I am stuck when it comes to fine tuning the model. I have a bunch of cover letters to use scraped from the web. I have fine tuned a base model and it outputs some coherent cover letters but I want to incorporate the skills needed in the job description for a more accurate an specific cover letter. Am I on the right track and if so how do I fine tune the model on the cover letters? Any suggestions or ideas are welcomed.   
<!-- A clear and concise description of the question. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to measure similarity of words?
Author: RakshaAg
Pull request: no
Labels: name: wontfix

I want to know it BERT based contextual embedding can be used to measure similarity of identical words in different contexts.
And, can I estimate a threshold for that.


Comments:
You should check out BLEU and ROUGE scores. These are often referred to as the precision and recall of NLP.

You may be able to craft a pseudo-f1 score out of these as well 
`f1 = 2*(bleu*rouge)/(bleu + rouge)`

I haven't tried this myself yet, but hopefully this helps!

[https://en.wikipedia.org/wiki/BLEU](https://en.wikipedia.org/wiki/BLEU)
[https://en.wikipedia.org/wiki/ROUGE_(metric)](https://en.wikipedia.org/wiki/ROUGE_(metric))

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
https://s3.amazonaws.com/models.huggingface.co/xxx/pytorch_model.bin failed or can not open at xxx/.cache/xxxxxxxxxx
Author: 578123043
Pull request: no

https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base/pytorch_model.bin
Could you access on https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base/pytorch_model.bin?


Comments:
The correct URL is `https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-pytorch_model.bin`

Did you see the url above somewhere?

![image](https://user-images.githubusercontent.com/16147509/72700543-21af9400-3b87-11ea-9179-97afb05bb486.png)
To resoleved this , adding `config_class pretrained_model_archive_map base_model_prefix` at Class definition (not def __init__) as the image
Regarding Fine-Tuning for Abstractive Summarization
Author: tahmedge
Pull request: no
Labels: name: wontfix

Hi,

What result did you obtain with your Abstractive Summarization code for CNN-DM? Also, is it implemented based on this paper: https://arxiv.org/abs/1908.08345 ?

It will be great if you provide some details regarding it.  

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix glue processor failing on tf datasets
Author: neonbjb
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2564?src=pr&el=h1) Report
> Merging [#2564](https://codecov.io/gh/huggingface/transformers/pull/2564?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6d5049a24d5906ece3fd9b68fb3abe1a0b6bb049?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2564/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2564?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2564      +/-   ##
==========================================
- Coverage    74.6%   74.58%   -0.02%     
==========================================
  Files          87       87              
  Lines       14802    14805       +3     
==========================================
  Hits        11043    11043              
- Misses       3759     3762       +3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2564?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/processors/glue.py](https://codecov.io/gh/huggingface/transformers/pull/2564/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvZ2x1ZS5weQ==) | `27.53% <0%> (-0.34%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2564?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2564?src=pr&el=footer). Last update [6d5049a...17f172e](https://codecov.io/gh/huggingface/transformers/pull/2564?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


A proper fix for this would probably be to add a unit test that sends tf datasets through GLUE. Let me know if you want me to add that in as well.

That's great, thanks @neonbjb
Worse performance of gpt2 than gpt
Author: Nealcly
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I just want to compare the performance of gpt and gpt2 as Language Model to assign Language modeling score. Like #473 , I implement my model as follows:
```
def gpt_score(text, model, tokenizer):
    input_ids = torch.tensor(tokenizer.encode(text)).unsqueeze(0)  # Batch size 1
    input_ids = input_ids.to('cuda')
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
    loss, logits = outputs[:2]
    sentence_prob = loss.item()
    return sentence_prob
a=['there is a book on the desk',
                'there is a rocket on the desk',
                        'he put an elephant into the fridge', 'he put an apple into the fridge']

tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')
model.to('cuda')
model.eval()

print([gpt_score(i,model,tokenizer) for i in a])

#config = GPT2Config.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.to('cuda')
model.eval()
print([gpt_score(i,model,tokenizer) for i in a])
```
And I get the following result:
```
[3.0594890117645264, 4.373698711395264, 5.336375713348389, 4.865700721740723]
[4.475168704986572, 4.266316890716553, 5.423445224761963, 4.562324523925781]
```
It seems that GPT get more sensible result than GPT2, but since gpt2 is literally gpt training with more data, how's that possible?


Comments:
@Nealcly Could you try to use the `gpt2-medium` model? It has more layers :)

Also I'm not sure that your testing procedure is statistically representative :)

Any new updates on this issue? I am also facing the same question. 

Based on the paper, only the largest model is called gpt2. The smallest model(117m) doesn't guarantee better performance than gpt.

how to finetune the language model with dataset and then get perplexity scores


@anonymous297 please check the [documentation examples](https://huggingface.co/transformers/examples.html#language-model-fine-tuning), in which there's exactly what you're looking for.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
edited a way to get  at AlbertAttention.forward
Author: jinkilee
Pull request: yes

'''
# Should find a better way to do this
w = (
    self.dense.weight.t()
    .view(self.num_attention_heads, self.attention_head_size, self.hidden_size)
    .to(context_layer.dtype)
)
b = self.dense.bias.to(context_layer.dtype)
'''

I thought the above code is not necessary.
it can be simply fixed by "merging" `context_layer` at forward().
I committed my "merging" function at AlbertAttention

'''
def merge_tensor(self, x):
    s = x.size()[-2]
    return torch.cat([x[:,:,i,:] for i in range(s)], dim=-1)
'''

I wanted to make a test by "make test' as described in CONTRIBUTING.md
but I couldn't because i faced some make error.

This is my first open source contribution. If i forgot something, please let me know, so I can fix and follow up.

Comments:
will commit with another pull request
BPE error when fine-tuning a CTRL model
Author: orenmelamud
Pull request: no
Labels: name: wontfix

Hi,

cc @keskarnitish

I'm using a slightly modified version of the exapmles/run_lm_finetuning.py code to fine-tune a CTRL model and getting this BPE warning: 

```
WARNING - transformers.tokenization_ctrl -   Saving vocabulary to /home/ubuntu/data/ctrl/pytrained4/merges.txt: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!
```

Is this something I should be concerned about?


Comments:
Can you post the code for the `run_lm_finetuning`? I am not as familiar with what this error message might entail. 

Sure. Here it is.

[run_ctrl_finetuning.py.zip](https://github.com/huggingface/transformers/files/3852748/run_ctrl_finetuning.py.zip)


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[closed] cls token in XLM
Author: JunjieHu
Pull request: no

## 🐛 Bug

The CLS token in XLM should be \<s\> rather than \</s\> in the current repo.
Here is the XLM's original BOS_WORD:
https://github.com/facebookresearch/XLM/blob/master/src/data/dictionary.py#L17

In the transformers' repo, cls_token is set to \</s\>.
https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_xlm.py#L562

And the cls token is used as the BOS token. This is not the same as the original one.
https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_xlm.py#L824


Model I am using (Bert, XLNet....): XLM


Comments:
I find that the first token in the original XLM is indeed using \</s\> rather than \<s\>.
Why isn't BERT doing wordpiece tokenization?
Author: jianwolf
Pull request: no

My code is
```
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = '''why isn't my card working'''
encoded = tokenizer.encode(text, add_special_tokens=False)
text_tokenized = tokenizer.decode(encoded, clean_up_tokenization_spaces=False)
print(text_tokenized)
```
and the output result (the tokenization) is
```
why isn ' t my card working
```

But this isn't the wordpiece nokenization BERT should be using. E.g., `working` should be tokenized as `work ##ing`. Is there anything wrong with my code? And will the fact we are not using wordpiece tokenization decrease BERT's performance?


Comments:
Even if I do `add_special_tokens=True` when encoding, I get

```
[CLS] why isn ' t my card working [SEP]
```

which is still not wordpiece tokenization.

When using `encode` and `decode` you're performing the full tokenization steps each time:

encode: tokenizing -> convert tokens to ids
decode: convert tokens to ids -> detokenizing
 
If you want to see the middle step, you can use the `tokenize` method:

```py
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = '''why isn't my card working'''
print(tokenizer.tokenize(text))  # ['why', 'isn', "'", 't', 'my', 'card', 'working']
```

All the words are in the vocabulary, but if you use more complex words:

```py
tokenizer.tokenize("Why isn't my text tokenizing")  # ['why', 'isn', "'", 't', 'my', 'text', 'token', '##izing']
```

You'll see the `##ing` you were looking for.



Looks great! Thank you!
Finetuning ALBERT using examples/run_lm_finetuning.py
Author: jianwolf
Pull request: no

## 🚀 Feature

The current run_lm_finetuning.py script seems to not have ALBERT added. We should be able to finetune ALBERT in the same way we do to other models in your library.


Comments:
You're right, ALBERT should work out of the box with the fine tuning script as addressed at #2008 by @thomwolf.  It's not too tough to fine-tune ALBERT with the script as reference, and there should also be a PR to add ALBERT and some other language models sometime in the near future

Thank you!
what's the structure of the model saved after fine-tuning ?
Author: JiangYanting
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
 Hello ! I'm wondering what is the structure of the model saved after fine-tuning. For example, after the sequence classification fine-tuning , how to show the layers information of newly-formed model ? Is new model's sentence vector different from the original one which is extracted from pre-trained model ? 

Comments:
I'm afraid I don't understand your question. The pretrained model is an architecture whose weights have already been trained on some task (typically (M)LM and NSP/SOP). When you finetune the model, the architecture stays exactly the same but the weights are finetuned to best fit your task.

@BramVanroy I tried “print(model)”，it showed the information of every layer. Thank you so much!
What is the f1 score of Squad v2.0 on bert-base? I only got f1 score 74.78.
Author: YJYJLee
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello, I am doing some experiment of squad v2.0 on bert-base (NOT bert-large).
According to the BERT paper, bert-large achieves f1 score 81.9 with squad v2.0.
Since I couldn't find the official result for bert-base, I am not sure if I am getting the right f1 score.
Has anyone tried running squad v2.0 on bert base? 

I got f1 score **74.78** for squad v2.0 result on bert-base, using below command:
sudo python3 ../../../run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --train_file $SQUAD2_DIR/train-v2.0.json \
  --predict_file $SQUAD2_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 4 \
  --learning_rate 4e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --version_2_with_negative \
  --overwrite_output_dir \
  --output_dir ../../../bert_base/$TASK_NAME/

Comments:
Your result is fine. This [poster](https://web.stanford.edu/class/cs224n/posters/15848021.pdf) says that they achieved **76.70**. Maybe you can get there as well when you train for 2 more epochs.

Thank you for your reply! :)

Please close the question if the answer suits your needs.

Sorry, I closed it!
[run_lm_finetuning] Train from scratch
Author: julien-c
Pull request: yes

Ability to train a model from scratch, rather than finetune a pretrained one.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2570?src=pr&el=h1) Report
> Merging [#2570](https://codecov.io/gh/huggingface/transformers/pull/2570?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/65a89a89768f5922e51cdc7d49990d731e3f2c03?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2570/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2570?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2570   +/-   ##
=======================================
  Coverage   74.61%   74.61%           
=======================================
  Files          87       87           
  Lines       14802    14802           
=======================================
  Hits        11044    11044           
  Misses       3758     3758
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2570?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2570?src=pr&el=footer). Last update [65a89a8...55939b5](https://codecov.io/gh/huggingface/transformers/pull/2570?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks good to me, feel free to merge it when you feel ready @julien-c 

Yo! merge this shit up!!  

Thanks, A Lot Guys your the best!
Recommended way for creating distillBERT container and serving 
Author: ishwara-bhat
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
As per documentation, I am supposed to load distilbert as below.

question_answering_model = torch.hub.load('huggingface/pytorch-transformers', 'modelForQuestionAnswering', 'distilbert-base-uncased-distilled-squad')
question_answering_tokenizer = torch.hub.load('huggingface/pytorch-transformers', 'tokenizer', 'distilbert-base-uncased-distilled-squad')

I'm using google Cloud Run. It bring up the container (and hence the model) only upon request. This causes download and load delay.

How to pre-download the model and serve? I am looking for dockerfile step where it could install the weights file and other files needed for the model. This way, I am hoping that my dynamic delays are reduced and I get inference must faster.

Please let me know if such a thing is possible.

thanks
Ishwar





Comments:
You can find the s3 URL of models here for distilbert: https://github.com/huggingface/transformers/blob/master/transformers/configuration_distilbert.py

If you build the docker on your machine, first download model files on your machine.
Then just add those files to your container through Dockerfile.

If you want your Docker build to download from s3, you can install `aws-cli` in Dockerfile and run `aws s3 cli`. But it will make it slower.

Naturally a model in your docker will make it a bit fatter.

I think the path has changed slightly. I found the file in "src" folder under master.  https://github.com/huggingface/transformers/blob/master/src/transformers/configuration_distilbert.py 

About downloading model files, in configuration_distilbert.py, I only found https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-distilled-squad-config.json file path. It just gives config. It is not a weights file / pickle file. Please suggest the path of files which I can download and make part of the local folder. Thanks.

Links to pre-trained models are available in the beginning of each `modeling_xxx.py` file, e.g. for [BERT](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L34-L56). Put this with a configuration in your folder and you can load them locally.

You could also use the `save_pretrained` method to automatically create a folder that can be used with `from_pretrained`.
Fine Tuning GPT2 on wikitext-103-raw
Author: snaik2016
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Running pytorch-transformers\examples\run_lm_finetuning.py. This is stuck at  load_and_cache_examples step. I just see message like
  WARNING - pytorch_transformers.tokenization_utils -   This tokenizer does not make use of special tokens. The sequence has been returned with no modification.
The train file has 1.8M rows with this rate it would take few days to just tokenize and cache training data.
Is this expected?  Did anyone face this before?

Thanks in advance for your help.

Comments:
@snaik2016 I ran into the same issue and had to parallelize my code to make it faster. Also getting rid of the while loop and list splicing in the TextDataset class with a for loop made it much quicker.

Please check #1830 . I made some tuning on a training part. But I guess it'll still take many days for 1.8M rows dataset (in fact, talking about the token count rather than rows count is more meaningful) .

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.