noamaanMulla-03/transformers-issues huggingface dataset:
Merge run_squad.py and run_squad2.py
Author: Liangtaiwan
Pull request: yes

Merge run_squad.py and run_squad2.py to a single file as [official's Bert](https://github.com/google-research/bert).
I did integration test by myself and run once.

The following is the scores with base-uncased model

| Task  | Exact | F1 |
| ------------- | ------------- |  ------------- |
| SQuAD v1.1 |  80.90823084200568| 88.03529050425266 |
| SQuAD v2.0 | 72.56801145456078|   75.65274647953608 |

Comments:
#152 run squad pull request with Squad 2.0 was originally one file but was asked to be separated into two files for commit. #174

@abeljim Thanks for your replying, but I think that they should merge to a single file due to easier maintain and too many repeat code.

How many epochs used in SQuAD2.0 in your test?

> @abeljim 

> How many epochs used in SQuAD2.0 in your test?
> 
> > @abeljim

Just 2

Ok I didn't notice the original implementation had these scripts merged in a single file.
I guess we merge these scripts together here also.
Do you want to update your branch to resolve the merge conflicts @Liangtaiwan?

Ok merged, thanks @Liangtaiwan 

Hi @Liangtaiwan and @abeljim , should models trained on SQuAD 2 use the `version_2_with_negative` flag when evaluating on SQuAD 1.1 dev sets? I'm noticing more than a 10 point difference with and without the flag. Thanks!

Results with the flag appear to be closer to the performance on the answerable examples in SQuAD 2, but I wanted to confirm.

Hi @samsontmr, that's a good question.

In my opinion, since there is still a plausible answer to the unanswerable questions in SQuAD 2.0, you should use the version ```verions_2_with_negative``` when training.

Thanks! I think you mean when testing?

Hi @samsontmr, sorry for misunderstanding your question.
If you trained the model with ```verions_2_with_negative``` on SQuAD 2.0, the only difference when you evaluate on SQuAD 1.1 is post-processing. 
With ```verions_2_with_negative``` flag when testing, the model would output unanswerable which is not a valid choice in SQuAD 1.1. This might the reason why there is a 10 point difference with or without the flag.
I don't think you need ```verions_2_with_negative``` flag when testing. 
Could you paste the result and the version of transformers you used? 

I see! Unfortunately I didn't save the results, but I got an F1 of 70+ with the flag, which is closer to the squad 2 "hasAns" results, and 80+ without the flag.
How to convert a tf2 pre-trained model to pytorch model?
Author: tomohideshibata
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I have trained a pre-trained model from scratch using a tensorflow 2.0 official script (run_pretraining.py).
https://github.com/tensorflow/models/tree/master/official/nlp/bert

My question is how to convert the pre-trained model to pytorch model?
Thanks in advance.

Comments:
Have you ever tried [convert_bert_original_tf_checkpoint_to_pytorch.py](https://github.com/huggingface/transformers/blob/1ab8dc44b3d84ed1894f5b6a6fab58fb39298fc7/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py)?

> ##  Questions & Help
> I have trained a pre-trained model from scratch using a tensorflow 2.0 official script (run_pretraining.py).
> https://github.com/tensorflow/models/tree/master/official/nlp/bert
> 
> My question is how to convert the pre-trained model to pytorch model?
> Thanks in advance.

Thanks for your comment.

This script is for tensorflow 1.0.
https://github.com/google-research/bert

The weight names are different between tf 1.0 and 2.0, and this script does not work for a tf2 pre-trained model.

If you have enough time, you can implement it, open a PR and share your source code with us

> Thanks for your comment.
> 
> This script is for tensorflow 1.0.
> https://github.com/google-research/bert
> 
> The weight names are different between tf 1.0 and 2.0, and this script does not work for a tf2 pre-trained model.

OK, I will try it.

As a tentative workaround I will ask how to convert a tf2 pre-trained model to a tf1 model in the official tensorflow BERT project.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@tomohideshibata did you ever succeed in converting your model to pytorch or tf1?

edit: seems it was added already to `transformers` in #5791

No.
Thanks for the information.
Onnx conversion for bert models with classification layers
Author: hrsmanian
Pull request: no
Labels: name: wontfix

Hi,
  I am trying to convert a NER model trained with BertForTokenClassification to Onnx format. Am able to convert it using the convert_graph_to_onnx.py script using the following params

convert(framework="pt", model="bert_small_ner", output="bert_small_onnx/bert_small_ner.onnx", opset=11)

There are no errors thrown either. However, when i run the inference, I see only the Bert layer outputs but not the output of the Linear classification layer as it is in BertForTokenClassification. 

Am I missing something? Or this is a know issue with some work around? Kindly let me know

Thanks in advance

Comments:
Might be of interest to @mfuntowicz 

I think the issue could be due to pipeline which extracts the model
# Allocate tokenizer and model
return pipeline("feature-extraction", model=model, tokenizer=tokenizer, framework=framework)

Seems like default is 'feature-extraction'
I could get this to work by changing it to 'ner'. Also passed in a config option
return pipeline("ner", model=model, config=config, tokenizer=tokenizer, framework=framework)

Not sure if this is the correct solution but it seems to work for me

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Slow Decoding Speed when using BertForLMModel
Author: JamesHujy
Pull request: no

I set the BertLMHeadModel as Decoder in my Seq2Seq model. It seems to work well in training. But when decoing, it decodes very slowly. I think there is no layer_past used in GPT2, XLNet in BertLMHeadModel and many attentions are computed repetitively? 


Comments:
Yeah, exactly - we should / could add the layer cache for `BertLMHeadModel`. It's not trivial to do it, but feel free to give it a try.
Simple train from the start for translation transformer
Author: felipeboffnunes
Pull request: no

Hi, sorry to bother. I am trying to train a translation transformer, I have seen the documentation but I am still really lost.

I have two datasets, the original message and the translated message.

Example:
dataset_x.txt
This is the message.
This is another message.
Another one.

dataset_y.txt
This<&>is the message<^>.
This is another<&> message.
Another one<%>.

I wanted a simple script which could tokenize these datasets and train any suited model from scratch.
Could anyone help me? Thanks a bunch!


Comments:
I found the repo simpletransformers, which uses this marvelous repo of yours. I got to run a transformer through there, so I'll be closing the issue now. Thanks anyway!
Training GPT2 and Reformer from scratch.  
Author: VikasRajashekar
Pull request: no

Hello, I am looking, for example, script/notebook to train GPT2 and Reformer model from scratch in German. 
Something similar to : 
https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb

I am trying to modify the same notebook but GPT2 doesn't seem to accept LinebyLineDataset or padding. 

Comments:
Hey @VikasRajashekar, 

We are trying to move "non-bug" related questions to https://discuss.huggingface.co/ - could you post your question there again? :-) 

Btw, for Reformer, you can check out these notebooks: https://github.com/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb and https://github.com/patrickvonplaten/notebooks/blob/master/Reformer_For_Masked_LM.ipynb
Autotokenizer not returning instance of LongformerTokenizerFast
Author: pratikdk
Pull request: no

## Environment info
     
- `transformers` version: 3.0.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: Google Colab

## Information

Model I am using :  Longformer 
Path: **'allenai/longformer-base-4096'** and **'allenai/longformer-large-4096'**

The problem arises when trying to load 'Fast' version for Longformer using Autotokenizer, the returned tokenizer instance is an object of LongformerTokenizer and not LongformerTokenizerFast. 

![Annotation 2020-08-13 160156](https://user-images.githubusercontent.com/20542313/90124665-68ace300-dd7e-11ea-8cc7-fdd80f070bec.png)

I require the offset mappings for a sub task of extracting word embeddings.

## To reproduce
Just as in the screenshot i am adding the code below to instantiate the tokenizer object:
```
longformer_tokenizer = AutoTokenizer.from_pretrained(
    pretrained_model_name_or_path = 'allenai/longformer-base-4096', use_fast=True)
print(longformer_tokenizer.is_fast)
print(longformer_tokenizer)
```
And since its not an instance of  transformers.LongformerTokenizerFast, I cannot `return_offsets_mapping=True`
As in the below code throws `NotImplementedError`

```
longformer_encoded_dict = longformer_tokenizer.encode_plus(text=sequence_3,
                                                    add_special_tokens = True,
                                                    max_length = 75,
                                                    truncation = True,
                                                    pad_to_max_length = False,
                                                    return_token_type_ids = False,
                                                    return_attention_mask = True,
                                                    return_overflowing_tokens = False,
                                                    return_special_tokens_mask = False,
                                                    return_offsets_mapping=True)
```
**Error**
`
NotImplementedError: return_offsets_mapping is not available when using Python tokenizers.To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast.`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->


@mfuntowicz
@patrickvonplaten


Comments:
Hi @pratikdk  thank you for reporting this! Just made a PR, will be fixed soon. Till then you can use `LongformerTokenizerFast` class
[LongformerTokenizerFast] add LongformerTokenizerFast in AutoTokenizer
Author: patil-suraj
Pull request: yes

This PR adds LongformerTokenizerFast in AutoTokenizer. Fixes #6459

@patrickvonplaten 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6463?src=pr&el=h1) Report
> Merging [#6463](https://codecov.io/gh/huggingface/transformers/pull/6463?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/54c687e97c92efe6eba9e537bd98b47d9005a279&el=desc) will **increase** coverage by `0.18%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6463/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6463?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6463      +/-   ##
==========================================
+ Coverage   79.91%   80.09%   +0.18%     
==========================================
  Files         153      153              
  Lines       28005    28005              
==========================================
+ Hits        22379    22431      +52     
+ Misses       5626     5574      -52     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6463?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6463/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `97.72% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6463/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `24.53% <0.00%> (-63.20%)` | :arrow_down: |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6463/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `88.42% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6463/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.44% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6463/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `96.58% <0.00%> (+27.51%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6463?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6463?src=pr&el=footer). Last update [54c687e...7f1278b](https://codecov.io/gh/huggingface/transformers/pull/6463?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Remove old doc page and add note about cache in installation
Author: sgugger
Pull request: yes

As discussed offline, removing the old page "Loading Google AI or OpenAI pre-trained weights or PyTorch dump" and move the not about cache in the installation folder (also, making that up to date ^^).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5027?src=pr&el=h1) Report
> Merging [#5027](https://codecov.io/gh/huggingface/transformers/pull/5027?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/bbad4c6989d489097f42bbe38001a3f8ca1c5c11&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5027/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5027?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5027   +/-   ##
=======================================
  Coverage   77.19%   77.19%           
=======================================
  Files         128      128           
  Lines       21877    21877           
=======================================
+ Hits        16888    16889    +1     
+ Misses       4989     4988    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5027?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5027/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.57% <0.00%> (+0.15%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5027?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5027?src=pr&el=footer). Last update [bbad4c6...2d3587c](https://codecov.io/gh/huggingface/transformers/pull/5027?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Hashing a tokenizer using the 🤗 nlp lib is not deterministic
Author: lhoestq
Pull request: no

In the `nlp` library it is common to use a tokenizer on a dataset.
The library takes care of caching the results, so that if you run the tokenization twice, it will reuse the previous results.
To make the caching work, we compute a hash of the tokenizer.

However the `unique_no_split_tokens` attribute of tokenizers is not deterministic, and it makes the hashing return different hashes for the same tokenizer over different sessions.

`unique_no_split_tokens` can be a list like `['[CLS]', '[MASK]', '[PAD]', '[SEP]', '[UNK]']` for example. But it happens that re-loading a tokenizer in another session shuffles the tokens in the list.

For example this code doesn't always return the same output over different sessions:

```python
from transformers import AutoTokenizer

model_name = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(model_name)

print(tokenizer.unique_no_split_tokens)
```

Reproduce on google colab: https://colab.research.google.com/drive/1nyskaLavcTCkXibZBlYX71bkG476uSzz?usp=sharing

Comments:
Regarding GPU use for LM
Author: shubhujf
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Hi,
I am running example given in README.md of language_modeling using following command:
export TRAIN_FILE=/path/to/dataset/wiki.train.raw
export TEST_FILE=/path/to/dataset/wiki.test.raw

python run_language_modeling.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE

It has started training but it is not using GPU (TITAN X) at all, when I see throug nvidia-smi command 
I am new to this So Can you please let me know if I'm missing anything here.

Thanks.-->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
 Hi,
I am running example given in README.md of language_modeling using following command:
export TRAIN_FILE=/path/to/dataset/wiki.train.raw
export TEST_FILE=/path/to/dataset/wiki.test.raw

python run_language_modeling.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE

It has started training but it is not using GPU (TITAN X) at all, when I see throug nvidia-smi command 
I am new to this So Can you please let me know if I'm missing anything here.

Thanks.

This is probably because torch doesn't detect that you have a GPU. Can you try launching a python console and running the following?

```py
import torch
print(torch.cuda.is_available())
```

Yeah, I also found out about it later after posting this issue.
I had to install cuda 9.1 and reboot the server then it worked.
Thank You for your reply :)
test_run_glue_with_pabee failing
Author: sshleifer
Pull request: no

examples/bert-loses-patience/test_run_glue_with_pabee.py::PabeeTests::test_run_glue

https://app.circleci.com/pipelines/github/huggingface/transformers/10373/workflows/0c9f2e61-2732-4857-84f0-71b59ddf10a9/jobs/71646


@JetRunner 

Comments:
@sshleifer Thanks for the issue. I noticed that this test fails sometimes. Do you have any idea how long has this problem been? Is it broken at the beginning or recently?

Started breaking within the last few days. It is breaking fairly consistently at this point.

It's I guess just flaky, for example it's not broken on master right now.

But it's not really a flaky "type of error". It's hitting IndexError on an embedding lookup.
```
IndexError: index out of range in self
```


It also fails with the assert: `self.assertGreaterEqual(value, 0.75)` in my case (three times out of four right now).
remove words from vocabulary
Author: vineeth567
Pull request: no
Labels: name: wontfix

Is there any way to remove the words from the vocabulary of pretrained model? and is there a way to see the vocabulary of the model?


Comments:
Please refer to this issue: [https://github.com/huggingface/transformers/issues/4827](https://github.com/huggingface/transformers/issues/4827)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Multi-output regression support for Transformer models
Author: mangalgishreyas
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

I am trying to build a shipping address to geo-code predictor using RoBERTa. Here the shipping address would be the text input and the output would be a gecode (latitude and longitude).
I tried using _robertaforsequenceclassification_ but it mentions that when the final layer consists of more than one class, a cross entropy loss would be used automatically. But I want to perform regression using the RMSE loss. It would be great if we can add the multi-output regression feature to the existing sequence classification pipeline.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->


Comments:
Hello! The cross entropy loss is only used if you provide the `labels` for the model to compute the loss. If you don't provide the labels, the model doesn't output any loss, only the logits!

You can then use these logits with your labels and the loss of your choosing.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
training BERT on coreference resolution 
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I really appreciate if you could add codes to train BERT on coref resolution dataset of CONLL-2012, thanks 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


There's a newer [approach](https://github.com/mandarjoshi90/coref) using BERT but it's using tensorflow 1.14. I wish if we could get this into hugginface.