noamaanMulla-03/transformers-issues huggingface dataset:
GPT2 Tokenizer Decoding Adding Space
Author: harkous
Pull request: no

## 🐛 Bug

The GPT-2 tokenizer's decoder now adds a space at the beginning of the string upon decoding.



(Potentially causing #1254)

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Run the following code:

```bash
from pytorch_transformers.tokenization_gpt2 import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.decode(tokenizer.encode("test phrase"))
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

The expected decoded string is "test phrase". However, currently it produces " test phrase".

## Environment

* OS: OSX
* Python version: 3.7.3
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): master (#e768f2322abd2a2f60a3a6d64a6a94c2d957fe89)

* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:


Comments:
Also getting this effect when using the reproduction code on my system.

It's not a bug. This is an artefact produced by BPE as explained here https://github.com/huggingface/pytorch-transformers/blob/d483cd8e469126bed081c59473bdf64ce74c8b36/pytorch_transformers/tokenization_gpt2.py#L106

I think the solution is to process whitespaces after the tokeniser.
BertTokenizer provides wrong encode function for Japanese BERT
Author: khaimaitien
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BertTokenizer 

Language I am using the model on (English, Chinese....): Japanese

I tried to load the tokenizer for Bert from pretrained [Bert for Japanese](http://nlp.ist.i.kyoto-u.ac.jp/index.php?BERT%E6%97%A5%E6%9C%AC%E8%AA%9EPretrained%E3%83%A2%E3%83%87%E3%83%AB). The tokenizer recognize "が" the same as "か"　although both appears in vocab file. Here is example code:

```
from pytorch_transformers import BertTokenizer 
tokenizer = BertTokenizer.from_pretrained('vocab.txt')
token_ids = tokenizer.encode('が')
print('token_ids: ', token_ids)
print(tokenizer.decode(token_ids))

token_ids = tokenizer.encode('か')
print('token_ids: ', token_ids)
print(tokenizer.decode(token_ids))
```
the result:
```
token_ids:  [90]
か
token_ids: [90]
か
```
The vocab.txt file can be downloaded from [here](https://drive.google.com/open?id=1f3k9GcyqEIjjFo8EgqqaOQmiSXxT1hqF): 
I also found that BertTokenizer also mis-recognized: 'て' and 'で', 'ば' and 'は'

Comments:
I had discovered that this phenomenon is due to function: _run_strip_accents(token) in class: BasicTokenizer. Perhaps, the authors should give an option to choose whether to remove accents or not because in some language such as Japanese, removing accents makes a new word

Hi, I have trained this Japanese BERT model, and made it public.
Please set `do_lower_case` option to false so that the function `_run_strip_accents` is disabled.
Optimize XLNet model to generate embedding of long documents
Author: FannySB
Pull request: no

We experiment generating embeddings with TransformerXL and XLnet. Our documents have 5000 to 80000 characters each.

We got an average of 0.8 second per document with TransformerXL and 1.3 second per document with XLNet.

To optimize XLNet we found that using only 200 tokens per call is optimal. 
A ratio around 350 tokens/second with xlnet-base-cased and 100 tokens/second with large.

Any idea if XLNet can be optimize? 

    tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
    model = XLNetModel.from_pretrained("xlnet-base-cased")

    outputs = model(text_tokens, mems=mems)

Comments:
AttributeError: 'RobertaTokenizer' object has no attribute 'add_special_tokens_sentences_pair'
Author: frankfka
Pull request: no

With the latest update to `Transformers`, has the function been removed? I still see it in the code, but I run into the error:

`AttributeError: 'RobertaTokenizer' object has no attribute 'add_special_tokens_sentences_pair'`

Comments:
Hey @frankfka 

Perhaps you are looking for tokenizer.add_special_tokens_sequence_pair instead of tokenizer.add_special_tokens_sentences_pair?

```
from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")

tokenizer.add_special_tokens_sequence_pair([31414], [31414])
```
This returns <br>
```
[0, 31414, 2, 2, 31414, 2]
```

> Hey @frankfka
> 
> Perhaps you are looking for tokenizer.add_special_tokens_sequence_pair instead of tokenizer.add_special_tokens_sentences_pair?
> 
> ```
> from transformers import RobertaTokenizer
> tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
> 
> tokenizer.add_special_tokens_sequence_pair([31414], [31414])
> ```
> 
> This returns
> 
> ```
> [0, 31414, 2, 2, 31414, 2]
> ```

Good catch, thanks! I suppose this was renamed in this release?
RobertaTokenizer documentation is off with the new transformers library
Author: cformosa
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Roberta

Language I am using the model on (English, Chinese....): NA

The problem arise when using:
* [ ] the official example scripts:

The tasks I am working on is:
NA

## To Reproduce

Steps to reproduce the behavior:

In the documentation for the tokenization_roberta.py, it says in the RobertaTokenizer class

```
    RoBERTa BPE tokenizer, derived from the GPT-2 tokenizer. Peculiarities:
        - Byte-level Byte-Pair-Encoding
        - Requires a space to start the input string => will add a space is there isn't.
          As a consequence, this tokenizer `encode` and `decode` method will not conserve
          the absence of a space at the beginning of a string: `tokenizer.decode(tokenizer.encode("Hello")) = " Hello"
```
However, with using the new transformers library, when I run this example I get
```
from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")

tokenizer.decode(tokenizer.encode("Hello"))
"Hello"
```
The leading space seems to no longer present as it was in pytorch_transformers, however if (per the source code) if I add the arg add_prefix_space = True, then it outputs with the leading space. Just a tiny fix to hopefully help out anyone else who gets confused by it.

Thanks and love the new updates to the library!


Comments:
You're right! Thanks for letting us know.
Typo in modeling_bert file
Author: ishan00
Pull request: no

I was looking at the code of BertModel adapted for different tasks here 
https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py

I noticed a small typo in line 882
`self.classifier = nn.Linear(config.hidden_size, self.config.num_labels)`
I think it should be either `self.num_labels` or `config.num_labels` in the second argument

The complete function
```
def __init__(self, config):
        super(BertForSequenceClassification, self).__init__(config)
        self.num_labels = config.num_labels

        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, self.config.num_labels)

        self.init_weights()
```

Comments:
Hi! Indeed it is inconsistent, but it  doesn't really change anything as the superclass `PreTrainedModel` assigns the config as one of its attributes: `self.config = config`. Referencing `config` or `self.config` therefore references the same object!
faster dataset building
Author: mgrankin
Pull request: yes

Now it takes around 1 minute to process 20mb and it takes forever for 200mb dataset (it's non-linear). This is a fix to make it linear.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1337?src=pr&el=h1) Report
> Merging [#1337](https://codecov.io/gh/huggingface/transformers/pull/1337?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a3e0dbba9512866064c20e9bc99c62725f6c36fb?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1337/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1337?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1337   +/-   ##
=======================================
  Coverage   84.73%   84.73%           
=======================================
  Files          84       84           
  Lines       12573    12573           
=======================================
  Hits        10654    10654           
  Misses       1919     1919
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1337?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1337?src=pr&el=footer). Last update [a3e0dbb...f71a457](https://codecov.io/gh/huggingface/transformers/pull/1337?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I was about to make this PR myself and then saw this!

Thanks a lot @mgrankin (was meaning to fix this as well haha)!
Add small  note about the output of hidden states (closes #1332)
Author: BramVanroy
Pull request: yes

Closes huggingface/transformers#1332

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1346?src=pr&el=h1) Report
> Merging [#1346](https://codecov.io/gh/huggingface/transformers/pull/1346?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/da2e47ad15e552b84815da20daf3282b517103f7?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1346/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1346?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1346   +/-   ##
=======================================
  Coverage   84.73%   84.73%           
=======================================
  Files          84       84           
  Lines       12573    12573           
=======================================
  Hits        10654    10654           
  Misses       1919     1919
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1346?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1346?src=pr&el=footer). Last update [da2e47a...15749bf](https://codecov.io/gh/huggingface/transformers/pull/1346?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Awesome, thanks @BramVanroy!
SQUAD: V2 referenced at top of Readme; V1 referenced in usage instructions
Author: descartesholland
Pull request: no

## ❓ Questions & Help

There seems to be an inconsistency in the README, namely that run_squad.py is cited to be trained on SQUAD v2 towards the top, but scrolling down to view the command shows that v1 is used. Running the command cited over a copy of the v2 dataset on my machine yields the following error:

```
Traceback (most recent call last):
  File "./examples/run_squad.py", line 533, in <module>
    main()
  File "./examples/run_squad.py", line 478, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
  File "./examples/run_squad.py", line 291, in load_and_cache_examples
    version_2_with_negative=args.version_2_with_negative)
  File "./examples/utils_squad.py", line 151, in read_squad_examples
    "For training, each question should have exactly 1 answer.")
ValueError: For training, each question should have exactly 1 answer.

```

Comments:
Need to use the flag 
--version_2_with_negative
can't find utils_glue
Author: YosiMass
Pull request: no
Labels: name: wontfix

import fails 

Comments:
It's hard to tell what could be the source of the error without describing with some minimum details what you tried to do/run and which version of the code you are running...

_If_ you are running `run_glue.py` in the examples, make sure that [`utils_glue.py`](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_glue.py) is present in the same working directory in which you are executing the script.

Thanks David. Turns out that the import in e.g., `run_glue.py` should be 

`from examples.utils_glue import ...`

instead of 

`from utils_glue import ...`

Maybe it's a problem of PyCharm IDE

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to use run_squad.py to produce multiple answers for a question?
Author: mushro00om
Pull request: no
Labels: name: wontfix

Hello,
I am using run_squad.py to build my own question answering system, the problem is that, I want the system can output multiple answers for a question.
The number of answers can be 0, or one, or multiple if possible, how can I do to the code to achieve this? Thank you

Comments:
soloved

Can you share how did you solve that problem?

@armheb 
Sure, some questions in my dataset have multiple answers, some have one answer, some no answer.

Firstly, I add a for loop in the "read_squad_example" method to allow the code to read all answers for each question and build N SquadExamples for each question, N is the number of answers (This is for my case, you don't have to do it, because I need to use all answers, the original squad code only reads the first answer of each question even the question has multiple answers).

The run_squad.py produces a "nbest_predictions.json" file, you can see the model provides top 20 possible answers for each question, with possibilities, so I just simply pick some of those answers according to their possibilities.

However, I have to admit that eventually the performance isn't that good. it works but just not that good, but I think it can be improved by some way.

@mushro00om 
Hi, 
Can you give sample codes for how you used your model for prediction given a text corpus and a question?

@Swathygsb Hi, sorry for late reply. Actually the code script I used is not this Pytorch version, I used the Tensorflow version provided by Google, because it is much more easier and they provide very clear guidance, Here is the link:
https://github.com/google-research/bert
The most of the code remained unchanged, I basically modified the read_squad_examples method to allow process multiple answers (in my task, a question may have more than one answer, the original code can only process one answer for each question). 
So if all your questions have only one particular answer, you can simply follow the guidance, or if your questions may have more than one answer, you can give me your email and i can send my code to you.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Just some typos
Author: ogabrielluiz
Pull request: yes


Comments:
👍 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1349?src=pr&el=h1) Report
> Merging [#1349](https://codecov.io/gh/huggingface/transformers/pull/1349?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d83d295763b738aa0c071f8b63ad6e155b6cf515?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1349/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1349?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1349   +/-   ##
=======================================
  Coverage   84.73%   84.73%           
=======================================
  Files          84       84           
  Lines       12573    12573           
=======================================
  Hits        10654    10654           
  Misses       1919     1919
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1349?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1349?src=pr&el=footer). Last update [d83d295...d2de5b9](https://codecov.io/gh/huggingface/transformers/pull/1349?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
max_len_single_sentence should be max_len - 2 for RoBERTa
Author: kaushaltrivedi
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): RoBERTa

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details) run_lm_finetuning
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details) . Language model finetuning

## To Reproduce

Steps to reproduce the behavior:

1. 
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
While using language model finetuning for roberta-base, I got an error cublas runtime error : resource allocation failed at /pytorch/aten/src/THC/THCGeneral.cpp:216.

When I checked the tokenized dataset, I observed that it had 514 tokens, i.e. 512 coming from max_len_single_sentence plus 2 special tokens.  RoBERTa tokenizer should have max_len_single_sentence set to 510 just like the one in BERT.  

max_len_single_sentence = max_len - 2


## Environment

* OS: Ubuntu
* Python version: 3.7
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): Master
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information: 

## Additional context

<!-- Add any other context about the problem here. -->
R

Comments:
I think you may be right and we've been meaning to fix this. cf recent discussion @LysandreJik @VictorSanh 

thanks. Adding LM fine-tuning to fast-bert.  Have added a workaround for now :)

Also see https://github.com/pytorch/fairseq/issues/1187
Fix tensorflow_dataset glue support
Author: agrinh
Pull request: yes

This PR fixes issue #1354 .

`glue_convert_examples_to_features` assumed that tensorflow_dataset examples contains the features `'sentence1'` and `'sentence2'`. This commit encapsulates the choice of features in the glue processor and uses that to parse examples.

Built with @philipp-eisen .

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1355?src=pr&el=h1) Report
> Merging [#1355](https://codecov.io/gh/huggingface/transformers/pull/1355?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ca559826c4188be8713e46f191ddf5f379c196e7?src=pr&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `50%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1355/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1355?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1355      +/-   ##
==========================================
- Coverage   84.73%   84.68%   -0.05%     
==========================================
  Files          84       84              
  Lines       12573    12592      +19     
==========================================
+ Hits        10654    10664      +10     
- Misses       1919     1928       +9
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1355?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/data/processors/utils.py](https://codecov.io/gh/huggingface/transformers/pull/1355/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2RhdGEvcHJvY2Vzc29ycy91dGlscy5weQ==) | `46.66% <100%> (+1.21%)` | :arrow_up: |
| [transformers/data/processors/glue.py](https://codecov.io/gh/huggingface/transformers/pull/1355/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2RhdGEvcHJvY2Vzc29ycy9nbHVlLnB5) | `27.98% <47.36%> (+1.76%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1355?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1355?src=pr&el=footer). Last update [ca55982...795b3e7](https://codecov.io/gh/huggingface/transformers/pull/1355?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Nice and clean, thanks a lot @agrinh and @philipp-eisen!

@thomwolf Happy to help, we're finding this package super useful!
Update run_lm_finetuning.py
Author: dennymarcels
Pull request: yes

The previous method, just as phrased, did not exist in the class.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1359?src=pr&el=h1) Report
> Merging [#1359](https://codecov.io/gh/huggingface/transformers/pull/1359?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ca559826c4188be8713e46f191ddf5f379c196e7?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1359/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1359?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1359   +/-   ##
=======================================
  Coverage   84.73%   84.73%           
=======================================
  Files          84       84           
  Lines       12573    12573           
=======================================
  Hits        10654    10654           
  Misses       1919     1919
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1359?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1359?src=pr&el=footer). Last update [ca55982...9478590](https://codecov.io/gh/huggingface/transformers/pull/1359?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks @dennymarcels!