noamaanMulla-03/transformers-issues huggingface dataset:
Is there any way to treat the whitespace characters same as other characters when tokenizing?
Author: Hans0124SG
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am trying to work on clinical notes data, and the whitespaces in the notes may contain useful information (e.g. section separation). Is there any way to encode the whitespaces as well during tokenization?

Comments:
Did you try replacing all meaningful whitespaces by a special token `<space>` and just add this new token to the tokenizer and train your model with it?

> Did you try replacing all meaningful whitespaces by a special token `<space>` and just add this new token to the tokenizer and train your model with it?

I guess it will be a bit tricky to define the "meaningful" whitespaces, but I will give it a shot. Thanks :)
Scores using BertForNextSentencePrediction are not Interpretable.
Author: Vivekjoshi731
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
The output of BertForNextSentencePrediction are not Interpretable.
What is seq_relationship_score?
Input example and their respective output is defined below.
1:
text = "[CLS] How old are you? [SEP] I am 193 years old [SEP]"
output=(tensor([[ 3.5181, -2.2946]], grad_fn=<AddmmBackward>)
2:
text = "[CLS] How old are you? [SEP] I am from Paris. [SEP]"
output=tensor([[ 3.9515, -2.5397]], grad_fn=<AddmmBackward>)

Following is my code:
```
import torch
from transformers import *
import torch.nn.functional as F

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = "[CLS] How old are you? [SEP] I am from Paris. [SEP]"
tokenized_text = tokenizer.tokenize(text)
segments_idss=[]
flag=0
for index,token in enumerate(tokenized_text):
	if flag==0:
		segments_idss.append(0)
	else:
		segments_idss.append(1)
	if tokenized_text[index]=='[SEP]':
		flag=1
	
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_idss])
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased')
model.eval()
predictions = model(tokens_tensor, segments_tensors)
print("predictions->",predictions[0])
```
Can someone tell me, Why scores are similar for different sentences and how to use BertForNextSentencePrediction to find next sentence score?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
inconsistency of the model (XLNet) output  / related to #475 #735
Author: cherepanovic
Pull request: no
Labels: name: wontfix

Related to #475 #735

Unfortunately, I lost the overview regarding this issue. 
 

what is the final solution for that problem? 

```
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
config = XLNetConfig.from_pretrained('xlnet-base-cased')
config.output_hidden_states=True
xlnet_model = XLNetModel(config)
xlnet_model.from_pretrained('xlnet-base-cased')
xlnet_model.eval()
```
this configuration is still inconsistent.

best regards 

Comments:
if I don't load the config the results are consistent 

```
tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
model = XLNetLMHeadModel.from_pretrained("xlnet-base-cased")
model.eval()
```


You should do this:
```
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
config = XLNetConfig.from_pretrained('xlnet-base-cased')
config.output_hidden_states=True
model = XLNetLMHeadModel.from_pretrained('xlnet-base-cased', config=config)
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to return  bert self attention, so that i can do visualization??
Author: alshahrani2030
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels= 2, output_attentions=True)
model.cuda()
I am useing the above code to return the attention weights, for visualizing the attention by BertViz.
But it gave me this error ( __init__( ) got an unexpected keyword argument 'output_attentions').
Also, please if you recommend any tutorial for beginner, which explain how to return the attention
<!-- A clear and concise description of the question. -->


Comments:
Hi! Could you specify which version of our library you are using? Thank you.

Hi
I am useing "pip install pytorch-pretrained-bert pytorch-nlp"

I believe the way to output attentions in `pytorch-pretrained-BERT` v0.6.2 was to specify the `output_all_encoded_layers` to `True` in the model forward call.

Please be aware that this version has been deprecated for some time now. The new version is called `transformers` and should be installed with `pip install transformers`.


Thank for the quick reply
how about if i want to use transformers how to output the attention ??

If you want to use transformers to output the attention you can specify it in the config:

```py
config = BertConfig.from_pretrained("bert-base-cased", output_attentions=True, num_labels=2)
model = BertForSequenceClassification.from_pretrained("bert-base-cased", config=config)
```

> I believe the way to output attentions in pytorch-pretrained-BERT v0.6.2 was to specify the output_all_encoded_layers to True in the model forward call.
> Please be aware that this version has been deprecated for some time now. The new version is called transformers and should be installed with pip install transformers.

Do you mean some things like this
# Forward pass
loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels, output_all_encoded_layers = True)

Yes, that’s what I meant!

> Yes, that’s what I meant!

I am getting  this error 
TypeError: forward() got an unexpected keyword argument 'output_all_encoded_layers'

> Yes, that’s what I meant!
I am getting this error. any idea please
TypeError: forward() got an unexpected keyword argument 'output_all_encoded_layers'
![attentionError](https://user-images.githubusercontent.com/55197626/66332937-e1901280-e903-11e9-8af6-0587d7a00296.PNG)



Which version of the lib are you using in that example?

> Which version of the lib are you using in that example?

old one 
pytorch-pretrained-bert pytorch-nlp

Is there any way you could update to `transformers`? That would make life easier.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pretty sure patch in Pull Request #1313 is incorrect
Author: jasonrohrer
Pull request: no

## 🐛 Bug

The bug was introduced in pull request #1313

If a stop_token is specified, but does not exist in **text**, then the last character of the text string is trimmed off.  text.find will return -1 in that case, which seems to remove the last character from the string.

Example:
--stop_token="wake"

`text = tokenizer.decode(o, clean_up_tokenization_spaces=True)`

text = ' fishing with two ladies and the same two ladies had been fishing with him and all three ladies had run'

` text = text[: text.find(args.stop_token) if args.stop_token else None]`

text = ' fishing with two ladies and the same two ladies had been fishing with him and all three ladies had ru'

See how that final 'n' is trimmed off, even though "wake" does not occur in this string?

The fix should be to first find out if stop_token occurs in the text before doing the trimming, maybe:

```
if args.stop_token :
    loc = text.find(args.stop_token)
    if loc != -1 :+1:
        text = text[:loc]
```

Comments:
And yes, the above code does fix it.

Example:
--stop_token="wake"

text = ' to the same conclusion: that her husband, the guy she had chosen for her favorite, had been'

trimmed text = ' to the same conclusion: that her husband, the guy she had chosen for her favorite, had been'



text = ' murdered the previous spring.  Oedipa closed her eyes and tried to wake up.'

trimmed text = ' murdered the previous spring.  Oedipa closed her eyes and tried to '

Thank you for raising this issue @jasonrohrer, there indeed was an error with the stop token. It's been fixed in 18a879f.
Updated examples/README and parser for run_summarization_finetuning
Author: DerekChia
Pull request: yes

1. Updated `examples/README.md` to change default `--model_type` and `--model_name_or_path` to `bert` and `bert_base_cased` because `bert2bert` just won't work
2. Updated `examples/run_summarization_finetuning.py` parser to take in `--do-train` instead of `--do-train=True` for consistency with other examples and `--model_type` + `--model_name_or_path`
3. Changed `add_special_tokens_single_sequence` to `build_inputs_with_special_tokens` in `examples/utils_summarization.py`  

Comments:
Let's wait that the summarization script is finalized before merging this.
Excessively Long text_b Raises Unnecessary Warnings in `encode_plus`
Author: frankfka
Pull request: no
Labels: name: wontfix

In `encode_plus`, `convert_ids_to_tokens` is called before truncating to `max_len`. However, if either text_a or text_b are longer than `max_len`, `convert_ids_to_tokens` will raise a warning. Since sequences are truncated to the right length afterwards in `encode_plus`, this warning is unnecessary.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fine-tuning GPT2 or BERT and adding new vocabulary?
Author: masoudh175
Pull request: no

## ❓ Questions & Help

I am fine-tuning the GPT2 on a domain-specific corpus and I was wondering if there is a way to add new vocabulary to the model. I am checking the ```vocab.json``` file after saving the model into ```output``` directory and don't find the any new vocabulary in there.


Comments:
Hi, you can add new vocabulary using the tokenizer's `add_tokens` method.

Thanks for the quick response!
```glue_convert_examples_to_features``` for sequence labeling tasks
Author: antgr
Pull request: no

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
I would like a function like ```glue_convert_examples_to_features``` for sequence labelling tasks.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->
The motivation is that I need much better flexibility for  sequence labelling tasks. Its not enough to have a final, and decided for me model, for this task. I want just the features (a sequence of features/embeddings I guess).

## Additional context
This can be generalized to any dataset with a specific format.

<!-- Add any other context or screenshots about the feature request here. -->


Comments:
duplicate to  2208
FR: Tokenizer function that can handle arbitrary number of sequences
Author: Peter-Devine
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Currently Tokenizers only support 1 or 2 sequences being added together, and them being concatenated with the appropriate SEP and CLS tokens for each model. My use case requires more sequences being added together, all separated by SEP tokens and having one CLS token at the start (or end for XLNet) of the entire sequence.

E.g., for BERT:
```
[CLS] This is my first sentence [SEP] This is my second sentence [SEP] And finally my third sentence [SEP]
```

Would it be possible to have a function supported for current and future models that can take a list of strings in and process them as outlined above.

I would be especially happy if there was an accompanying feature for token type ids, which would alternate between successive sequences.
Using the above example, this would mean:
```
0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0
```

Thanks!


Comments:
Looks like a pretty simple and natural extension, what do you think @LysandreJik?

It would be easy to implement indeed, do you use this for dialog (because of the alternating token type ids)?

Not particularly, but I can certainly imagine that being a useful use-case.

I am more interested in adding extra features directly to the language model when classifying a piece of text. For example:
`[Location] + [Occupation] + [Social media post]`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
update Roberta checkpoint conversion
Author: armancohan
Pull request: yes

- update to fix fairseq Roberta checkpoint conversion 

Fairseq had removed `in_proj_weight` and `in_proj_bias` from the self attention module:
https://github.com/pytorch/fairseq/commit/4c6b689eebe66a53717dacf28cba7a11b6ffa64f

- create save directory if not exist

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1959?src=pr&el=h1) Report
> Merging [#1959](https://codecov.io/gh/huggingface/transformers/pull/1959?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5e289f69bc564c94132f77c89a34e5f1dd69a592?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1959/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1959?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1959   +/-   ##
=======================================
  Coverage   81.47%   81.47%           
=======================================
  Files         122      122           
  Lines       18342    18342           
=======================================
  Hits        14945    14945           
  Misses       3397     3397
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1959?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1959?src=pr&el=footer). Last update [5e289f6...5190320](https://codecov.io/gh/huggingface/transformers/pull/1959?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks a lot @armancohan 

I think we'll want to keep the possibility to import both types of models for backward compatibility. Can you add a switch based on an identification of the model type?

@armancohan (rebased on top of master so I force-pushed to your fork)

So, the fairseq weights themselves didn't change, it's the multiheadattention API that did, in fairseq `0.9.0`. So I'll just check that the fairseq version is >= 0.9 in the script.

I've also updated the script to not hardcode the vocab length, which should make it compatible with other roberta-like fairseq models such as CamemBERT + XLM-R out of the box.

cc @myleott @louismartin 

thanks @julien-c
Decode now calls private property instead of public method
Author: LysandreJik
Pull request: yes

Removes the warning raised when the decode method is called.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1087?src=pr&el=h1) Report
> Merging [#1087](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1087?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/e00b4ff1de0591d5093407b16e665e5c86028f04?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `33.33%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1087/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1087?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #1087      +/-   ##
=========================================
- Coverage   79.61%   79.6%   -0.02%     
=========================================
  Files          42      42              
  Lines        6898    6898              
=========================================
- Hits         5492    5491       -1     
- Misses       1406    1407       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1087?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1087/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3V0aWxzLnB5) | `85.9% <33.33%> (-0.33%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1087?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1087?src=pr&el=footer). Last update [e00b4ff...2ba1a14](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1087?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Yes! Thanks @LysandreJik LGTM
Fix distributed barrier hang
Author: VictorSanh
Pull request: yes

This is bug reported in issue #998 (and is also valid for `run_squad.py`).

What is happening?
When launching a distributed training on one of the task of the GLUE benchmark (for instance this suggested command in the README [here](https://github.com/huggingface/pytorch-transformers#fine-tuning-bert-model-on-the-mrpc-classification-task) for GLUE or [here](https://github.com/huggingface/pytorch-transformers#run_squadpy-fine-tuning-on-squad-for-question-answering) for SQUAD), the training is performed in a distributed setting (expected behavior). Evaluation can be tricky for certain metrics in a distributed setting so the evaluation is performed solely by the master process (cf L476: `if args.do_eval and args.local_rank in [-1, 0]:`).

During the evaluation, the process hangs/gets stucked at L290 (`torch.distributed.barrier()`). It turns out that all the processes except the master one already exit at L476 and thus never enter the symmetric `torch.distributed.barrier()` at L254-255. It means that the master process is waiting at L290 for his process friends who already left the party without telling him (printing a `torch.distributed.get_world_size()` at L290 during evaluation reveals torch is expecting `$NGPU` processes). 

Adding a `and not evaluate` condition both at L254 and L289 is a solution to fix the bug (the master process is the only surviving process at evaluation, so no need to wait for others...)


Comments:
Ok great, thanks a lot @VictorSanh
Better examples
Author: LysandreJik
Pull request: yes

Refactored the examples section: removed old and not up-to-date examples and added new examples for fine-tuning and generation.

The `examples` file is not in the `/doc/source` folder anymore but in the `/examples` folder. It is therefore visible when users open the folder on GitHub. 

Note: In order to generate the current documentation, a symlink has to be done between the `examples/README.md` file to a `docs/source/examples.md`. The corresponding documentation has been added to the documentation README, alongside the command necessary to create the symlink.

**The new examples are visible on: http://lysand.re/examples.html**

Comments:
There were indeed quite a few artifacts. I fixed them in the two latest commits.

# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1214?src=pr&el=h1) Report
> Merging [#1214](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1214?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/5ac8b62265efac24f0dbfab271d2bce534179993?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1214/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1214?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1214   +/-   ##
=======================================
  Coverage   81.29%   81.29%           
=======================================
  Files          57       57           
  Lines        8015     8015           
=======================================
  Hits         6516     6516           
  Misses       1499     1499
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1214?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1214?src=pr&el=footer). Last update [5ac8b62...3f91338](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1214?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Awesome! Merging