noamaanMulla-03/transformers-issues huggingface dataset:
Different output encode and encode_plus
Author: Stuffooh
Pull request: no

Hi everyone,

I'm struggling with the following scenario:

I have the following input:

```
sentence = "Sheldon : If a photon is directed through a plane with two slits in it and either is observed . Sheldon : . it will not go through both . If unobserved , it will . UNKNAME : If it 's observed after it left the plane , before it hits its target . Sheldon : . it will not have gone through both slits . Agreed . Leonard : What 's your point ? Sheldon : There 's no point , I just think it 's a good idea for a T shirt . UNKNAME : Excuse me . Hang on . Leonard : One across is Aegean , eight down is Nabokov . Leonard : Twenty six across is MCM . Leonard : Fourteen down is . Leonard : Move your finger . UNKNAME : . phylum , which makes 14 across Port Au Prince . Leonard : See , Papa Doc 's capital idea , that 's Port Au Prince . Leonard : Haiti . UNKNAME : Can I help you ? Yes . UNKNAME : Um , is this the high IQ sperm bank ?"
```

I am using bert-base-uncased and I get different length of tokens depending if I use `tokenizer.encode` or `tokenizer.encode_plus`.

Below is an example:

```
test = tokenizer.encode_plus(sentence, add_special_tokens=True, max_length=512)["input_ids"]
print(test)
214
```
```
test2 = test2 = tokenizer.encode_plus(sentence, add_special_tokens=True, max_length=512)["input_ids"]
print(test2)
189
```

In above scenario I expect the amount of tokens to be the same length. I looked at the documentation but I cannot find an explanation for the difference. It is problematic for me because I need to use tokenizer.batch_encode_plus but for my model I expect and need the length of 189 instead of 214.

Can someone please explain why the output is different and how to make encode.plus output the same as encode?

Thanks in advance ;)








Comments:
I made a mistake in my code. The output is the same.
[examples] summarization/bart/finetune.py supports t5
Author: sshleifer
Pull request: yes

- we were passing attention_mask as an arg, not a kwarg, causing `test_step` to break.
- That case is now covered in the unittest, unittests also cover the t5 model.
- renamed run_bart_sum.py to finetune.py since it is model agnostic. 
- The `bart/` and `t5` subdirectories should be consolidated in a future PR.

This took 15 mins because of underlying infrastructure: unittests for examples and tiny models on S3 :)





Comments:
Super! LGTM

Thanks @sshleifer for the quick fix. Just a small query where it will save the output sentences. A file is generated in output_dir with only losses specified when --do_predict is passed in Argument.
What if I want to generate for unknown inputs using fine-tuned model.
Why does `examples/translation/t5` test on newstest2013 rather than newstest2014?
Author: tholiao
Pull request: no

# Details
<!-- Description of your issue -->
The example in examples/translation/t5 uses `newstest2013`, but authors report against `newstest2014` (presumably newstest2014.full):
> Since this is our final set of experiments, we report results on the test set rather than
the validation set. For CNN/Daily Mail, we use the standard test set distributed with the dataset.
For the WMT tasks, this corresponds to using newstest2014 for English-German
[Original paper, p30](https://arxiv.org/pdf/1910.10683.pdf).

Is this intentional?

Comments:
Hey @tholiao,

Thanks for the catch! You're right it should be the newstest2014! Do you want to open a PR to change it? Or I can do it as well

No worries, I'll submit a PR.

@patrickvonplaten, would you mind running evaluate_wmt.py on this branch and computing sacreBLEU via command line? (`cat newstest2014_de_translations.txt | sacrebleu -t wmt14 -l en-de --tokenize intl`)

Thanks a lot for the PR! I will running the script once the PR is merged :-)
i want help to create saved model(.pth) from Pytorch Dump(pytorch_model.bin) if possible!
Author: pumpkinband
Pull request: no

i have, PROJECT(folder)
                     ├── pytorch_model.bin
                     ├── bert_config.json
                     └── vocab.txt 

i tried saving it with 
`torch.save( pytorch_model.bin , PATH)` 

but came with error 
`-bash: syntax error near unexpected token `pytorch_model.bin,'`

what am i doing wrong ?
and please help me convert pretrained model to saved model( .pth)!!



Comments:
i used `transformers-cli convert` to make `python_model.bin` from checkpoints

Did you try to put in in quotes? If you have a model you should do `torch.save(model.state_dict(), PATH)`.

Please take a look at the [PyTorch documentation](https://pytorch.org/tutorials/beginner/saving_loading_models.html). We prefer using `model.save_pretrained(PATH)`, however, as it saves the configuration object alongside it which is necessary when loading the model afterwards.
Fix token_type_id in BERT question-answering example
Author: siboehm
Pull request: yes

`token_type_id` wasn't being set correctly in the code examples for BERT Question answering. It's being turned into the sequence embedding, hence needs to highlight whether each token belongs to sequence 0 or 1.

For this small example the model returns the correct answer even though the parameter was incorrectly set, but for bigger paragraphs that is not the case.

I changed the code to use encode_plus which returns the correct `token_type_id`.

Comments:
[CI] Add CircleCI workflow to build docs for preview
Author: harupy
Pull request: yes

This PR adds a CircleCI workflow to build the documentation and store it as an artifact so that we can preview it and verify it's rendered properly.

Comments:
The built documentation:

https://30100-155220641-gh.circle-artifacts.com/0/docs/_build/html/index.html

How to view the built documentation:

![doc](https://user-images.githubusercontent.com/17039389/79070944-ba335b80-7d13-11ea-9d19-71e8fc278226.gif)

@LysandreJik this might interest you!

@sshleifer Thanks for the comment!

[The CircleCI doc](https://circleci.com/docs/2.0/artifacts/) says:

> Artifacts are stored on Amazon S3 and are protected with your CircleCI account for private projects. There is a 3GB curl file size limit. **Artifacts will be accessible for thirty days after creation**.
adding --fp16 to run_language_modeling and increase batch size but cuda out of memory error
Author: mahdirezaey
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
Hi all

I am using colab ,  1 GPU , Tesla P100-PCIE-16GB

code below ran OK 

!python /content/transformers/examples/run_language_modeling.py \
    --output_dir=/content/outputs \
    --model_type=bert \
    --model_name_or_path=bert-base-cased \
    --num_train_epochs 1\
    --do_train \
    --do_eval \
    --per_gpu_train_batch_size 152\
    --train_data_file=/content/input_data/trn.txt \
    --eval_data_file=/content/input_data/val.txt \
    --evaluate_during_training \
    --learning_rate 1e-4\
    --overwrite_output_dir\
    --tokenizer_name /content/token/ \
    --block_size 64\
    --mlm





(and batch_size 152 was max num i was able to run without cuda out of memory )
then installing apex by 




%%writefile setup.sh

export CUDA_HOME=/usr/local/cuda-10.1
git clone https://github.com/NVIDIA/apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./apex

!sh setup.sh



then adding "    --fp16\"  to code but i was not able to increase batch size , even abit

do you know that ?
@thomwolf , @VictorSanh , @aaugustin , @BramVanroy , @julien-c , @LysandreJik

is it also the case with GTX 1080 , any one tried ?


and one more thing , 
does any function in those scripts , concatenate the short lines to each other ?
in order not to be enforced to pad each line so much   

Please don't mass-tag people — thanks.

solve :
that was because i was using p100
Writing predictions in a separate output file
Author: vikas95
Pull request: no

## 🚀 Feature

 Request for providing the final predictions (and probabilities of each class for classification task) on the validation/test set in a separate .txt or .json file  

## Motivation

 Since many of us will be using the provided models (RoBERTa, XLnet, BERT etc.) on various other NLP tasks and we will be probably using custom evaluation functions for different tasks, it would be very helpful if the final output predictions on val/test set cab be written in a separate .txt or .json output file. For example, the original BERT tensorflow codes (https://github.com/google-research/bert) writes the final predictions of GLUE tasks in an output file "eval_results.txt" and "predictions.json" for SQuAD. 

## Additional context

I have printed the predictions from the function "evaluate(args, model, tokenizer, prefix="")" in line 189 of run_glue.py but I found the sequence of predictions is not the same as the input validation file. I will hopefully resort the original sequence of predictions in run_glue.py but I think I might have to do more than this in reading comprehension model predictions for SQuAD. I feel many of the users would be looking for this feature and it might help many others if everyone doesnt have to edit evaluate functions on their own individually. 
Looking forward for your kind response and thanks for the help. 

Comments:
Solved it, apologies for raising this silly request.

Hi, how did u solve this?

Hi, I would like to know how can I do it. Thanks

You can access to label predictions thanks to the variable "preds" (line 318 after squeeze function). You can save it in a text file in a similar way of the line 323.
Question Answering support for Albert and Roberta in TF
Author: Pierrci
Pull request: yes

This PR simply adds `TFRobertaForQuestionAnswering` and `TFAlbertForQuestionAnswering` classes (I needed them to do some model conversions!)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3812?src=pr&el=h1) Report
> Merging [#3812](https://codecov.io/gh/huggingface/transformers/pull/3812?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/301bf8d1b43d99efe1fdb5ba15871e975b3cb6cf&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3812/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3812?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3812      +/-   ##
==========================================
+ Coverage   78.27%   78.31%   +0.04%     
==========================================
  Files         106      106              
  Lines       17964    17996      +32     
==========================================
+ Hits        14061    14094      +33     
+ Misses       3903     3902       -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3812?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/3812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `98.98% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/3812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `68.62% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/3812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hbGJlcnQucHk=) | `86.25% <100.00%> (+0.67%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/3812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.00% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3812?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3812?src=pr&el=footer). Last update [301bf8d...44c92f3](https://codecov.io/gh/huggingface/transformers/pull/3812?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add support for the null answer in `QuestionAnsweringPipeline`
Author: bryant1410
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3441?src=pr&el=h1) Report
> Merging [#3441](https://codecov.io/gh/huggingface/transformers/pull/3441?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/010e0460b22ddd7f74e31163f69ab3da2e9741ba&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `66.66%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3441/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3441?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3441      +/-   ##
==========================================
- Coverage   77.61%   77.60%   -0.01%     
==========================================
  Files         100      100              
  Lines       16972    16978       +6     
==========================================
+ Hits        13172    13175       +3     
- Misses       3800     3803       +3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3441?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/3441/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `72.44% <66.66%> (-0.09%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3441/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `83.94% <0.00%> (-0.18%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3441?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3441?src=pr&el=footer). Last update [010e046...8671bc3](https://codecov.io/gh/huggingface/transformers/pull/3441?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi @bryant1410, thanks for pushing this into the QA pipeline.

My only concern is about the name of the parameter `version_2_with_negative` introduced in this PR. It seems very tight up to SQuAD2 and might be hard for newcomers to directly understand what it does.

Would you mind changing the name of the parameter `version_2_with_negative` to `handle_impossible_answer` ? 

Sure. Btw, is it okay for it to default to `False`?

LGTM too, thanks @bryant1410
Optionally convert output of FeatureExtraction pipeline to list
Author: lambdaofgod
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

What is the purpose of .tolist in FeatureExtraction pipeline?
Why is it called? Is this because of some kind of compatibility issue?

If someone needs to use __call__ a lot, it only slows it down. I've tried subclassing FeatureExtractionPipeline, but it's very ugly since then I can't just use it with pipelines.pipeline.

Comments:
Hi @lambdaofgod,

.tolist() call is there as Python's lists are more compatible with CSV/JSON serialisation than numpy array.

Did you have a chance to get number of how actually slower it is ? If the difference is non negligible then we might have a look to optimise .tolist() only when serialising through JSON/CSV

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
Author: neilyboi
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Pipelines "question-answering" with the bert-large-cased-whole-word-masking-finetuned-squad model.

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ X] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details)

Essentially I have a dataset of queries and contexts, and I want to generate a bunch of predictions for answers. The issue is that I cannot get the code to run on GPUs because it seems like the tokenized tensors are not added to the GPU on your end.

## To Reproduce

Steps to reproduce the behavior:

1. Simply try to do pipeline QA on a GPU
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
`import torch
  from transformers import *
  import pandas as pd
  import time

  qa = pipeline(task='question-answering', model= "bert-large-cased-whole-word-masking-   finetuned-squad", device=0, binary_output=True)

df = df.sample(frac=1).reset_index(drop=True)
df['answer'] = ""

context = [str(n) for n in list(df['body'])]

j = 0

for i in range(5,len(context),100):
    start = time.time()
    df.loc[j:i-1,'answer'] = qa(**{'question': list(df['query']),'context': context[j:i]})
    if (i == 5):
       df.to_csv("neil_answers.csv", mode = 'w') 
    else:
        df.to_csv("neil_answers.csv", mode = 'a')
    j = i
    print(time.time()-start)



`
## Expected behavior

<!--  -->
The hope is for the pipeline to generate QA answers and append it to a csv file. The code was working (slowly) before I tried adding a GPU. The issue seems to be around adding the tokenized examples to the GPU

## Environment

* OS:
* Python version: 3.5
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch):
* Using GPU Yes
* Distributed or parallel setup ?
* Any other relevant information:
Running code on a GCP Jupyter Notebook, with one NVIDIA T4 GPU with CUDA `10`

## Additional context


<img width="1709" alt="Screen Shot 2020-01-10 at 2 48 37 PM" src="https://user-images.githubusercontent.com/51249406/72191858-73526480-33b8-11ea-8011-b7091398a1af.png">

Comments:
same issue for me

Hi,

Can you try with the latest 2.4.0 transformers release and let us know if you still observe the same ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Convert pytorch-pretrained-bert to new version (transformers)
Author: Oxi84
Pull request: no

So I have this code for albert:

```py
from transformers import AlbertTokenizer, AlbertForMaskedLM

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertForMaskedLM.from_pretrained('albert-base-v2')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)
loss, prediction_scores = outputs[:2]
```

How do i convert it to old format:

```py
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM

 tokenizer = BertTokenizer.from_pretrained('bert-large-cased')
 model = BertForMaskedLM.from_pretrained('bert-large-cased') 
 tokenized_text_tmp = tokenizer.tokenize(text)
 indexed_tokens_tmp = tokenizer.convert_tokens_to_ids(text_tokens_tmp)
 predictions = model(tokens_tensors, segments_tensors, attention_mask_tensors)
```

How do i get functions tokenizer.convert_tokens_to_ids and tokenizer.tokenize from old version into new one. 
I need to first tokenize text and only after convert it to ids. Because in old code I do this way, changing in new format would take a very long time, because I did a lot of speed optimization and adding of padding.







Comments:
Have you tried using the exact same methods `tokenize` and `convert_tokens_to_ids`?

```py
from transformers import AlbertTokenizer, AlbertForMaskedLM
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2)
model = AlbertForMaskedLM.from_pretrained('albert-base-v2)   

text = "[CLS] Who was Jim Henson ? [SEP]"
tokenized_text = tokenizer.tokenize(text)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)

segments_ids = [0 for x in range(0,len(tokenized_text))]
tokens_tensor = torch.tensor([indexed_tokens])
tokens_tensor = tokens_tensor.to('cuda')

with torch.no_grad():
    predictions_0 = model(tokens_tensor)

print(tokenized_text)  
print(predictions_0)  
del predictions_0 
```

I cannot copy but predictions_0 does not contain 2 elements, but just one.
so:  

loss, prediction_scores = outputs[:2] gives me an error (not in range)

prediction_scores = outputs[0] works, but i do not know what is the output, i hope logits.

I had to format your first comment as it was unreadable, please [see how to use code blocks](https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks).

In your first snipped you're using:

```py
outputs = model(input_ids, masked_lm_labels=input_ids)
```

while in the second snippet:

```py
predictions_0 = model(tokens_tensor)
```

You're not sending the `masked_lm_labels`, which is what is used to compute the loss. If you were to use these labels, the loss would be computed, resulting in a tuple with 2 elements as an output.

Here's the [documentation](https://huggingface.co/transformers/model_doc/albert.html#transformers.AlbertForMaskedLM) for the `AlbertForMaskedLM` model.


Thanks,
Flaky TF pipelines test on CircleCI
Author: sshleifer
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

name: TensorFlow

name: Tests

Environment: CircleCI

Test: `tests/test_pipelines.py::MultiColumnInputTestCase::test_tf_question_answering`

Traceback: https://circleci.com/gh/huggingface/transformers/15691?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

Diff where I changed nothing relevant and the test started passing: https://github.com/huggingface/transformers/pull/2745/commits/a4edf2e878d23346f45715ac213f1f870ae8ec0c

Happy to look deeper if helpful!

Comments:
Indeed, this is a recurring error. I have not yet found the time to dive into it yet, we should do so shortly.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.