noamaanMulla-03/transformers-issues huggingface dataset:
ValueError: For training, each question should have exactly 1 answer.
Author: RAXAI
Pull request: no
Labels: name: wontfix

Tried to run run_squad.py with the squad 2.0 dataset and came up with this error, ValueError: For training, each question should have exactly 1 answer. How do I solve this?

Comments:
Please give more information: the command used (arguments passed), traceback (the command line output), and version (can use `pip show pytorch_pretrained_bert`)

I faced a similar problem with `read_squad_examples` when passing `input_file=dev.json` and `is_training=True`. 

I have this problem when training on SQUADv2 without `--version_2_with_negative` option. Basically for squad 2.0, it is possible there is no answer for questions. Adding this option in training command fixed the problem for me.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi,

I see the same error. Some of the questions have multiple answer spans. 

The error suggests that the current preprocessing codes cannot handle multiple answer spans for a given question.

 Has anyone fixed this ??

Thanks

I also receive this error when using the `--version_2_with_negative` flag paired with training data from SQuAD 2.0. It looks like it may be caused by some logic in `utils_squad.py`, lines 150-152; it seems that answerable questions are expected to only have one answer. I'm not familiar enough with the task and data set to know if this is a correct assumption, but since I'm using data from the SQuAD 2.0 web site I would think it should train fine.

For training, the assumption is true.

On Mon, Nov 18, 2019 at 09:56 Allen Kim <notifications@github.com> wrote:

> I also receive this error when using the --version_2_with_negative flag
> paired with training data from SQuAD 2.0. It looks like it may be caused by
> some logic in utils_squad.py, lines 150-152; it seems that answerable
> questions are expected to only have one answer. I'm not familiar enough
> with the task and data set to know if this is a correct assumption, but
> since I'm using data from the SQuAD 2.0 web site I would think it should
> train fine.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/554?email_source=notifications&email_token=AIEAE4GLKEIMFR3BUJQK5VTQUHY4BA5CNFSM4HI7NMRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEI5VNI#issuecomment-554818229>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIEAE4H7OARRY4INFJTGU6DQUHY4BANCNFSM4HI7NMRA>
> .
>


Thanks for clarifying!
how to use convert_pytorch_checkpoint_to_tf2.py 
Author: chiyuzhang94
Pull request: no

## ❓ Questions & Help
I am wondering how to use convert_pytorch_checkpoint_to_tf2.py to convert pytorch checkpoint to tensorflow checkpoint.  I found Comparing-PT-and-TF-models.ipynb have an example to use pytorch_pretrained_bert. But It doesn't work for convert_pytorch_checkpoint_to_tf2. Could you please give me any help?
<!-- A clear and concise description of the question. -->


Comments:
Hi, you can show the help with the following command:

```
python transformers/convert_pytorch_checkpoint_to_tf2.py --help  
```

What is your use-case?

> Hi, you can show the help with the following command:
> 
> ```
> python transformers/convert_pytorch_checkpoint_to_tf2.py --help  
> ```
> 
> What is your use-case?

Thanks for your suggestion. I used convert_bert_pytorch_checkpoint_to_original_tf. It works for me.
Adds CamemBERT to Model architectures list
Author: Paethon
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1859?src=pr&el=h1) Report
> Merging [#1859](https://codecov.io/gh/huggingface/transformers/pull/1859?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0477b307c7501ea76e01b03cb387a2312db752b3?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1859/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1859?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1859   +/-   ##
=======================================
  Coverage   84.08%   84.08%           
=======================================
  Files          97       97           
  Lines       14316    14316           
=======================================
  Hits        12037    12037           
  Misses       2279     2279
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1859?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1859?src=pr&el=footer). Last update [0477b30...63af013](https://codecov.io/gh/huggingface/transformers/pull/1859?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks! Slightly reworded in next commit
Cannot install the library 
Author: YimingSun60
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 659, in unpack_http_url
    hashes)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 882, in _download_http_url
    _download_url(resp, link, content_file, hashes)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 603, in _download_url
    hashes.check_against_chunks(downloaded_chunks)
  File "/usr/lib/python2.7/dist-packages/pip/utils/hashes.py", line 46, in check_against_chunks
    for chunk in chunks:
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 571, in written_chunks
    for chunk in chunks:
  File "/usr/lib/python2.7/dist-packages/pip/utils/ui.py", line 139, in iter
    for x in it:
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 560, in resp_read
    decode_content=False):
  File "/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py", line 436, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py", line 384, in read
    data = self._fp.read(amt)
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/filewrapper.py", line 63, in read
    self._close()
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/filewrapper.py", line 50, in _close
    self.__callback(self.__buf.getvalue())
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/controller.py", line 275, in cache_response
    self.serializer.dumps(request, response, body=body),
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/serialize.py", line 87, in dumps
    ).encode("utf8"),
MemoryError


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run glue problem
Author: ZTurboX
Pull request: no

## ❓ Questions & Help

How to fix this problem
![image](https://user-images.githubusercontent.com/5669444/69006304-2f1e8700-0968-11ea-9ebf-0fe4c4c19f83.png)


Comments:
Could you please respect the issue template so that we can help you better? It's hard to help without seeing the versions or the full error.
Training time increased from 45 min per epoch to 6 hours per epoch in colab 
Author: vamshi975
Pull request: no
Labels: name: wontfix

## 📚 Migration

<!-- Important information -->

Model I am using (Bert): PyTorch-pretrained-Bert and BERT in pytorch-transformers

Language I am using the model on : English

The problem arise when using:
*  the official example scripts: run_squad.py in pytorch-transformers

The tasks I am working on is:
* an official GLUE/SQUaD task: Fine Tuning SQuAD

Details of the issue:

Hi finetuned model over SQuAD with the following code in colab:
%run run_squad.py --bert_model bert-base-uncased \
                  --do_train \
                  --do_predict \
                  --do_lower_case \
                  --fp16 \
                  --train_file SQUAD_DIR/train-v1.1.json  \
                  --predict_file SQUAD_DIR/dev-v1.1.json \
                  --train_batch_size 12 \
                  --learning_rate 3e-5 \
                  --num_train_epochs 2.0 \
                  --max_seq_length 384 \
                  --doc_stride 128 \
                  --output_dir debug_squad9

The above code took 45 min for one epoch with  "exact_match": 81.94891201513718, "f1": 89.02481046726041. I ran this code(PyTorch-pretrained-Bert) somewhere between July 20th - 27th 2019.

And now the same file with the above code(PyTorch-pretrained-Bert) is taking the 6 hours for one epoch. Why is that?  

I have tested with pytorch-transformers as well, It is also taking 6 hours for one epoch.
 
I am unable to understand. Is there any change with apex or implementation ??
 
## Environment

* OS: google colab
* Python version: Python 3.6.8
* PyTorch version: 1.2
* PyTorch Transformers version (or branch): the existing version pytorch-transformers
* Using GPU ? - Yes I guess K80 single GPU
* Distributed of parallel setup ? No
* Any other relevant information:
- The above environment details on based on the exiting run which took 6 hours for one epoch.
- I haven't collected the environment information when I tested it for the first time over PyTorch-pretrained-Bert.

## Checklist

- [yes ] I have read the migration guide in the readme.
- [yes ] I checked if a related official extension example runs on my machine.

## Additional context

Is there anything new that I am missing from pytorch-Transformers or implementation change in PyTorch-pretrained-Bert ??

Comments:
Hi, 

Found a solution to my problem to some extent. I had cloned the latest apex repo and testing out with PyTorch-pretrained-Bert which is causing the problem to take more time for execution. I took the older apex repo and tested the code and working as earlier (45 min per epoch)

But when I was using the pytorch-transformers with latest apex repo it is taking 6 hours for one epoch. 

Is that usual to take 6 hours for one epoch in colab ??


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
What is the best CPU inference acceleration solution for BERT now?
Author: guotong1988
Pull request: no
Labels: name: wontfix

Thank you very much.
Thank you very much.
Thank you very much.

Comments:
Give us a little more details about your `(latency, compute)` constraints.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
XLNetForSequenceClassification and CLS token
Author: ghost
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, 
As we can see in the official XLNet code ([https://github.com/zihangdai/xlnet]()), the [SEP] and [CLS] tokens are added before the zero-padding. (in convert_single_sequence() in classifier_utils.py)
(example : "This is my sequence ! [SEP][CLS]") 
What is strange is that in the XLNetModelForSequenceClassification, we use a SequenceSummary which will keep the last value from the hidden state, i.e. the last token from padding (which doesn't correspond to the real [CLS] token)? (As I can see, it's working like that in the official code too).

Maybe someone here can explain me if this is an error, or if I forgot something ?  

Comments:
Ok, I found why : the padding is done before the sentence, and not after.
run_squad hangs for small max_seq_length
Author: immawatson
Pull request: no

## 🐛 Bug

I am experimenting with minimal examples to understand how sequences are generated, setting `max_seq_length` very low to trigger document chunking, and I hit this bug:

When `max_seq_length` is smaller than the query, `max_tokens_for_doc` becomes negative causing an infinite loop. 

https://github.com/huggingface/transformers/blob/f3386d938348628c91457fc7d8650c223317a053/examples/utils_squad.py#L241-L258

Obviously having `max_seq_length` smaller than the query isn't very useful, and I should be setting `max_query_length`, But it would still be nice to have an assertion catch this.

### Steps to reproduce
1. `rm $SQUAD_DIR/cached*`
2. train with `--max_seq_length 16`
```
python examples/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --do_lower_case \
    --do_train \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --per_gpu_train_batch_size 12 \
    --learning_rate 3e-5 \
    --num_train_epochs 2.0 \
    --max_seq_length 16 \
    --doc_stride 8 \
    --output_dir /tmp/squad_seq_len_bug
```


Comments:
Great, thanks for letting us know!
GPT2 Tokenizer Special Token ID Bug
Author: sshearing
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2 Tokenizer

The problem arise when using:
my own modified script: The problem arises when I try to add special tokens to the GPT2 tokenizer, specifically a pad token and a sep token.

The tasks I am working on is:
* [ ] my own task or dataset: summarization on the xsum dataset, however, the current bug does not actually affect the model, but the pre-processing.

## To Reproduce
<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

from transformers import AutoTokenizer
encoder = AutoTokenizer.from_pretrained('gpt2')
encoder.add_special_tokens({'pad_token': '[PAD]', 'sep_token': '[SEP]'})
print(encoder.sep_token, encoder.sep_token_id)
print(encoder.pad_token, encoder.pad_token_id)
print('All special tokens:', encoder.all_special_tokens)
print('All special ids:', encoder.all_special_ids)

## Expected behavior

The expected output of this script is below:
[SEP] 50258
[PAD] 50257
All special tokens: ['
', '[PAD]', '[SEP]']
All special ids: [50256, 50257, 50258]

The actual output is:
[SEP] 50258
[PAD] 50257
All special tokens: ['
', '[PAD]', '[SEP]']
All special ids: [50256, 50256, 50256]

As you can see, All special IDS do not match the actual ids of the [SEP] and [PAD] token. Not sure why this is the case. Am I misunderstanding something about how add_special_tokens works?

## Environment

* OS: Ubuntu 18.04
* Python version: 3.7
* PyTorch version: 1.0.0
* PyTorch Transformers version (or branch): 2.1.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information: None

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Hi, there have been a few fixes done on the tokenizers recently that haven't been released on Pypi yet. When I run your script with the library installed from source, I obtain: 

```py
[SEP] 50258
[PAD] 50257
All special tokens: ['[PAD]', '[SEP]', '
']
All special ids: [50257, 50258, 50256]
```

Would you mind installing from source using `pip install git+https://github.com/huggingface/transformers`  and let me know if it fixes your problem,?

That fixed it, thank you!
Some questions about the abstractive summarization code.
Author: linWujl
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, thanks for sharing the code,  i was confused about some code and implements when i reading it.
a) In run_summarization_finetuning.py, 
![image](https://user-images.githubusercontent.com/9161371/69008385-bf6ac500-0984-11ea-8f02-72806bb68675.png)
should cls_token_id be replaced by sep_token_id?
b) In utils_summarization.py
![image](https://user-images.githubusercontent.com/9161371/69008407-17a1c700-0985-11ea-8ff4-a16cee796213.png)
This will make sequence [cls] a b [sep] c d [sep] encoded as 0 0 0 1 1 1 0, should the order be changed?
c) In the decoder, the deocder_mask only mask the pad token, i think look_ahead_mask is more accurate for we can't see the future words in advance.

Looking forward to reply. Thanks!



Comments:
Hi!

First, please check the `run_summarization.py` script in the `example-summarization` branch as it is more up-to-date. To answer your questions:

a) We are following this implementation: https://arxiv.org/pdf/1908.08345.pdf where they add an extra [CLS] token for each new sentence.

b) I would need to triple-check the authors’ code, but I hope it does not matter.

c) Good point. If you look at the way the encoder-decoder is implemented you will see that masks passed to the decoder are automatically turned into causal (“look-ahead”), so the code does have the expected behavior :)

Thank you so much.
Clarifications about the Quick-tour of the fine-tuning scripts?
Author: bulutsuzku
Pull request: no

## ❓ Questions & Help

The [quick tour](https://github.com/huggingface/transformers#quick-tour-of-the-fine-tuningusage-scripts) mentions an example of fine-tuning on different GLUE tasks.

My task/trained model has nothing to do with GLUE tasks, i.e.: I already have a pre-trained  pytorch model file and the formatted corpus files (train.tsv, dev.tsv and test.tsv). 

Is the run_glue.py script meant to be edited by me for doing a new *NON GLUE* fine-tuning? 

Or it is a matter of adjusting parameters on the CLI?

If none of them applies, where I can find an example of fine-tuning with support for "customized" task?




Comments:
Hi, what exactly is your task? Is it question answering, sequence classification, language modeling, other?

> Hi, what exactly is your task? Is it question answering, sequence classification, language modeling, other?

Hi LysandreJik

I have two tasks: First one is sequence classification. Second one is Relation Extraction.


For sequence classification, you can indeed take inspiration from the script `run_glue.py`. Those scripts are meant as examples showcasing how to manage models for different tasks, so that users may train our models in any way they see fit. The models are standard PyTorch models so they can be trained like any other model.

For relation extraction, there are no models specifically targeting this task but feel free to adapt a standard model by adding a few layers on top. For training there are examples in our [HMTL repository](https://github.com/huggingface/hmtl) which may be of help.
run_squad.py predictions
Author: Arjunsankarlal
Pull request: no
Labels: name: wontfix

## ❓ Can anyone explain how start_logit and end_logit value is been determined ? And how it can used to measure the reliability of the answer span
In some issues I saw that if start_logit and end_logit = -1 then it is concluded to be unanswerable question.

There are some cases where I get the

> "1": [
{
"text": "hover your mouse pointer",
"probability": 1.0,
"start_logit": -12.987695693969727,
"end_logit": -12.40383529663086
}
],

In this place, what these values of start_logit and end_logit actually mean ? Since logit is natural log of odds, what is considered here as odds ?

In some cases, if the number of nbest predictions is 1, then [here in utils_squad.py](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_squad.py#L613-L614) the answer span is made `empty`, does it means that it is the right answer ?

            if len(nbest)==1:
                nbest.insert(0,
                    _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))


But in the [same file](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_squad.py#L618-L620), when there is no nbest predictions, again the answer span is made `empty`

        if not nbest:
            nbest.append(
                _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))


For two different situations why the same answer is been set ?  

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


The comments say this situation is when the answer is a single null , but I don't see any conditionals to filter for this. hmmm did you figure it out?
Can the HuggingFace GPT2DoubleHeadsModel either for regular language modelling or solving multiple-choice questions? or is it only for solving multiple-choice questions?
Author: h56cho
Pull request: no

Hello,

According to the HuggingFace Transformer's website (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel), GPT2DoubleHeadsModel is the GPT2 Model transformer with a language modelling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks.

Does this mean that the GPT2DoubleHeadsModel can be used for both the regular language modelling tasks (predicting the next token) and also for solving multiple-choice questions?

Or does this mean that GPT2DoubleHeadsModel can be used to test machines on the multiple-choice type questions only?

Thank you,


Comments:
Is there a straightforward way to classify documents at the sentence level, while using surrounding sentences for context?
Author: pydn
Pull request: no

## ❓ Questions & Help

I'm curious how one would go about classifying documents at the sentence level using the Sequence Classification classes where the classification of a document's sentence would use other sentences in the document for context.

For example, if a document says, "I truly hate this product. Thanks [company name]!" The two sentences should be classified as negative. However, if you classify each sentence separately, "I truly hate this product" would be classified as negative, while "Thanks [company name]!" would likely be classified as positive.

I understand I could classify the document as a whole, but I'm looking for a more granular level of text classification.

Any guidance on making adjustments for this need would be greatly appreciated.

Thanks!


Comments:
Maybe you should start with [XLNet](https://arxiv.org/abs/1906.08237).

@iedmrc Thanks, I'm sure that most of these models could do this. I'm looking for guidance on adjusting the architecture of these transformer models to look for context form surrounding sentences to classify the current sentence. XLNetForSequenceClassification takes in a document as a whole for classification.

I imagine many others have confronted this problem, so I'm reaching out for guidance on where to start.

@pydn You are probably looking for a model like this: https://github.com/allenai/sequential_sentence_classification

@armancohan Thank you! This is exactly what I was looking for.