noamaanMulla-03/transformers-issues huggingface dataset:
Cannot import EvalPrediction from transformers
Author: vanh17
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.  python3 ./examples/text-classification/run_glue.py     --model_name_or_path bert-base-uncased     --task_name $TASK_NAME     --do_train     --do_eval     --data_dir $GLUE_DIR/$TASK_NAME     --max_seq_length 128     --per_device_eval_batch_size=2       --per_device_train_batch_size=2       --learning_rate 2e-5     --num_train_epochs 3.0     --output_dir /tmp/$TASK_NAME/cd ../..

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. --> 
It should run normally but then give error: cannot import module EvalPrediction

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Linux-4.4.0-165-generic-x86_64-with-debian-stretch-sid
- Python version: 3.6.5
- PyTorch version (GPU?): 1.2.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


Comments:
Hi @vanh17 , your transformers version is old, `EvalPrediction` is not available in 2.5.1. You can install transformers from source and then run the examples.
args.output_dir seems like been ignored
Author: yilanchen6
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/transformers/blob/a638e986f45b338c86482e1c13e045c06cfeccad/examples/run_squad.py#L814
It seems that args.output_dir is been ignored in checkpoints when args.eval_all_checkpoints is True.
It probably should be like this: 
global_step = checkpoint.split("-")[-1] if (len(checkpoints) > 1 and checkpoint != args.output_dir) else ""

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to apply Torchtext convenience classes to prepare data for a Transformer?
Author: celsofranssa
Pull request: no
Labels: name: wontfix

Hello,
Reading the tutorial [Language Translation with torchText](https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html) I wondered how someone could use those convenience classes (`Field, BucketIterator`) to train/fine-tune a `Transformer`.

For instance, I'm currently working with a large dataset distributed in jsonl files which looks like:
```python
{ "query": "this is a query 1", "doc": "relevant document regarding query 1" },
{ "query": "this is a query 2", "doc": "relevant document regarding query 2" },
               ...
 
```

Now, to forward this data into a transformer like Bert, it is necessary to convert this dataset into the format:

```python3
(
#queries
 {
    'input_ids': tensor([
        [  101,  2023,  2003,  1037, 23032,  1015,   102,     0],
        [  101,  2023,  2003,  1037, 23032,  1016,   102,     0]]), 
    'attention_mask': tensor([
        [1, 1, 1, 1, 1, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1, 0]])
 }, 

 #docs
 {
    'input_ids': tensor([
        [ 101, 2023, 2003, 2028, 7882, 6254, 4953,  102],
        [ 101, 2023, 2003, 2028, 7882, 6254, 4953,  102]]), 
    'attention_mask': 'input_ids': tensor([
        [1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1]])
}
```
So, what would be a clear and efficient approach to apply those convenience classes to tokenize a text dataset to fit it in the required format of a transformer?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Make the order of additional special tokens deterministic
Author: gonglinyuan
Pull request: yes

In `SpecialTokensMixin.all_special_tokens_extended`, deduplication is performed by `all_toks = list(set(all_toks))`. However, this will change the ordering of additional special tokens, and the order depends on the hash seed of set data structure. This will result in non-deterministic id of additional special tokens added to `AutoTokenizer.from_pretrained` method. Therefore, I changed this problematic line to `all_toks = sorted(list(set(all_toks)))`.

Comments:
Added first description of the model
Author: onepointconsulting
Pull request: yes
Labels: name: model card

Added general description, information about the tags and also some example usage code.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5672?src=pr&el=h1) Report
> Merging [#5672](https://codecov.io/gh/huggingface/transformers/pull/5672?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/201d23f2854c7a13d3c32df4947af9fd7365c2cd&el=desc) will **decrease** coverage by `0.12%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5672/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5672?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5672      +/-   ##
==========================================
- Coverage   77.34%   77.22%   -0.13%     
==========================================
  Files         146      146              
  Lines       25948    25948              
==========================================
- Hits        20070    20038      -32     
- Misses       5878     5910      +32     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5672?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5672/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5672/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.95% <0.00%> (-2.51%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5672/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `93.36% <0.00%> (+49.37%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5672?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5672?src=pr&el=footer). Last update [223084e...4c2abf8](https://codecov.io/gh/huggingface/transformers/pull/5672?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
How to fine-tune tinyBERT for question-asnwering
Author: anirbansaha96
Pull request: no

# ❓ Questions & Help
How do I fine-tune tiny-BERT for question answering on my custom dataset. I've a list of questions and corresponding answers in a csv format, How do I fine-tune tiny-BERT on this dataset. 
[You can find it at StackOverFlow here](https://stackoverflow.com/questions/62710931/huggingface-transformers-model-for-legal-question-answering)




Comments:
Fine Tuning Using /question-answering/run_squad.py
Author: anirbansaha96
Pull request: no

How do I use `/question-answering/run_squad.py` to fine-tune my own custom dataset?  

[StackOverflow][1]

[1]: https://stackoverflow.com/questions/62752709/how-to-fine-tune-bert-for-question-answering

Comments:
Hi, there's a [readme](https://github.com/huggingface/transformers/tree/master/examples/question-answering) in `question-answering`!

Yes, I've gone through this resource. What I want to achieve is to fine-tune on a custom resource and not on SQuAD dataset. I want to train it on Question-Answer pairs.
You can find more about the issue [here](https://ai.stackexchange.com/questions/22358/how-to-fine-tune-bert-for-question-answering/22362?noredirect=1#comment34182_22362).
[Proposal] Small HfAPI Cleanup
Author: sshleifer
Pull request: no
Labels: name: wontfix

- Implement __dict__ and __str__ so that one need not guess the class attributes of `ModelInfo` and `S3Object`
- Delete redundant `S3Obj` class. (`S3Object` seems to be a superset).
- Rename to `HFApi`?

Thoughts @julien-c @LysandreJik ?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
rename the functions to match the rest of the test convention
Author: stas00
Pull request: yes

no functional change

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5692?src=pr&el=h1) Report
> Merging [#5692](https://codecov.io/gh/huggingface/transformers/pull/5692?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0befb513278f6e42b722be340dbc667e0ba2718e&el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5692/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5692?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5692      +/-   ##
==========================================
- Coverage   78.26%   78.09%   -0.17%     
==========================================
  Files         146      146              
  Lines       25998    25998              
==========================================
- Hits        20348    20304      -44     
- Misses       5650     5694      +44     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5692?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5692/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `20.27% <0.00%> (-74.92%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5692/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.70% <0.00%> (-2.51%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5692/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `81.02% <0.00%> (-1.29%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5692/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `81.49% <0.00%> (+0.29%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/5692/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `95.53% <0.00%> (+69.51%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5692?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5692?src=pr&el=footer). Last update [0befb51...921cabc](https://codecov.io/gh/huggingface/transformers/pull/5692?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Is it possible to get document embeddings using GPT-2? If so, how?
Author: youssefavx
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

I posted a question here on SO: https://stackoverflow.com/questions/61641257/how-to-get-document-embeddings-using-gpt-2?noredirect=1#comment109035633_61641257

But unfortunately was informed that "Please notice that SO is not a tutorial service, and also that recommendation requests for external resources are explicitly off-topic."

## Details
I'm curious if using GPT-2 might yield a higher accuracy for document vectors (with greatly varying length) or not (would it surpass the state of the art?)

Really I'm most interested in document embeddings that are as accurate as possible. I'm wondering if using GPT-2 will get results that are more accurate than Paragraph Vectors for example.

I heard that in order to get vectors from GPT-2 "you can use a weighted sum and/or concatenation of vector outputs at its hidden layers (typically the last few hidden layers) as a representation of its corresponding words or even "meaning" of the entire text, although for this role BERT is used more often as it is bi-directional and takes into account of both forward and backward contexts."

As a machine learning and NLP beginner, I'd love to know how to go about this, or to be pointed in the right direction to learn more about how to attempt this in Python.

I've tried fine-tuning GPT-2 before but I have no idea how to extract vectors from it for text.

**A link to original question on Stack Overflow**: https://stackoverflow.com/questions/61641257/how-to-get-document-embeddings-using-gpt-2?noredirect=1#comment109035633_61641257


Comments:
Not from Huggingface, but for this, I would use something like SentenceBERT. As far as I know.

@moinnadeem Thank you for pointing me in that direction! But would sentencebert really work for documents?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to get output from BERTNextSentencePrediction by passing One Sentence and Next Sentence getting predicted
Author: ali4friends71
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
How to get output from BERTNextSentencePrediction by passing One Sentence and Next Sentence getting predicted ?

I got the Prediction score but how and where should I pass the first sentence so I can get the next sentence predicted ?

Codefrom torch.nn.functional import softmax
from transformers import BertForNextSentencePrediction, BertTokenizer


seq_A = 'I like cookies !'
seq_B = 'Do you like them ?'

# load pretrained model and a pretrained tokenizer
model = BertForNextSentencePrediction.from_pretrained('bert-base-cased')
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

# encode the two sequences. Particularly, make clear that they must be 
# encoded as "one" input to the model by using 'seq_B' as the 'text_pair'
encoded = tokenizer.encode_plus(seq_A, text_pair=seq_B, return_tensors='pt')
print(encoded)
# {'input_ids': tensor([[  101,   146,  1176, 18621,   106,   102,  2091,  1128,  1176,  1172, 136,   102]]),
#  'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]]),
#  'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
# NOTE how the token_type_ids are 0 for all tokens in seq_A and 1 for seq_B, 
# this way the model knows which token belongs to which sequence

# a model's output is a tuple, we only need the output tensor containing
# the relationships which is the first item in the tuple
seq_relationship_logits = model(**encoded)[0]

# we still need softmax to convert the logits into probabilities
# index 0: sequence B is a continuation of sequence A
# index 1: sequence B is a random sequence
probs = softmax(seq_relationship_logits, dim=1)

print(seq_relationship_logits)
print(probs)
# tensor([[9.9993e-01, 6.7607e-05]], grad_fn=<SoftmaxBackward>)
# very high value for index 0: high probability of seq_B being a continuation of seq_A

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GPT2Tokenizer.decode() slow for individual tokens
Author: Damiox
Pull request: no
Labels: name: wontfix

`GPT2Tokenizer.decode()` is considerable slow when invoking it for each token individually, not for a set of tokens... I need to pass in a set of tokens and get back a set of strings for each token. I tried convert_ids_to_tokens() but I'm getting results with the Ġ character. Should I simply remove that character from the results?

I also tried `GPT2TokenizerFast`, but it failed with `"encoder() takes from 2 to 3 positional arguments but 6 were given"` when I was trying to use batch_encode_plus() to pad my tokens with the following arguments: tokens (that were already encoded), pad_to_max_length=True and return_tensors='pt' . transformers ver 2.6.0 - is this a bug?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Has anyone successfully used TF2.0 to load pre-trained transformer-XL (wt103) weights and reproduce their sota results
Author: menghuanlater
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I tried to reproduce, but the final result was 35.6 (vs paper 18.3), I carefully checked and found that the matrix multiplication result of TF2.x and TF1.12 (paper use) have some differences, about 0.0001, and then Many layers of linear transformation and multiplication, the difference in output characterization of the last layer has reached about 0.1, which is enough to affect the results, and if the mem_len and tgt_len values in the experiment are swapped, ppl actually drops to 19.5, I can ensure that all loaded The parameters are correct, the code is completely referenced to the author's source code and changed to 2.0 version, so I want to ask if anyone has reproduced it with TF2.0
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Need correction in GPT-2 perplexity equation based on computation
Author: states786
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

I would like to formulate GPT-2 perplexity [calculation  ](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L266)in terms of equation as under:

![image](https://user-images.githubusercontent.com/64096105/81352017-2d8b7b80-90ce-11ea-8cdd-b44fdccf0ea3.png)

![image](https://user-images.githubusercontent.com/64096105/81352211-ab4f8700-90ce-11ea-990d-524c9756addf.png)

I am confused which equation can best describe the process. 

Following are the representation:
1. N is orignal word length
2. M is the BPE word length after tokenization
3. w represents `n_position` size, which is 1024 for GPT-2 pretrained model.
4. i=0 till M

I try to make the first equation, as GPT-2 cannot calculate perplexity in one go, if M is greater than w. It needs to calculate mean of cross-entropy loss.

Can you please share your opinion in formulating an equation?


Comments:
Off-topic, but it's high time that Github supports KaTeX =)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.