noamaanMulla-03/transformers-issues huggingface dataset:
Extending Encoder Decoder to GPT-2
Author: satwikkottur
Pull request: no
Labels: name: wontfix

Adding GPT2 initialization for EncoderDecoder model as pointed out in the issue below.

> Currently, only Bert works as a decoder. We might add GPT2 in a couple of weeks. Note that no model has `cross-attention` layers if it is not already an encoder-decoder model (like Bart or T5) and in this case it does not make sense to use the encoder-decoder wrapper. The model is initialized with random weights for the cross attention layers which will have to be fine-tuned. I agree, that this should be made clearer in the documentation!

_Originally posted by @patrickvonplaten in https://github.com/huggingface/transformers/issues/4517#issuecomment-638058577_

Comments:
It's on the roadmap :-) 

Thank you! Look forward to it :)

Hi - I've actually been working on this myself the past couple days, should I submit a PR when finished? 

That'd be great!

Will do - likely sometime this week.  

@djw1809 Any update on the PR? :)

@patrickvonplaten Hello Patrick, I am watching with much interest EncodeDecoder from transformers :) . Any updates on supporting GPT2 with EncodeDecoder ?

Got sidetracked with other research - coming back to it in several days,
working on my end, just need to play nice with the rest of the repo.

On Tue, Jul 7, 2020 at 3:32 PM Mihai Ilie <notifications@github.com> wrote:

> @patrickvonplaten <https://github.com/patrickvonplaten> Hello Patrick, I
> am watching with much interest EncodeDecoder from transformers :) . Any
> updates on supporting GPT2 with EncodeDecoder ?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/4961#issuecomment-655170674>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AG3PODZXYPBB33F4CBSNZLDR2OO7VANCNFSM4N4QTZQA>
> .
>


-- 
Dylan Weber, Research Assistant | PhD Candidate
School of Math and Statistical Sciences
WXLR642/BYENG593 Arizona State University


@djw1809 - also feel free to already open a PR with unfinished code yet so that I can take a look early on and help you :-) 

Working on it now. Also linking this PR: #4483

@patrickvonplaten  Hello Patrick. 
As I see from https://github.com/huggingface/transformers/commit/1d6e71e1167dea9e026391ec5a1a2d7ec33d22af current cross attention implimentation assume that encoder have same hidden size as GPT-2. I have encoder with hidden size 512 and want to combine it with GPT-2 medium with hidden size 1024. I have done it by Fairseq code and now want to do same by Huggingface. Could you update your solution to support any suitable encoder hidden space size?

Hey @Squire-tomsk, 

I see what you mean - this would mean to add a new config param for each model that has cross-attention...is this common practice? Would be great if you could open a new issue for that :-) 

Done https://github.com/huggingface/transformers/issues/6645

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Tokenize got an unexpected keyword argument 'pad_to_max_length', 'return_attention_mask'
Author: Prithvi103
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

This works fine when I run on my GPU but gives the above error when I try to run on my CPU. They bought have the same environment setup.

Error:
  File "C:\ProgramData\Anaconda3\lib\site-packages\transformers\tokenization_utils.py", line 786, in encode_plus
    first_ids = get_input_ids(text)
  File "C:\ProgramData\Anaconda3\lib\site-packages\transformers\tokenization_utils.py", line 778, in get_input_ids
    return self.convert_tokens_to_ids(self.tokenize(text, **kwargs))
  File "C:\ProgramData\Anaconda3\lib\site-packages\transformers\tokenization_utils.py", line 649, in tokenize
    tokenized_text = split_on_tokens(added_tokens, text)
  File "C:\ProgramData\Anaconda3\lib\site-packages\transformers\tokenization_utils.py", line 646, in split_on_tokens
    else [token] for token in tokenized_text), [])
  File "C:\ProgramData\Anaconda3\lib\site-packages\transformers\tokenization_utils.py", line 646, in <genexpr>
    else [token] for token in tokenized_text), [])
TypeError: _tokenize() got an unexpected keyword argument 'pad_to_max_length'

environment:
Python 3.7
transformers 3.0.2
torch 1.5.1


Comments:
You GPU and CPU env might have different version of transformers. Could you try updating to master ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
bert finetuning for multilingual question answering
Author: haenvely
Pull request: no
Labels: name: wontfix

Hi, I'm just learning BERT now, and wanna use pyTorch and apply BERT for question answering.
I browsed the models, and it seems that run_squad.py is appropriate for question answering task.
But I wanna apply it to other languages' question answering tasks, so what I can do is using run_squad.py and switching the train/dev file paths to other languages datasets?
And what if I wanna apply the task to some small data(written in another language. so it will be actual and specific task), the process will be finetuning the "run_sqaud.py" with that language datasets, and then finetuning it again with the small data?

Thank you,

Comments:
Hi @haenvely , if you have the datset in same format as SQuAD then you can us the `run_squad` script. You can specify train and eval file using `--train_file` and `--predict_file` argument.

But you'll need a model which pre-trained on the language that you want.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
old nlp causes error that pip install -e. can't fix.
Author: sshleifer
Pull request: no
Labels: name: wontfix

```
Traceback (most recent call last):
  File "finetune.py", line 15, in <module>
    from lightning_base import BaseTransformer, add_generic_args, generic_train
  File "/home/shleifer/transformers_fork/examples/lightning_base.py", line 10, in <mo
dule>
    from transformers import (
  File "/home/shleifer/transformers_fork/src/transformers/__init__.py", line 23, in <
module>
    from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
  File "/home/shleifer/transformers_fork/src/transformers/configuration_albert.py", line 18, in <module>
    from .configuration_utils import PretrainedConfig
  File "/home/shleifer/transformers_fork/src/transformers/configuration_utils.py", line 25, in <module>
    from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_url
  File "/home/shleifer/transformers_fork/src/transformers/file_utils.py", line 68, in <module>
    import nlp  # noqa: F401
  File "/home/shleifer/miniconda/envs/torch1.5/lib/python3.8/site-packages/nlp/__init__.py", line 41, in <module>
    raise ImportWarning(
ImportWarning: To use `nlp`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition.
If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.
```

Comments:
Fixed with `pip install nlp --upgrade`.
This problem only happens if you do have nlp installed, but an old version.
we could make assertions about` nlp.__version___` to avoid this.

I face this even now after running pip install on requirements.txt . I just upgrade the version for pyarrow and it works fine then.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Run_glue.py, how can I continue previous fine-tuning training?
Author: lzl19971215
Pull request: no
Labels: name: wontfix

My previous training on the fine-tuning of glue's SST had been shut down, and now I want to continue my training on the base of the latest checkpoint, how can I implement this ?
<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
If you're using the Trainer API, it's all automatic by running the same command as before.

Thanks a lot！

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Added 12 model cards for Indian Language Models
Author: kushalj001
Pull request: yes
Labels: name: model card

# What does this PR do?
This PR adds model cards for 12 language models which have been uploaded to the model hub recently over [here](https://huggingface.co/neuralspace-reverie). These cover 3 Indian languages and for each language there are 4 model variants namely: BERT, DistilBERT, RoBERTa and XLM-R.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->



Comments:
Wow, so cool! Thanks for your contribution.
TypeError: __init__() got an unexpected keyword argument 'vocab_file' in transformers/tokenization_gpt2.py", line 380
Author: memray
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- `tokenizers` version: 0.9.2
- Platform: Linux-3.10.0-1062.4.1.el7.x86_64-x86_64-with-redhat-7.7-Maipo
- Python version: 3.7.6
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help @mfuntowicz 
## Information

Model I am using (Bert, XLNet ...): RoBERTa-base

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
fairseq

## To reproduce

I use the **RobertaTokenizerFast** and it seems an arg name mismatch.
Steps to reproduce the behavior:

1. self.tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base', cache_dir=args.cache_dir)

In transformers.tokenization_gpt2.py L376 it is:
`ByteLevelBPETokenizer(
                vocab_file=vocab_file,
                merges_file=merges_file,
                add_prefix_space=add_prefix_space,
                trim_offsets=trim_offsets,
            )`

But in tokenizers.implementations.ByteLevelBPETokenizer it is expected to be `vocab`.


## Expected behavior
`  File "/zfs1/hdaqing/rum20/kp/fairseq-kpg/fairseq/data/encoders/hf_bpe.py", line 31, in __init__
    self.tokenizer = RobertaTokenizerFast.from_pretrained(args.pretrained_model, cache_dir=args.cache_dir)
  File "/ihome/hdaqing/rum20/anaconda3/envs/kp/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1428, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/ihome/hdaqing/rum20/anaconda3/envs/kp/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1575, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/ihome/hdaqing/rum20/anaconda3/envs/kp/lib/python3.7/site-packages/transformers/tokenization_roberta.py", line 380, in __init__
    **kwargs,
  File "/ihome/hdaqing/rum20/anaconda3/envs/kp/lib/python3.7/site-packages/transformers/tokenization_gpt2.py", line 380, in __init__
    trim_offsets=trim_offsets,
TypeError: __init__() got an unexpected keyword argument 'vocab_file'`


Comments:
same issue

Hello! I think this is due to a mismatch between your `transformers` and `tokenizers` versions. `transformers` version v3.3.1 expects `tokenizers == 0.8.1.rc2`.

If you want to use `tokenizers == 0.9.2` you should work on the current `master` branch or wait for version v3.4.0 which should be released sometimes today.

Thank you! I upgraded both and it works.
doc: fix typo
Author: monperrus
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes a typo


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
[Benchmark]
Author: Debraroberts1975
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!


Comments:
Why is the accuracy rate of pre-trained GPT-2 model only ~26%?
Author: h56cho
Pull request: no

Hello,
I have been trying to analyze HellaSwag dataset with the pre-trained GPT2DoubleHeadsModel. I fine-tuned the model by disabling any change fin weights of the main body (12 layers + embedding layer) while training weights of the multiple-choice head with moderate learning rate.

My understanding is that, since the main body of the model is already pre-trained, I should get a reasonably high accuracy rate for the HellaSwag task as long as I do a good job in training the weights from the multiple-choice head. However, the accuracy rate of the pre-trained GPT2DoubleHeadsModel on the HellaSwag task is only ~26% (although my training loss is only ~1.40).

Why is my accuracy rate so low? is this because I am not fine-tuning the weights of the main body of the model during the training? Any advice would be highly appreciated.

Thank you,

Comments:
Hi, we try to keep the github issues for bugs only. Could you open a thread on the [forum](https://discuss.huggingface.co) instead? Thank you!
[Notebooks] Add new encoder-decoder notebooks
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Adds 2 community notebooks


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
[bart] 2 SinusoidalPositionalEmbedding fixes
Author: stas00
Pull request: yes

This PR:
* `embedding_dim` param for `SinusoidalPositionalEmbedding` can now be odd.
* fixes a bug "RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation" appearing in pytorch-1.8+ (this var requires no grad, so make it so before we do anything grad-related with it).

Fixes: #8021

@sshleifer, @LysandreJik 

Comments: