noamaanMulla-03/transformers-issues huggingface dataset:
Fix importing transformers on Windows - SIGKILL not defined
Author: mfuntowicz
Pull request: yes

On Windows platform there is no such `signal.SIGKILL`, we need to send `CTRL_C_EVENT` which is `Ctrl+C`. 

This PR aims at: 

1. Not importing the `signal.SIGKILL` on Windows platforms as it's undefined
2. Using the right `signal.CTRL_C_EVENT` on Windows platforms and `signal.SIGKILL` on Unix*.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4997?src=pr&el=h1) Report
> Merging [#4997](https://codecov.io/gh/huggingface/transformers/pull/4997?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9931f817b75ecb2c8bb08b6e9d4cbec4b0933935&el=desc) will **increase** coverage by `0.72%`.
> The diff coverage is `66.66%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4997/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4997?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4997      +/-   ##
==========================================
+ Coverage   76.89%   77.61%   +0.72%     
==========================================
  Files         128      128              
  Lines       21854    21856       +2     
==========================================
+ Hits        16804    16964     +160     
+ Misses       5050     4892     -158     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4997?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/benchmark/benchmark\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4997/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrX3V0aWxzLnB5) | `72.96% <66.66%> (-0.14%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4997/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.71% <0.00%> (-1.56%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/4997/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG0ucHk=) | `75.85% <0.00%> (+19.75%)` | :arrow_up: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4997/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `56.68% <0.00%> (+28.02%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4997?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4997?src=pr&el=footer). Last update [9931f81...d6f30c5](https://codecov.io/gh/huggingface/transformers/pull/4997?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks @mfuntowicz
Add position_ids in TFElectra models docstring
Author: sgugger
Pull request: yes

Just a small thing, but it was forgotten when adding those I guess.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5021?src=pr&el=h1) Report
> Merging [#5021](https://codecov.io/gh/huggingface/transformers/pull/5021?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1affde2f10c653e36601dd7a3e6a2525ae7ced57&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5021/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5021?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5021   +/-   ##
=======================================
  Coverage   77.26%   77.27%           
=======================================
  Files         128      128           
  Lines       21847    21847           
=======================================
+ Hits        16880    16882    +2     
+ Misses       4967     4965    -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5021?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/5021/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `91.28% <ø> (ø)` | |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5021/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `76.35% <0.00%> (-2.30%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5021/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.20% <0.00%> (-0.24%)` | :arrow_down: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/5021/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `34.07% <0.00%> (+5.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5021?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5021?src=pr&el=footer). Last update [1affde2...7ffe4b3](https://codecov.io/gh/huggingface/transformers/pull/5021?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
How can I print output attention on test data?
Author: laetokang
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I fine-tuned using BART model to carry out Summarization task.
I want to print output attention scores about test data, but I don't know which layer I should change "output_attention" be True.

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
You should do it on the forward function of the model your are using so probably:
```model = BartForConditionalGeneration.from_pretrained("bart-large")
   model(input_ids, output_attentions=True)
```
Fast Tokenizers: `batch_encode_plus` error
Author: stefan-it
Pull request: no
Labels: name: Core: Tokenization

name: Fast Tokenizers

Hi,

I just want report the following error using fast tokenizers for better tracking.

The combination of:

* `batch_encode_plus` method
* `is_pretokenized=True` and
* `return_tensors=True`

returns the following error message:

```bash
~/.venvs/flair-2/lib/python3.8/site-packages/transformers/tokenization_utils.py in batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, max_length, stride, truncation_strategy, pad_to_max_length, is_pretokenized, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_lengths, **kwargs)
   2501                 stack = tf.stack(stack, axis=0)
   2502             elif return_tensors == "pt":
-> 2503                 stack = torch.stack(stack, dim=0)
   2504             # elif not return_tensors and len(stack) == 1:
   2505             #     stack = stack[0]

RuntimeError: stack expects each tensor to be equal size, but got [34] at entry 0 and [8] at entry 1
```

The code for re-producing this error is:

```python
from transformers import BertTokenizerFast

model_name = "bert-base-cased"
tokenizer = BertTokenizerFast.from_pretrained(model_name)

sentences = ["Schloß Nymphenburg is a nice castle in Munich".split(), 
             "Berlin and Munich are cool .".split()]

output = tokenizer.batch_encode_plus(batch_text_or_text_pairs=sentences,
                                     is_pretokenized=True,
                                     return_tensors="pt",
                                    )
```

Notice: This bug only occurs, when input sentences have different lenghts and the `return_tensors="pt"` is used!

----

Versions, that I've installed:

* Latest Transformers version from `master`, 4e817ff41885063e08bb3bcd63e5adfd835b9911
* Tokenizers in version *0.7.0*
* PyTorch *1.5.0*

Comments:
This issue should be submitted in another repo.
https://github.com/huggingface/tokenizers.
Cannot save and load pretrained MarianTokenizer
Author: erikchwang
Pull request: no

# 🐛 Bug

## Information

I want to save a pretrained MarianTokenizer to a local directory ("my_dir") and then load it:
> import transformers
> transformers.AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de").save_pretrained("my_dir")
> tokenizer = transformers.AutoTokenizer.from_pretrained("my_dir")

But the above code failed: 

> Traceback (most recent call last):
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_utils.py", line 239, in get_config_dict
>     local_files_only=local_files_only,
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/file_utils.py", line 267, in cached_path
>     raise EnvironmentError("file {} not found".format(url_or_filename))
> OSError: file my_dir/config.json not found
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "<input>", line 1, in <module>
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 195, in from_pretrained
>     config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_auto.py", line 196, in from_pretrained
>     config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_utils.py", line 252, in get_config_dict
>     raise EnvironmentError(msg)
> OSError: Can't load config for 'my_dir'. Make sure that:
> - 'my_dir' is a correct model identifier listed on 'https://huggingface.co/models'
> - or 'my_dir' is the correct path to a directory containing a config.json file
> 

Comments:
 #4371
The correct way to save and load pretrained MarianTokenizer?
Author: erikchwang
Pull request: no

I want to save a pretrained MarianTokenizer to a local directory ("my_dir") and then load it:
> import transformers
> transformers.AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de").save_pretrained("my_dir")
> tokenizer = transformers.AutoTokenizer.from_pretrained("my_dir")

But the above code failed: 

> Traceback (most recent call last):
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_utils.py", line 239, in get_config_dict
>     local_files_only=local_files_only,
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/file_utils.py", line 267, in cached_path
>     raise EnvironmentError("file {} not found".format(url_or_filename))
> OSError: file my_dir/config.json not found
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "<input>", line 1, in <module>
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 195, in from_pretrained
>     config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_auto.py", line 196, in from_pretrained
>     config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_utils.py", line 252, in get_config_dict
>     raise EnvironmentError(msg)
> OSError: Can't load config for 'my_dir'. Make sure that:
> - 'my_dir' is a correct model identifier listed on 'https://huggingface.co/models'
> - or 'my_dir' is the correct path to a directory containing a config.json file
> 

Comments:
 #4371
Save pretrained MarianTokenizer
Author: NonaryR
Pull request: no
Labels: name: marian

Hello! 
I can't save pretrained MarianTokenizer, it's raising NotImplementedError as a subclass of PretrainedTokenizer.


Comments:
Also, how can I transfer to cuda `BatchEncoding` object which return after call `prepare_translation_batch` method on `MarianTokenizer`

This could be fixed by specifiying a `save_vocabulary` function, like it is used in:

https://github.com/huggingface/transformers/blob/7defc6670fa76e857109e1b99f3e919da8d11f42/src/transformers/tokenization_xlm_roberta.py#L292-L311

(both source and target vocab needs to be written),

Additionally, it would be awesome if the `MarianTokenizer` class would be `pickle`-able using `__getstate__` and `__setstate__` methods :)

/cc @sshleifer 

@NonaryR `BatchEncoding.to(device:str)` should work.
Will fix vocab saving, great catch!


Thank you,  `.to(device)` helped!
I can close an issue if you want it. 
Also, many thanks for this new release with all these new awesome translation models, they will bring significant changes to the community!!

But how can you load the saved MarianTokenizer from your local directory? 
Can anyone give a complete example about how to save and load MarianTokenizer?

Can anyone help with this issue: #5040 ?
[MarianTokenizer] implement save_vocabulary and other common methods
Author: sshleifer
Pull request: yes
Labels: name: marian

- adds `test_tokenization_marian.py` which runs the common tokenizer tests.
- adds `save_vocabulary` and other methods to `MarianTokenizer` to make the common tests pass.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4389?src=pr&el=h1) Report
> Merging [#4389](https://codecov.io/gh/huggingface/transformers/pull/4389?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/48c3a70b4eaedab1dd9ad49990cfaa4d6cb8f6a0&el=desc) will **decrease** coverage by `0.37%`.
> The diff coverage is `96.22%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4389/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4389?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4389      +/-   ##
==========================================
- Coverage   78.41%   78.03%   -0.38%     
==========================================
  Files         123      123              
  Lines       20432    20477      +45     
==========================================
- Hits        16021    15980      -41     
- Misses       4411     4497      +86     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4389?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/4389/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `93.10% <96.22%> (+5.77%)` | :arrow_up: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4389/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `28.66% <0.00%> (-28.03%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4389/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4389?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4389?src=pr&el=footer). Last update [48c3a70...1bd2df5](https://codecov.io/gh/huggingface/transformers/pull/4389?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Can anyone help with this issue: #5040 ?
Wrong model or tokenizer for MarianMT
Author: NonaryR
Pull request: no

While we [can't save](https://github.com/huggingface/transformers/issues/4371) `MarianTokenizer` to local directory, I found model weights and configs on [this page](https://huggingface.co/Helsinki-NLP/opus-mt-en-ROMANCE), in `List all files in model ` link.
I downloaded  these files, but I think this is wrong configs, because I can't reproduce even simplest example from [here](https://huggingface.co/transformers/model_doc/marian.html)
This text
```
1)  '>>fr<< this is a sentence in english that we want to translate to french',
2) '>>pt<< This should go to portuguese',
3) '>>es<< And this to Spanish'
should became
1) "c'est une phrase en anglais que nous voulons traduire en français",
2) 'Isto deve ir para o português.',
3) 'Y esto al español'
```
with model and configs downloaded from the link above produces these results:
```
1) "c'est une phrase en anglais que nous voulons traduire en français" (as expected)
2) "Questo deve ir in portughese"
3) "E questo a spagnol"
```
This is definitely wrong, can you help?

Comments:
Never mind, my mistake

What was your mistake?

Is your system Windows?
I am trying to reproduce the colab tutorial from https://blogs.helsinki.fi/language-technology/2020/05/14/helsinkinlp-in-huggingface/ on Windows but I get errors.


@R4ZZ3 Hello! I just appended `>>tag<<` for text in cycle, and my text quickly became a mess of tags. And no Windows, sorry, can't help.

Can anyone help with this issue: #5040 ?
Finetuning GPT-2 
Author: sadhikamalladi
Pull request: no
Labels: name: wontfix

It looks like there used to be a script `run_lm_finetuning.py` that has been replaced by `run_language_modeling.py`. It's unclear to me how to use this script to run finetuning. I want to finetune GPT-2 on a variety of downstream tasks, and would love some help!

Comments:
You can use `run_language_modeling.py` for this purpose. Just set `--model_type` to `gpt2`, set `--model_name_or_path` to the gpt2 model checkpoint you want (`gpt2`) and set `--train_data_file` to your dataset and you should be ready to go.

Thanks! The issue is that I want to use the pre-trained version of GPT-2. I remember `run_lm_finetuning` had a few lines of code where it would download and load that pre-trained model.

You can specify `gpt2` in `--model_name_or_path`. That corresponds to one of the pre-trained checkpoints that it'll download and use. The other possible pre-trained models that you can specify there are `gpt2-medium`,`gpt2-large`,`gpt2-xl` and `distilgpt2`.

If specifying `gpt2` there downloads the checkpoints then how do you train from scratch? I've been specifying that parameter and it seems like it is training from scratch (starting perplexity ~1000).

I believe that if you want to train from scratch, you'll have to point that to a folder with a config file (with the parameters of the model) and no `pytorch_model.bin` checkpoint file in that folder.

@Genius1237 Hi there, 
I am finetuning the 124M model based on my dataset (almost 2mb) and I am using Colab by Max Wolf. I am wondering if there is a way that I can generate texts based on my trained model + internet context (not only metadata). I wanna generate some notes regarding current issues (such as COVID-19) based on my trained model. 
Could you help me with that, please? thanks.
 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
inconsistent tokenize output
Author: michaelmoju
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): Chinese

The problem arises when using: BertTokenizer

I am using the BertModel. When predicting the result, it got inconsistent output (the output differs from time to time). It turns out that the tokenizer gets different output when the input string is "\n" or "\n\n". 

for example: 
<pre><code> 
from transformers.tokenization_bert import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
tokens = tokenizer.tokenize("\n\n")
output: ['[SEP]'] or ['[PAD]'] or ['[CLS]']
</code></pre>



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.