noamaanMulla-03/transformers-issues huggingface dataset:
Keyword arguments {'add_prefix_space': False} not recognized.
Author: rabeehkarimimahabadi
Pull request: no

Hi
I am getting  this message when running finetune_trainer.py on TPUs any idea how to remove it? thanks 

Comments:
Hi @rabeehkarimimahabadi. Could you please respect the issue template when opening issues? It is necessary for us to help you, otherwise we'll have to ask again and again for you to provide more information.

Please complete the issue template with the information related to your environment, to the command you launched, and the full error with the stack trace. Thanks.

Hi
I am sorry
here is the full command I use , it is running finetune_trainer with T5
model. There is not an error, only it gives this warning. thanks


python xla_spawn.py --num_cores 8 finetune_trainer.py
--learning_rate=3e-5         --n_val 1000  --model_name_or_path t5-small
--data_dir data/wmt_en_de/ --output_dir /home/rabeeh/temp/
--overwrite_output_dir  --tpu_num_cores=8 --max_source_length=64
--max_target_length=64 --per_device_train_batch_size=32
--per_device_eval_batch_size=32 --label_smoothing=0.1 --task="translation"
--logging_steps=200 --eval_steps=500 --num_train_epochs=6 --save_steps=500
--max_source_length=128  --max_target_length=128
--val_max_target_length=128 --test_max_target_length=128



Since you're using the T5 model I believe you can safely ignore that warning, which is useful only for the byte-level BPE tokenizers (think GPT-2, RoBERTa).

Maybe @patil-suraj or @sshleifer know why such a warning is thrown when using T5 with the `Seq2SeqTrainer`

Yeah it's happening in seq2seq dataset, you can safely ignore. Fixed in #8329 

Hi,
thank you. Is this applied to the repo? thanks

On Thu, Nov 5, 2020 at 4:15 PM Sam Shleifer <notifications@github.com>
wrote:

> Yeah it's happening in seq2seq dataset, you can safely ignore. Fixed in
> #8329 <https://github.com/huggingface/transformers/pull/8329>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8322#issuecomment-722441777>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHH2WLTMU6NJ7B2K2Y33SOK6SZANCNFSM4TLKSTWQ>
> .
>


Yes, the fix has just been merged. 

Hi there, this is not fixed for me. and still I am getting it with running T5 using finetune_trainer. 

Have you pulled from `master` to get the updated version?

In any case, as I've said before, this is a warning and not an error. You can safely ignore it.
TFTrainerArguments: ImportError: Method `device` requires PyTorch.
Author: vlreinier
Pull request: no

from transformers import TFTrainer, TFTrainingArguments
training_args = TFTrainingArguments(output_dir='models',   
                                    num_train_epochs=1,             
                                    per_device_train_batch_size=32, 
                                    per_device_eval_batch_size=32,  
                                    warmup_steps=500,               
                                    weight_decay=0.05,              
                                    logging_dir='logs'
                                   )

Above code produces the following error:

ImportError                               Traceback (most recent call last)
<ipython-input-3-017a860e7c76> in <module>
      5                                     warmup_steps=500,
      6                                     weight_decay=0.05,
----> 7                                     logging_dir='logs'
      8                                    )

<string> in __init__(self, output_dir, overwrite_output_dir, do_train, do_eval, do_predict, evaluate_during_training, evaluation_strategy, prediction_loss_only, per_device_train_batch_size, per_device_eval_batch_size, per_gpu_train_batch_size, per_gpu_eval_batch_size, gradient_accumulation_steps, eval_accumulation_steps, learning_rate, weight_decay, adam_beta1, adam_beta2, adam_epsilon, max_grad_norm, num_train_epochs, max_steps, warmup_steps, logging_dir, logging_first_step, logging_steps, save_steps, save_total_limit, no_cuda, seed, fp16, fp16_opt_level, local_rank, tpu_num_cores, tpu_metrics_debug, debug, dataloader_drop_last, eval_steps, dataloader_num_workers, past_index, run_name, disable_tqdm, remove_unused_columns, label_names, load_best_model_at_end, metric_for_best_model, greater_is_better, tpu_name, poly_power, xla)

/usr/local/lib/python3.6/dist-packages/transformers/training_args.py in __post_init__(self)
    352             self.run_name = self.output_dir
    353 
--> 354         if self.device.type != "cuda" and self.fp16:
    355             raise ValueError("AMP (`--fp16`) can only be used on CUDA devices.")
    356 

/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py in wrapper(*args, **kwargs)
   1171             return func(*args, **kwargs)
   1172         else:
-> 1173             raise ImportError(f"Method `{func.__name__}` requires PyTorch.")
   1174 
   1175     return wrapper

ImportError: Method `device` requires PyTorch.

Seems like a bug?

Comments:
Indeed. It has been fixed on master and will be in the next release (early next week). In the meantime you can install from source to have the fix.

Ah ok, thanks for the quick response 👍
Update README.md
Author: hassoudi
Pull request: yes
Labels: name: model card

Fix websitr address

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Comments:
Some added tests for TokenClassificationArgumentHandler
Author: Narsil
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Comments:
Question: Differentiable Search in generate
Author: Palipoor
Pull request: no
Labels: name: wontfix

Hi,
This might be a very dumb question to ask, but is there a way that we can have differentiable search and pickings in the "generate" method in models like T5 and GPT-2? If not, is there an example of using the pure real-valued model output in a task? Or an article giving some intuition on what the real-valued output means?
I'd really appreciate your help.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How can I fine-tune on custom model
Author: pinedbean
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

I would like to use TFBert as encoder 
then I will add additional layer on-top of it (with custom model class)

So, I would like to fine-tune all layer down to encoder layer 

Specifically, I would like to do BERT-BiLSTM-CRF for NER task 

Is there a way to do it? 

Thank you for your answer


<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

Comments:
You can try this：
https://github.com/huggingface/transformers/pull/3009/commits/489dd7608c5b3d4acaf997a2b4fbccc3d7144cf3

but there is no bilstm layer. You can add it by yourself

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Distributed training doesn’t work
Author: mcloarec001
Pull request: no
Labels: name: wontfix

Even if CUDA detects all the GPU of the machine, there is no distributed training : 

`RuntimeError: CUDA out of memory. Tried to allocate 3.82 GiB (GPU 0; 11.17 GiB total capacity; 7.59 GiB already allocated; 594.31 MiB free; 10.28 GiB reserved in total by PyTorch) (malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:289)`

The training is launched only on the first GPU.
I’m using language modeling code. I tried to set local_rank at 0 and to use the Torch environment variables MASTER_ADDR, MASTER_PORT, WORLD_SIZE and RANK, but without success. Is there another way to do distributed training with transformers ? 


Comments:
Hi! What are you using to launch your ditributed training? What script are you using? Could you show the command you used, and could you paste your environment information as required by the template? Thank you.

## Environment info
- `transformers` version: 2.11.0
- Platform: Databricks
- Python version: 3.6.9
- PyTorch version: 1.3.1 
- Tensorflow version : 2.0
- Using GPU in script: yes
- Using distributed or parallel set-up in script: try to


## Information

I am using the model xlm-roberta-base. The tasks I am working on is a further train on my own dataset.

The problem arises when using the script examples/language_modeling/run_language_modeling.py with the following command :

```
python transformers/examples/language-modeling/run_language_modeling.py 
    --model_type xlm-roberta 
    --model_name_or_path xlm-roberta-base 
    --train_data_file data/processed/piaf.txt 
    --output_dir ./output --learning_rate 0.1 
    --per_gpu_train_batch_size 2 
    --local_rank 0 
    --num_train_epochs 1 
    --do_train 
    --mlm
```

Okay, could you try using the `torch.distributed.launch` utility used to launch a distributed training?

Your command, given that you have 8 GPUs, would be:

```
python -m torch.distributed.launch \
    --nproc_per_node 8 transformers/examples/language-modeling/run_language_modeling.py 
    --model_type xlm-roberta 
    --model_name_or_path xlm-roberta-base 
    --train_data_file data/processed/piaf.txt 
    --output_dir ./output --learning_rate 0.1 
    --per_gpu_train_batch_size 2 
    --local_rank 0 
    --num_train_epochs 1 
    --do_train 
    --mlm
```

Feel free to modify this part: `--nproc_per_node 8` according to the number of GPUs.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Roberta Large not working with SNLI dataset always gives random baseline that is 33 percent accuracy.
Author: bhavdeep98
Pull request: no
Labels: name: wontfix

I have been working with parts of SNLI dataset, even after hours and hours of pretraining the model always gives 33.24 percent accuracy no matter the amount of the data used in pretraining. This issue is fixed if I revert my transformers version to 2.11.0 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bertology-like Analysis for BART, T5?
Author: gwenger4
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

In my current project, I am working on training encoder-decoder models (BART, T5, etc.) and the Transformers library has been absolutely invaluable! After seeing several Bertology analyses (i.e. looking at the information the model's attention mechanism learns to attend to), I would like to know if a similar analysis is possible with the BART and T5 models in the Hugging Face library. Any recommendations are certainly appreciated!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error in run_ner.py
Author: moniquebm
Pull request: no
Labels: name: wontfix

## Environment info

- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.3
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
 
@stefan-it 

## Information

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: Rare Entities task: WNUT’17 (English NER) dataset
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

When I try to execute run_ner.py, the following error occurs:

08/27/2020 16:10:10 - INFO - filelock -   Lock 2174122787120 acquired on ./data_wnut_17\cached_dev_BertTokenizer_128.lock
08/27/2020 16:10:24 - INFO - utils_ner -   Creating features from dataset file at ./data_wnut_17
08/27/2020 16:12:04 - INFO - filelock -   Lock 2174122787120 released on ./data_wnut_17\cached_dev_BertTokenizer_128.lock
Traceback (most recent call last):
  File "C:\Users\Monique\Documents\IA\NLP\transformers\examples\token-classification\utils_ner.py", line 247, in __init__
    examples = token_classification_task.read_examples_from_file(data_dir, mode)
  File "C:\Users\Monique\Documents\IA\NLP\transformers\examples\token-classification\tasks.py", line 27, in read_examples_from_file
    for line in f:
  File "C:\ProgramData\Anaconda3\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 46: invalid start byte

My only local change was the following in preprocess.py:

`with open(dataset, "rt", encoding='utf-8') as f_p:`

(I added encoding='utf-8' due to a similar error in preprocessing).

## Expected behavior

The dataset should have been created without encoding errors.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How can I get rid of this message every time I run GPT2?
Author: BigSalmon2
Pull request: no
Labels: name: wontfix

```
Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Character-level tokenization?
Author: gwenger4
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

In a project I'm using the Hugging Face library for, I would like to use transformers to perform arithmetic (framed as a text-to-text problem). However, it seems that the current method of tokenization adopted by most existing models in the Hugging Face library (BPE, WordPiece, etc.) are not character-level, and thereby, would not allow for the network to properly process the input for later arithmetic. Are there any ways to get this sort of analysis to work properly with the existing models in Hugging Face (and their accompanying tokenizers), otherwise, what recommendations do you have?

For reference, I am currently using the BART model from Hugging Face.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
What is the size of the context window in the 'openai-gpt' pre-trained model?
Author: lzl19971215
Pull request: no
Labels: name: wontfix

What is the size of the context window in the 'openai-gpt' pre-trained model?
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Maybe global step should be initialized to 0
Author: HuipengXu
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2

in script：https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py
location： line 143

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.