noamaanMulla-03/transformers-issues huggingface dataset:
test_run_eval_search SLOW failure
Author: sshleifer
Pull request: no

```bash
RUN_SLOW=1 pytest examples/seq2seq/test_seq2seq_examples.py -k search
```


```
FAILED examples/seq2seq/test_seq2seq_examples.py::test_run_eval_search[patrickvonplaten/t5-tiny-random] - ValueError: could not convert string to float: '--data_dir'
FAILED examples/seq2seq/test_seq2seq_examples.py::test_run_eval_search[sshleifer/tiny-mbart] - ValueError: could not convert string to float: '--data_dir'
```

Investigating

Comments:
@stas00 
Definitely need this change
![image](https://user-images.githubusercontent.com/6045025/93801070-753f1980-fc0f-11ea-83c6-b544fa4042cf.png)


But then the parsing of the search arg seems to break.
Could you take a look?

oh, yes, thank you - will fix shortly

Fixed: https://github.com/huggingface/transformers/pull/7297

Apologies for pushing in a broken test :( I think I was testing the wrong branch

No worries!
test fsmt finetuning
Author: sshleifer
Pull request: no
Labels: name: fsmt

Would be interested to know whether this works with finetune.py.
Just add your tiny guy to 
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/test_seq2seq_examples.py#L378
and edit l382 to set task='translation'.

If it doesn't pass, we can investigate, but I expect it to pass.

Happy for either of us to take this one @stas00 

(this is behind tiny fsmt in the queue, but writing it down so I don't forget.)

Comments:
I did a quick:

```
def test_finetune():
    model = "stas/tiny-wmt19-en-de"
    task = "translation" 
    args_d: dict = CHEAP_ARGS.copy()
    args_d["label_smoothing"] = 0.1 if task == "translation" else 0
    [...]
```
getting:
```
       torch.nn.modules.module.ModuleAttributeError: 'FSMTModel' object has no attribute 'shared'
/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/modules/module.py:778: ModuleAttributeError
```
`fsmt` doesn't have `.shared` as most of the models aren't shared embeds and dicts. So it won't work as is. Will need to do some tweaking.

but please assign this to me, as I need to learn these parts.

Done: https://github.com/huggingface/transformers/pull/7263
Error importing MBart from transformers
Author: MemduhG
Pull request: no

## Environment info
     
- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-47-generic-x86_64-with-glibc2.29
- Python version: 3.8.2
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 Bart: @sshleifer

## Information

Model I am using: MBart, Bart has the same problem

The problem arises when using:
* [X] the official example scripts: Just copied it from the documentation here: https://huggingface.co/transformers/master/model_doc/mbart.html

The tasks I am working on is:
* [X] my own task or dataset

## To reproduce

Steps to reproduce the behavior:

1. `from transformers import MBartForConditionalGeneration, MBartTokenizer`


## Expected behavior
I would expect MBart to be imported so I could use it, but I get this error instead:

```
ImportError                               Traceback (most recent call last)
<ipython-input-23-20acbef89a41> in <module>
----> 1 from transformers import MBartForConditionalGeneration, MBartTokenizer

ImportError: cannot import name 'MBartForConditionalGeneration' from 'transformers' (/home/memduh/hf/venv/lib/python3.8/site-packages/transformers/__init__.py)
```



Comments:
Note: I created a new venv and installed torch before transformers. That way I'm not getting this problem. When I had this problem I had installed transformers first, and I assume it tries to use tensorflow by default when you do that.

It assumes that you have already installed one of them, your fix is 👍
Wrong arg order for `nested_xla_mesh_reduce` in trainer.py
Author: allenwang28
Pull request: no

## Environment info
Python 3.7 on Google Cloud TPUs

### Who can help
@sgugger 


## Information
When training examples from [run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_glue.py) using Cloud TPUs, we run into this error:
```
TypeError: _xla_rendezvous(): incompatible function arguments. The following argument types are supported:

1. (arg0: int, arg1: str, arg2: str, arg3: List[int]) -> List[bytes]
```

This issue is just due to the wrong arg order in [this line](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L1337) where the [args should be switched](https://github.com/huggingface/transformers/blob/656c27c3a3345d0d2cf31c16f780b573c3dea09a/src/transformers/trainer_utils.py#L162).

This was introduced in:  https://github.com/huggingface/transformers/commit/492bb6aa486856f8243dfeb533ed1b23e996e403 

## To reproduce

Steps to reproduce the behavior:

1. Running the provided example on Cloud TPU.

## Expected behavior
Should not fail with `TypeError`.

Comments:
Indeed, I switched the args, sorry about that. Will make a PR to fix this tomorrow morning.
Fix the tests for Electra
Author: jplu
Pull request: yes

Add the tests for `TFElectraForSequenceClassification` and `TFElectraForMultipleChoice` and fix them.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6284?src=pr&el=h1) Report
> Merging [#6284](https://codecov.io/gh/huggingface/transformers/pull/6284?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c67d1a0259cbb3aef31952b4f37d4fee0e36f134&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6284/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6284?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6284      +/-   ##
==========================================
- Coverage   79.19%   79.18%   -0.01%     
==========================================
  Files         147      147              
  Lines       27120    27120              
==========================================
- Hits        21478    21476       -2     
- Misses       5642     5644       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6284?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/6284/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbGVjdHJhLnB5) | `81.55% <100.00%> (ø)` | |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6284/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `81.70% <0.00%> (-5.02%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/6284/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `94.24% <0.00%> (+4.71%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6284?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6284?src=pr&el=footer). Last update [c67d1a0...1c5c550](https://codecov.io/gh/huggingface/transformers/pull/6284?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I would like to use `TFElectraForSequenceClassification`

I assume I would need to install from source.
Any info on when the next release to pypi will be ? 

Thanks!

@nemani It will be available in the next few days
Fix TF CTRL model naming
Author: jplu
Pull request: yes

This PR fixes an issue with the naming of some layers in the TensorFlow version of CTRL.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6134?src=pr&el=h1) Report
> Merging [#6134](https://codecov.io/gh/huggingface/transformers/pull/6134?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/641b873c1341f553b40fd82c990b80884b585f0b&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6134/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6134?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6134      +/-   ##
==========================================
- Coverage   78.64%   78.63%   -0.01%     
==========================================
  Files         146      146              
  Lines       26326    26333       +7     
==========================================
+ Hits        20704    20708       +4     
- Misses       5622     5625       +3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6134?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/6134/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9jdHJsLnB5) | `97.84% <100.00%> (+0.05%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6134/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.20% <0.00%> (-2.51%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6134/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `83.44% <0.00%> (-0.65%)` | :arrow_down: |
| [src/transformers/tokenization\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/6134/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtX3JvYmVydGEucHk=) | `95.23% <0.00%> (+10.71%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6134?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6134?src=pr&el=footer). Last update [641b873...1bc31c0](https://codecov.io/gh/huggingface/transformers/pull/6134?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Text generation with xlnet
Author: matin-amiri
Pull request: no

how to use xlnet for text generation?




Comments:
```python 
from transformers import pipeline
xlnet_generator = pipeline("text-generation", model="xlnet-base-cased", tokenizer="xlnet-base-cased")
print(xlnet_generator("Today is a nice day and"))
```

Also this should help: https://huggingface.co/transformers/task_summary.html#text-generation
Add num workers cli arg
Author: chadykamar
Pull request: yes

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6316 


Comments:
I've addressed the comments regarding the docstring and help message. 

I'm a little less familiar with TensorFlow 2.0, but it seems like any preprocessing is done by the user before passing `train_dataset` and `eval_dataset` to the `TFTrainer` so there isn't an opportunity to set `num_parallel_calls` (I wasn't able to find any calls to `map` or `interleave` save for some markdown examples).

Indeed there is no call directly in the trainer and these functions have to be run directly in the example script. Nevertheless, this parameter is still useful as it can be used directly in the example script, instead of updating it manually.
Dataloader number of workers in Trainer
Author: ggaemo
Pull request: no

https://github.com/huggingface/transformers/blob/175cd45e13b2e33d1efec9e2ac217cba99f6ae58/src/transformers/trainer.py#L252


If you want to use the Trainer from trainer.py, you only have the option to use only 0 number of workers for your dataloader.

However, even if I change the source code to have 10 number or workers for the data loader, the model still uses the same thread.



Comments:
+1.

I suppose it should not be difficult to add additional argument to ```self.args``` specifying number of workers to use and to use it as:
```
return DataLoader(
            self.train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.num_workers
        )
```

Is there any particular reason why this was not yet implemented? 

+1

+1

No particular reason why it was not implemented, would welcome a PR!
Question about model configuration
Author: dmortem
Pull request: no

# ❓ Questions & Help

Hi,

Now I am trying gpt-2 model to generate sentences. I have a question about the model configuration. 

I load model using [this line in the provided code](https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py#L221). I found "eos_tokens_ids" and "pad_token_id" both equal to 0. But when I checked the tokenizer, I found the token with id 0 is '!'. So when I use the provided code to generate sentences, sentences always end with '!'. Is there any mistake in the vocabulary or the configuration? Many thanks.

Comments:
```python
from transformers import GPT2Config
config = GPT2Config()
print("Pad token id", config.pad_token_id) # should give `None`
print("EOS token id", config.eos_token_id)  # should give `50256`
```

=> So it seems like your config is wrong. Hope this helps!

hi @patrickvonplaten 
I found I used transformer version 2.4.1, after I installed 2.8.0, the output is the same as yours. Also, I found the generate function used "eos_token_ids" instead of. "eos_token_id" as the stop token in 2.4.1. "Eos_token_ids" is not initialized in the configuration and it is 0 by default. Now in 2.8.0, the generation function don't use "Eos_token_ids" any longer.

Thank you!
Trainer Evaluates at each step (Not of epoch end) , indentation bug
Author: WissamAntoun
Pull request: no

## Environment info
   
- `transformers` version: 3.2.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: NA

### Who can help
 
 Trainer: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
- [ ] the official example scripts:
- [x] my own modified scripts:

The tasks I am working on is:
- [ ] an official GLUE/SQUaD task:
- [x] my own task or dataset: 

Basic Single Sentence Classification Dataset loaded via a Dataset class

## To reproduce

Steps to reproduce the behavior:

1. Use the trainer training function with `training_args.evaluation_strategy = EvaluationStrategy.EPOCH`


## Expected behavior

Evaluation should happen after each epoch ends, but instead it happens after each step (batch).

Indentation bug

## Suggested Fix


Move the if condition on line https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L829 to before line https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L834 and remove 1 indentation block


Comments:
Clean up `benchmark_args_utils.py` "no_..." arguments
Author: patrickvonplaten
Pull request: no
Labels: name: Good First Issue

# 🚀 Feature request

Currently we have a mixture of negative and positive formulated arguments, *e.g.* `no_cuda` and `training` here: https://github.com/huggingface/transformers/blob/0054a48cdd64e7309184a64b399ab2c58d75d4e5/src/transformers/benchmark/benchmark_args_utils.py#L61. 

We should change all arguments to be positively formulated, *e.g. from `no_cuda` to `cuda`. These arguments should then change their default value from `False` to `True`.

Also the help text should be updated to something that is better formulated: "Don't ...." as a help text is not very easy to understand.

The motivation is clear: It's better to be consistent in a library and have the code as easy and intuitive to understand.

## Your contribution

This is a "good first issue", so I'm happy to help anybody who wants to take a shot at this :-) 


Comments:
Hmm I seem to have forgotten to change the references in the tests
AttributeError: 'TFTrainingArguments' object has no attribute 'args'
Author: ZeroCool2u
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.2.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 -->
 Trainer: @sgugger 
 tensorflow: @jplu 


## Information

Model I am using (Bert, XLNet ...): `distilbert-base-uncased`

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
    IMDB Sequence Classification

## To reproduce

Steps to reproduce the behavior:

1. Follow the [fine-tuning tutorial here and use TensorFlow](https://huggingface.co/transformers/master/custom_datasets.html#fine-tuning-with-trainer)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```python
AttributeError                            Traceback (most recent call last)

<ipython-input-10-c5306faf2c2f> in <module>()
     12 )
     13 
---> 14 with training_args.strategy.scope():
     15     model = TFDistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased")
     16 

4 frames

/usr/local/lib/python3.6/dist-packages/transformers/training_args_tf.py in _setup_strategy(self)
    120         logger.info("Tensorflow: setting up strategy")
    121 
--> 122         if self.args.xla:
    123             tf.config.optimizer.set_jit(True)
    124 

AttributeError: 'TFTrainingArguments' object has no attribute 'args'
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
[WIP - Benchmark] Add generate function
Author: patrickvonplaten
Pull request: yes
Labels: name: wontfix

This PR adds the `generate` function to Benchmarks. You can try it out by running:

```
python examples/benchmarking/run_benchmark.py --models gpt2 --batch_sizes 1 --sequence_lengths 20 --no_inference --generate
```

## TODO
- [ ] Add for TF as well

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
encode_plus wrongly tokenizing a symbol
Author: IbtihalFerwana
Pull request: no
Labels: name: wontfix

When I used a the bio dataset (linnaeus-IOB) to train bert on, the dataset includes the symbol (>=)  [greater than or equal], the tokenizer is separating them into two tokens 

**Expected output is either**:
`>`
`##=`
**or** 
`>= `



Comments:
What tokenizer are you using? Is it one already trained? If so, which? If not, on what did you train your tokenizer? What is your code? What is your output?

If possible, please fill the template. It's much easier to help you if you provide all the information we need.

1. Trained using a pretrained biobert for NER task:
`tokenizer = BertTokenizer.from_pretrained("monologg/biobert_v1.0_pubmed_pmc")`

2. encode_plus method
`encoded_dict = tokenizer.encode_plus(
                        sent_str,                  # Sentence to encode.
                        add_special_tokens = True, 
                        max_length = 75,          
                        pad_to_max_length = True,
                        return_attention_mask = True,   
                        return_tensors = 'pt',     
                   )`
3. Dataset of 
> linnaeus-IOB: 
includes in one line `>= label O`

3. Current output
the line of `>=` is split into two symbols without hashes
I get 
`> label O`
`= label O`

Same for the bio dataset BC4CHEMD-IOBES, it has (R), and the tokenizer split them into three tokens without hashes

Right, I fail to understand why you think this is wrongly tokenized? This tokenizer does not have the token `>=` in its vocabulary.

You can check with:

```py
">" in tokenizer.get_vocab()  # Returns True
">=" in tokenizer.get_vocab()  # returns False
```

@LysandreJik yes sir, thank you

But I'm trying to understand why `encode_plus` did not add hashes `##` before `=`  while tokenization

I was expecting to see 
`>`
`##=`
so they would relate to the same token
but that was not the case

Is my question clear?


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Migrate Marian models names to ISO-639-3 where possible
Author: sshleifer
Pull request: no
Labels: name: wontfix

name: marian

Proposed renamings:

ROMANCE-> roa
CELTIC -> cel
NORWAY -> no  (check for conflicts w existing bilingual norwegian models.)
remove ch_group

@julien-c for NORTH_EU, why is the code ["multilingual"](https://github.com/huggingface/moon-landing/blob/d50efc4725a7546fb8159626470f3babacd44911/server/lib/ModelInfo.ts#L134-L170)
Should it be the same for all groups that do not map cleanly to an iso-639-3 group?


I will do this via mv, not cp, unless anyone objects.
cc @julien-c 

Unresloved:
how will people know if a model requires a target language code?
we will still have many cases where model name is not a language code.

Comments:
> for NORTH_EU, why is the code "multilingual"

Couldn't find a better qualifier and I think there were a lot of languages

> how will people know if a model requires a target language code?

How do they know now? Maybe add it to the model card? or you could detect if `code.includes('_')`?

> we will still have many cases where model name is not a language code.

Not that many, right?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.