noamaanMulla-03/transformers-issues huggingface dataset:
Language modelling head has zero weights in pretrained TFMobileBertForMaskedLM
Author: mknz
Pull request: no

## Description
The `TFMobileBertForMaskedLM` example returns all zero logits while `MobileBertForMaskedLM` example works fine.
https://huggingface.co/transformers/model_doc/mobilebert.html#tfmobilebertformaskedlm

I checked language modeling head weights for both models and found that TF pretrained model has zero weights.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@jplu @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): TFMobileBertForMaskedLM

The problem arises when using:
- [x] the official example scripts: (give details below)

## To reproduce

```
from transformers import MobileBertForMaskedLM
from transformers import TFMobileBertForMaskedLM

# PyTorch
model = MobileBertForMaskedLM.from_pretrained('google/mobilebert-uncased')
print(list(model.cls.parameters())[0])

# Parameter containing:
# tensor([-7.2946, -7.4302, -7.5401,  ..., -7.4850, -7.4503, -2.7798],
#       requires_grad=True)

# TensorFlow
model = TFMobileBertForMaskedLM.from_pretrained('google/mobilebert-uncased')
print(model.layers[1].get_weights()[0])

# array([0., 0., 0., ..., 0., 0., 0.], dtype=float32)
```
## Expected behavior
Language modeling head for TFMobileBertForMaskedLM should have the same weights as of MobileBertForMaskedLM


Comments:
Hello!

Indeed there is an issue in the naming for `TFMobileBertForMaskedLM`. This will be fixed in the next release.

OK, thanks for the quick response!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Please provide format of the dataset to finetuning wav2vec using run_asr.py script
Author: vigneshgig
Pull request: no

Hi @patrickvonplaten, thanks for the great work, could you please provide some examples for the dataset format to train the wav2vec model using run_asr.py script.

Comments:
We are organizing a "fine-tuning XLSR-53" event. Check this announcement: https://discuss.huggingface.co/t/open-to-the-community-xlsr-wav2vec2-fine-tuning-week-for-low-resource-languages/4467. Would be awesome if you want to participate :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[deepspeed] test on one node 2 gpus max
Author: stas00
Pull request: yes

Deepspeed devs, who will be running our deepspeed tests as part of their tests, discovered that we had an unbound number of nodes and gpus in the tests so it was firing on multiple nodes and many gpus, so wasn't quite ready for it. So fixing it.

Comments:
ok, it's all working in the Deepspeed team's test suite - yay!
config.json not found when loading fasttext-language-id model
Author: nbeuchat
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Linux-5.4.0-60-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@julien-c 

## Information

Model I am using (Bert, XLNet ...): [julien-c/fasttext-language-id](https://huggingface.co/julien-c/fasttext-language-id)

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("julien-c/fasttext-language-id")
model = AutoModel.from_pretrained("julien-c/fasttext-language-id")
```

Which returns the following error:
```
404 Client Error: Not Found for url: https://huggingface.co/julien-c/fasttext-language-id/resolve/main/config.json
```

## Expected behavior
The model should load or the config file should be present


Comments:
Hi @nbeuchat this is a fasttext model, not a `transformers` model, so you can't load it that way.

I've updated the main button on the webpage to make it clearer that you need to use the model in fasttext:

<img width="1067" alt="Screenshot 2021-01-11 at 19 21 07" src="https://user-images.githubusercontent.com/326577/104222282-11d03600-5410-11eb-9b03-307fc776f197.png">
<img width="885" alt="Screenshot 2021-01-11 at 19 21 56" src="https://user-images.githubusercontent.com/326577/104222284-14329000-5410-11eb-842a-05fa8e05c1ca.png">




Also cc'ing @thomwolf and @celebio <3

Got it, thanks for the info and for the quick update! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Benchmark]
Author: soapland-master69
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!

Comments:
Trainer iterable dataset
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds full support for `IterableDataset` training set in the main Trainer (just the training set, evaluation/prediction will require way more work). Up until now, the Trainer kind of support training dataset that are instances of `IterableDataset`, but in a distributed setting, the training will be on the same data on all processes, which is... not ideal. This PR fixes that and adds some tests.

Comments:
position_ids generated from Roberta
Author: hwijeen
Pull request: no

Hi,

Roberta created `position_ids` from `input_ids` using [this function](https://github.com/huggingface/transformers/blob/3d339ee6595b9e42925559ae21a0f6e77f032873/src/transformers/models/roberta/modeling_roberta.py#L1494).
When the max sequence length is 512, I expect the `position_ids` to be [0, 1, ..., 512].
However, the function gives me [1, 2, ..., 513] which later results in an CUDA index error for position embedding.
I would appreciate if someone could tell me what I am doing wrong.
```python

def create_position_ids_from_input_ids(input_ids, padding_idx, past_key_values_length=0):
    mask = input_ids.ne(padding_idx).int()
    incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask
    return incremental_indices.long() + padding_idx

ipdb> input_ids
tensor([[    2,    20,   630,  ...,    22,    20,     3],
        [    2,   168,   106,  ...,     4, 31532,     3],
        [    2,   287, 14603,  ...,   284,  1594,     3],
        ...,
        [    2,     4,   873,  ...,  5549, 24276,     3],
        [    2,    12,    56,  ...,    87,   419,     3],
        [    2, 30683,   419,  ...,   761,   312,     3]], device='cuda:0')

ipdb> incremental_indices
tensor([[  1,   2,   3,  ..., 510, 511, 512],
        [  1,   2,   3,  ..., 510, 511, 512],
        [  1,   2,   3,  ..., 510, 511, 512],
        ...,
        [  1,   2,   3,  ..., 510, 511, 512],
        [  1,   2,   3,  ..., 510, 511, 512],
        [  1,   2,   3,  ..., 510, 511, 512]], device='cuda:0',
       dtype=torch.int32)

ipdb> padding_idx
0

```

Comments:
See https://github.com/huggingface/transformers/issues/10736#issuecomment-800175342

Tip: if you search on this Github repo "position ids roberta", you get a lot of answers.
[Benchmark]
Author: autumnX1515
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!

Comments:

```

> # 🖥 Benchmarking `transformers`
> 
> ## Benchmark
> 
> Which part of `transformers` did you benchmark?
> 
> ## Set-up
> 
> What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?
> 
> ## Results
> 
> Put your results here!
mBART Large-50 MMT provides incorrect translation when the source and target language are the same
Author: xhluca
Pull request: no

mBART Large-50 MMT provides incorrect translation when the source and target language are the same, e.g. when translating from "en_XX" to "en_XX"


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.0+cu101 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help

@patrickvonplaten, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): 
`facebook/mbart-large-50-one-to-many-mmt`

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

```python
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast
article_en = "The head of the United Nations says there is no military solution in Syria"
model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-one-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-one-to-many-mmt", src_lang="en_XX")
model_inputs = tokenizer(article_en, return_tensors="pt")
generated_tokens = model.generate(
    **model_inputs,
    forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"]
)
decoded = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
print(decoded)
```

Returns:
```
['Şeful Naţiunilor Unite declară că nu există o soluţie militară în Siria']
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Should return something in english, preferably the same content as the original input

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@patil-suraj Was this resolved? If so I'll close this issue

HI @xhlulu 

I don't think if this is an issue with the model implementation, also the model is not really expected to do well on paraphrasing (which English-English), I've seen few other issues where models output text in the wrong language but it's the same with original model in `fairseq` as well. From my experience, multilingual models tend to do this in few cases.

Thanks, that makes sense! Glad you clarified it 😊
Git does not find the model folder and does not commit model files in the hugging face
Author: MLDovakin
Pull request: no

I work for Google Colab. This is how I save my training model files
````
trainer.save_model("./kvantorium-small")
tokenizer.save_pretrained("/content/For_tokenize", legacy_format=False)
````

Next, I want to commit my files to the hugging face repository. As shown in the guide (https://huggingface.co/welcome)

all these lines of code compile successfully:

````
!sudo apt-get install git-lfs
!pip install huggingface_hub
!huggingface-cli login
!huggingface-cli repo create simple-small-kvantorium

!git lfs install
!git clone https://huggingface.co/Fidlobabovic/simple-small-kvantorium
````
But when I want to make a push request to save files to the repository, I get an error that there is no such repository. How do I rewrite the request to publish the files to the repository? Is this a problem that I work for Google Colab? Thanks a lot in advance, you are helping me a lot


#9878 

````
!git add .
!git commit -m "commit from $Fidlobabovic/simple-small-kvantorium"
!git push

fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
````

Comments:
hi @IndianMLGay 

You should `cd` into `simple-small-kvantorium` directory and then do `git add/commit/push` etc

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
mBART 50 models not found in model shortcut name list
Author: codingnoobneedshelp
Pull request: no

Transformers version: 4.4.0.dev0

Hello, I'm trying to fine-tune mBART 50 with your seq2seq examples.

Getting this error:

Model name 'facebook/mbart-large-50' not found in model shortcut name list (facebook/mbart-large-en-ro, facebook/mbart-large-cc25).

Traceback (most recent call last):
  File "/content/transformers/examples/seq2seq/run_seq2seq.py", line 668, in <module>
    main()
  File "/content/transformers/examples/seq2seq/run_seq2seq.py", line 349, in main
    use_auth_token=True if model_args.use_auth_token else None,
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/auto/tokenization_auto.py", line 399, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 1789, in from_pretrained
    resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 1806, in _from_pretrained
    **(copy.deepcopy(kwargs)),
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 1860, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/mbart/tokenization_mbart.py", line 109, in __init__
    self.set_src_lang_special_tokens(kwargs.get("src_lang", "en_XX"))
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/mbart/tokenization_mbart.py", line 199, in set_src_lang_special_tokens
    self.cur_lang_code = self.lang_code_to_id[src_lang]
KeyError: None

Any Ideas on how to fix this?

Thanks

Comments:
Hi @codingnoobneedshelp , thank for reporting this issue. Right now MBart50Tokenizer does not work with `AutoTokenizer`.
There will be a new script for translation in the next ~2 weeks that will handle this issue. For now, you could just modify the script to  use `MBart50Tokenizer`, instead of `AutoTokenizer`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @codingnoobneedshelp 

This is now resolved, the `run_translation.py` script now supports fine-tuning mBART-50.
Adding pipeline task aliases.
Author: Narsil
Pull request: yes

# What does this PR do?

Two tasks were sort of not aligned with the pipeline names.

`sentiment-analysis` -> `TextClassificationPipeline`
`ner` -> `TokenClassificationPipeline`

In order to make this change backward compatible, yet make the code more consistent, this PR 
introduces a TASK_ALIASES dictionary, which remaps a task name to another *canon* task name.

Previously working code should still be working, simply we are adding `text-classification` and `token-classification` tasks
available to `pipeline(...)` function.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@LysandreJik 
@philschmid 

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
Support for set_epoch in IterableDataset
Author: sgugger
Pull request: yes

# What does this PR do?

I merged #11254 a bit too fast and forgot to actually call the `set_epoch` method in the main training loop at the beginning of each epoch.
Also, it looks like the Datasets library will deal internally with the RNG logic by having a `set_epoch` method, this PR allows support for that.

Comments:
Tokenizer fast save
Author: sgugger
Pull request: yes

# What does this PR do?

This PR changes the behavior of `save_pretrained` to have the fast tokenizer unified json file being saved as well as the files of the "slow" tokenizer. The default of `legacy_format` is changed to None with the following behavior:
- unset -> a fast tokenizer is saved with both formats (tokenizer.sjon + legacy format)
- set to True -> a fast tokenizer is saved in legacy format only
- set to False -> a fast tokenizer is saved with just the tokenizer.json format

Along with that, a slight change in the `from_pretrained` method is needed since the added tokens for a fast tokenizer are often inside the tokenizer.json file, so already added before we get to a possible added_tokens.json. There is currently a bug where loading a tokenizer from a folder with both files (tokenizer.json and added_tokens.json) will fail, this PR fixes it.

Comments:
Wrong shape output for loss of TFGPT2LMHeadModel
Author: ssss1029
Pull request: no

## Environment info
     
- `transformers` version: 4.0.0
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): 2.3
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik
@jplu

## Information

Model I am using (Bert, XLNet ...): TFGPT2LMHeadModel

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

The following in a Python interpreter:
```python
import transformers
model = transformers.models.gpt2.TFGPT2LMHeadModel.from_pretrained('gpt2')
input_ids = tf.ones((1, 1024), dtype=tf.int32)
labels = tf.ones((1, 1024), dtype=tf.int32)
print(model(input_ids, labels=labels, return_dict=True, training=True).loss.shape)
```
Outputs
```
TensorShape([1023])
```

It seems the loss output is dependent on batch size: 
```python
labels = tf.ones((2, 1024), dtype=tf.int32)
input_ids = tf.ones((2, 1024), dtype=tf.int32)
print(model(input_ids, labels=labels, return_dict=True, training=True).loss.shape)
```
Outputs
```
TensorShape([2046])
```
## Expected behavior

According to the docs (https://huggingface.co/transformers/model_doc/gpt2.html#tfgpt2lmheadmodel), the loss is of shape `(1,)`. However, this is not the shape that is returned.

Comments:
Hello !

This is an error in the documentation. TF doesn't apply a mean across all the values so you basically get a loss of shape 1023 (sequence length - 1 because of the right shift). Thanks for having spotted this!

This issue has been stale for 1 month.