noamaanMulla-03/transformers-issues huggingface dataset:
make docs failing
Author: kamalkraj
Pull request: no

https://github.com/huggingface/transformers/blob/2a606f9974feb0f7578e6a638c7e5b548523ecb4/CONTRIBUTING.md?plain=1#L211-L221

```bash
make docs
cd docs && make html SPHINXOPTS="-W -j 4"

Application error:
config directory doesn't contain a conf.py file (source)
make[1]: *** [html] Error 2
make: *** [docs] Error 2
```

transformers version: master branch

@sgugger 

Comments:
Yes we are not using sphinx anymore to build the doc. The contributing guide will be updated in the coming days.

Okay
Fix the build documentation job
Author: sgugger
Pull request: yes

# What does this PR do?

The last two releases were not automatically documented. That's fine for v4.14.0 since we yanked it, and I did a manual update for v4.14.1, but in the future it would be nice to not have to do this!

The first fix is to make sure Transformers is installed from the current checked out branch and not the content of the master branch. The second fix is to run that job on branches with the patter vxxx-release

Comments:
transformers_4.13.devo giving error during saving model
Author: Aniruddha-JU
Pull request: no

## Environment info


- `transformers` version: 4.13
- Platform: linux
- Python version: 1.80
- PyTorch version (GPU?): gpu


@patil-suraj
Model :mt5-base

input : python  run_summarization.py     --model_name_or_path google/mt5-base     --do_train  --do_predict      --train_file /home/aniruddha/mt5_data/bengali_8_shot.json    --test_file /home/aniruddha/mt5_data/ben_dev.json    --source_prefix "summarize: "     --output_dir mt5_ben_16_667/  --overwrite_output_dir     --per_device_train_batch_size=1     --per_device_eval_batch_size=4    --predict_with_generate --seed 667 --save_steps 14000000 --num_beams 3


error:Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 13.5967, 'train_samples_per_second': 1.765, 'train_steps_per_second': 0.883, 'train_loss': 12.063149770100912, 'epoch': 3.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:13<00:00,  1.13s/it]
[INFO|trainer.py:1995] 2021-11-06 20:20:47,931 >> Saving model checkpoint to mt5_ben_16_667/
[INFO|configuration_utils.py:417] 2021-11-06 20:20:47,932 >> Configuration saved in mt5_ben_16_667/config.json
Traceback (most recent call last):
  File "run_summarization.py", line 648, in <module>
    main()
  File "run_summarization.py", line 571, in main
    trainer.save_model()  # Saves the tokenizer too for easy upload
  File "/home/aniruddha/anaconda3/envs/ani/lib/python3.8/site-packages/transformers/trainer.py", line 1961, in save_model
    self._save(output_dir)
  File "/home/aniruddha/anaconda3/envs/ani/lib/python3.8/site-packages/transformers/trainer.py", line 2009, in _save
    self.model.save_pretrained(output_dir, state_dict=state_dict)
  File "/home/aniruddha/anaconda3/envs/ani/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1053, in save_pretrained
    del state_dict[ignore_key]
KeyError: 'encoder\\.embed_tokens\\.weight'
![Capture](https://user-images.githubusercontent.com/36475622/140614764-1f42f310-3ef1-4caf-87aa-0edb1651d9c7.PNG)








 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add Speaker Diarization and Verification heads
Author: anton-l
Pull request: yes

# What does this PR do?

This adds the Audio Frame classification (equivalent of token classification) and X-vector (speaker embedding extraction) heads to `Wav2Vec2` and `UniSpeech-SAT` models. The target tasks for the heads are SUPERB's Speaker Diarization and Speaker Verification respectively. These were mainly motivated by `UniSpeech-SAT`, since the model performs better on those tasks, rather than on ASR.

Sources for the models and weights from the SUPERB's `s3prl` framework:
* `ModelForAudioFrameClassification`: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/diarization/model.py (with `use_rnn=False`)
* `ModelForXVector`: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/sv_voxceleb1/model.py#L261

The heads for both W2V2 and US-SAT were finetuned from scratch, since the official checkpoints use custom (better) heads that are incompatible with SUPERB's evaluation protocol.


Comments:
@patrickvonplaten the code is ready for a full review now. I'll post the model links and evaluation results (DER and EER metrics) shortly.

checking tomorrow after the `-large` checkpoint problem is solved :-)
some error when I run the pytorch example wav2vec2 by rum_common_voice.py
Author: olochi
Pull request: no
Labels: name: New model

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/models/facebookav2vec2-base/revision/main

when I went to the web https://huggingface.co in browser, I found that the page https://huggingface.co/api/ is 404.

Has the structure of https://huggingface.co changed? and what should I do?

Comments:
[Generation] Make `generate()` method compatible with speech and vision inputs while keeping 100% backward compatibility.
Author: patrickvonplaten
Pull request: no

# 🚀 Feature request

As mentioned by @NielsRogge and @LysandreJik [here](https://github.com/huggingface/transformers/pull/14139#discussion_r736788040) the `generate` function should be made more general by not assuming that the inputs are necessarily `input_ids`. We should also allow inputs to be called `input_values`, `input_features`, `pixel_values` to cope with speech and vision models at 100% backward compatibility. 

@anton-l - if you want to dive a bit into `generate()`, feel free to give this a stab :-)


Comments:
If you're too busy with other things - don't worry, I can take care of it in like 2 weeks.

Also as a note, the following hacks should then be removed:

- [this hack](https://github.com/huggingface/transformers/blob/b567510cff606c9dd67cb7c56169bc596590e700/src/transformers/trainer_seq2seq.py#L169) in `Seq2SeqTrainer`
- remove `attention_mask=None` inputs currently defined in the forward pass of `modeling_vit.py` and `modeling_deit.py`.

For the record, this

```
remove attention_mask=None inputs currently defined in the forward pass of modeling_vit.py and modeling_deit.py
```

is addressed in this PR #14148, which is still under review.

PR will be finished soon: https://github.com/huggingface/transformers/pull/14784
Implement head_mask for Flax BERT and other models copied from BERT
Author: stancld
Pull request: yes

# What does this PR do?

This PR implements attention head masking for Flax BERT models, and other models copied from BERT, i.e. Roberta, BigBird, Electra. This PR thus narrows the gap between Flax and other implementations.


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@LysandreJik 


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Hi @patil-suraj, thank you a lot for the review. I commented on the issue with BigBird head-masking. As I mentioned, I tried to follow the current `PyTorch` implementation, and I'm definitely willing to have a look at handling head masking for different attention mechanisms.

I forgot to write this earlier, do you think you could also update the flax templates, the templates test are failing, 
If not let me know, I will take care of it , thanks :) 

@patil-suraj Oh, good point. I'll update the template :]
Loading RoBERTa pytorch_model.bin checkpoint in fairseq for evaluation
Author: jivatneet
Pull request: no

Hi,
I have fine-tuned a RoBERTa model using transformers and am facing issues when I'm loading that checkpoint in fairseq. The evaluation framework poses constraints so I have to load the model using fairseq.

```
self.model = RobertaModel.from_pretrained(
            roberta_model_dir, checkpoint_file=roberta_model_name
        )
```

I pass `checkpoint_file` as pytorch_model.bin

I get `KeyErrors` in fairseq` load_checkpoint_to_cpu function` (https://github.com/pytorch/fairseq/blob/main/fairseq/checkpoint_utils.py#L281). I have seen there exists a function for converting fairseq model to huggingface model but not able to find a fix for this. I even have models saved using `torch.save()` if that helps.

Would really appreciate help on this. Thanks.


Comments:
Hi! We have a script to convert from `fairseq`'s checkpoints to `transformers`, but we don't have the script that would do it the other way around. You could try and adapt the script to do the conversion the other way, as it's essentially just renaming layers.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Can you get Q/A links from LayoutLM
Author: gregg-ADP
Pull request: no

We are fine tuning the LayoutLM model from microsoft and we can’t see where you can get the links from the inference output. We get the list of labels for each word (if it is a question, answer, title) but we don’t get the links from the question to the answer.
Can someone tell us where to find that?

I think @NielsRogge is an expert on this?

Comments:
Hi,

This is a question that is asked a lot. The LayoutXLM authors have implemented a `LayoutLMv2ForRelationExtraction`, of which an example script can be found [here](https://github.com/microsoft/unilm/tree/master/layoutxlm#fine-tuning-for-relation-extraction).





Thanks so much for the quick response @NielsRogge.  That does look like it will help with what we are looking for.  
Right now we are fixing dependency and other issues to get it running.  We will share what we learned so that the next person won't have to go through the fixes.

Thanks again!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
LED models give: `IndexError: index out of range in self`
Author: nicola-decao
Pull request: no

Using a Longformer on more an input with than 1024 tokens doesn't seem to work. But it should as the Longformer was specifically designed to handle up to 16k tokens as input. See the code below.

To reproduce:
```python
from transformers import LEDTokenizer, LEDForConditionalGeneration
  
tokenizer = LEDTokenizer.from_pretrained("allenai/led-base-16384")
model = LEDForConditionalGeneration.from_pretrained("allenai/led-base-16384")

# this works (tokens < 1024)
model(**tokenizer("hello " * 120, return_tensors="pt"))

# this does not work! (tokens > 1024)
model(**tokenizer("hello " * 1200, return_tensors="pt"))
```

Versions:
* python 3.8.11
* pytorch 1.11.0.dev20210928+cu111
* transformers 4.11.0

Comments:
cc @patrickvonplaten 

Hey @nicola-decao,

That's a great issue! The problem here is the following:

1. The forward pass of encoder-decoder models expects both `input_ids` (for the encoder) and `decoder_input_ids` (for the decoder). Now since LED was derived from BART, the logic of automatically generating `decoder_input_ids` from `input_ids` is kept which was used for BART for pre-training. This means that if you do `model(input_ids)` than `decoder_input_ids` is not provided and thus automatically created by shifting `input_ids`.

2. LED can handle up to 16K `input_ids` tokens, but **not** 16K `decoder_input_ids` tokens. The idea of LED was really to be able to process very long inputs (articles to summarize) with the assumption that the decoder outputs don't have to be very long (summaries). There can be at most 1024 `decoder_input_ids` tokens which is violated if `1200` tokens are automatically generated.

This being said the following example should work:

```python
from transformers import LEDTokenizer, LEDForConditionalGeneration
  
tokenizer = LEDTokenizer.from_pretrained("allenai/led-base-16384")
model = LEDForConditionalGeneration.from_pretrained("allenai/led-base-16384")

inputs = tokenizer("hello " * 12000, return_tensors="pt")
# let's make sure `decoder_input_ids` are provided so that they are < 1024
inputs["decoder_input_ids"] = inputs["input_ids"][:, :512]

model(**inputs)
```

We probably should add a warning message here: https://github.com/huggingface/transformers/blob/29dfb2dbb10cdba6327ff287db56b182c1db29b1/src/transformers/models/led/modeling_led.py#L2212 to make sure the user is aware that `decoder_input_ids` are automatically generated. In case you would be interested to open a PR, that would be great!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Weird assumptions in the PLM collator
Author: hiaoxui
Pull request: no

Hi, I'm a researcher working with the HF codebase. I'm using the PLM collator, but some assumptions of that module confuse me and block my experiments. Can you explain?

The PLM collator assumes the input length is even so it can split it into two halves. The explanation starts [here](https://github.com/huggingface/transformers/blob/1c76a51615ccd5c8e60570771b29ef16a9c3bc17/src/transformers/data/data_collator.py#L1269). It says that we need to avoid leaking information so we can only permute half of the sequences if we assume half of the sequence is mems, if I understand correctly.

However, it confuses me:
1. The input sequence here contains only the new tokens, and the mems should be a separate variable. So why do we assume half of the sequence is going to be reused?
2. It points me to read the documentation of `mems`, while the documentation of `mems` is copied from other transformer models. It says that `mems` is used to speed up decoding by avoiding repeated computation, which doesn't fit the situation of XLNet, where mems are used to store history information. Is the documentation correct? If not, where can I find the correct version?

Thanks in advance!

@sgugger 

Comments:
This is community contributed so you should ask the person that contributed it :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Comparison Chart (Table) for all the existing BERT models.
Author: pratikchhapolika
Pull request: no

# 🚀 Feature request

Can we have a Comparison Chart (Table) for all the existing BERT models in one place. 
For an example:

- Data the model is trained on?
- Hidden layers?
- Architecture difference?
- How it can be loaded for classification/generation - 2-3 lines of Code ?
- Does it has pooler layer or not?
- Parameters
- Model Size
- General Inference time 
- Training time


The chart table then will have link to deep-dive in details directed to the model page.


## Motivation

Sometimes it becomes overwhelming to go through each of the models ( too many of them). If its in a comparison chart then it becomes very simplified.


It's just an idea. We can elaborate or work on this idea. 




Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How FNet handle PAD token?
Author: celsofranssa
Pull request: no

How to avoid [FNet](https://huggingface.co/transformers/model_doc/fnet.html#transformers.FNetForPreTraining.forward) process `PAD` token as it has no `attention_mask` alternative argument in `forward` method?

```python
from transformers import FNetTokenizer, FNetModel
tokenizer = FNetTokenizer.from_pretrained('google/fnet-base')
model = FNetModel.from_pretrained('google/fnet-base')

features=tokenizer.encode(text="This is new encoder", max_length=16, padding="max_length",
                                           truncation=True, return_tensors="pt")
# tensor([[   4,  325,   65,  351, 1703,  242,   14,    5,    3,    3,    3,    3,
#            3,    3,    3,    3]])
```
Or does this model handle `PAD` token (with `token_id=3`) internally?

Comments:
cc @gchhablani @patrickvonplaten 

Good observation! The model actually processes the PAD token just like any other token, which means that padding an input of length 32 to 128 yields different results to just forwarded the input of length 32. For both pre-training and fine-tuning each sequence length was always padded to 512 tokens, so it's recommended to do the same for inference

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
DeBERTa checkpoints contain `config` keys
Author: Rocketknight1
Pull request: no

We noticed some issues when loading PyTorch checkpoints for `DeBERTa`; the `state_dict` contained a `config` key with the config settings for the model, and this confused the cross-loading method. I don't think PyTorch state dicts are usually supposed to contain those. If possible, could we upload some new checkpoints that don't contain the configs?

@BigBird01 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@BigBird01 pinging on this one again!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add an argument to set bucket_cap_mb for PyTorch DDP
Author: changlan
Pull request: yes

# What does this PR do?

`bucket_cap_mb` determines the bucket size that PyTorch's `DistributedDataParallel` will group gradient parameters in. In some occasions, tuning this parameter will have significant impact on the distributed training performance. This PR allows user to change the `bucket_cap_mb` parameter via a new training argument `--ddp_bucket_cap_mb`.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
There is just the formatting issue to deal with before we can merge. Can you run `make style` on your branch to fix it?
Thanks!

Done - thanks for the review!

Mmm, are you sure you installed the proper versions of the formatting tools with `pip install transformers[dev]` or `pip install transformers[quality]` ? The failure is still there.


My bad - I forgot to push the latest commit. Should be working now?

Yes, the failure is unrelated. Thanks again for your PR!