noamaanMulla-03/transformers-issues huggingface dataset:
Missing of token_type_ids parameter in OPTForCausalLM.forward
Author: Tuan-Lee-23
Pull request: no

### Feature request

According to OPT's [document](https://huggingface.co/docs/transformers/model_doc/opt#transformers.OPTForCausalLM)
the OPTForCausalLM's forward method is missing a parameter `token_type_ids`.

I notice that other GPT variants almost have the `token_type_ids` in their forward method
Please upgrade it for our community

### Motivation

None

### Your contribution

None

Comments:
Hello @Tuan-Lee-23!

@patrickvonplaten, @younesbelkada or @ArthurZucker can give more details, but we aim to follow the original implementation as closely as possible. The original implementation does not leverage token type IDs and nor do the checkpoints, so there was no need to implement them for OPT.

Token Type IDs are not a required parameter for several models, so this is not an isolated case.

I'm sorry that I didn't notice about the original implementation of OPT
@LysandreJik Thank you for your clarification 

First of all the issues of non attendance or lack of participation is due to the situation with the hack/ bot whatever it os limits my screen time talk time communications all at will. Sorry but I’m no coder and came here to get hrlelp to learn how to take care of my own 

Retaliation sucks
Remove/ablating particular head in a Transformer model
Author: Jirigesi
Pull request: no
Labels: name: bug

### System Info

```shell
I am currently working on some research in which I am to analyze the important heads, which is similar to the work done in the paper "Are Sixteen Heads Really Better than One?". Basically, what they did is to iteratively ablate each head of each layer one by one, and then observe the size of the reduction in the final model prediction performance. 
If the prediction metric value obtained after ablating a head is much lower than the original model in that all heads are used, it means that the ablated head is important.
The approach they used is to time 0 to the output of the ablated head and time 1 to the rest of the considered heads. 
I am wondering if the Transformer models in huggingface could do something similar to this to evaluate the importance of each head? 

(Sorry to raise this as a bug. If I should not raise this question as bug, please advise and I will convert this from Bug to others.)
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

No reproduction for this, maybe link to the paper's repo: https://github.com/pmichel31415/are-16-heads-really-better-than-1 


### Expected behavior

```shell
Hope someone can give me indication how to evaluate which head is relatively more important in a Transformer model. 

Thanks
```


Comments:
I resolved this problem. Refer to Issue 850. https://github.com/huggingface/transformers/issues/850

Thanks
Fixing docstrings for cvt
Author: AnugunjNaman
Pull request: yes

# What does this PR do?

This PR does the following:
1. Remove the error in `README.md` where `CvT` description was copy of `CTRL`
2. Fix `size` of image for `feature extractor` which was set to `224`.
3. fix the input docstring for forward classes of `CvtModel` and `CvtForImageClassification` (head mask etc not needed).
4. Adding the largest `CvT` model. 

TODO: **Transfer the largest sized (1GB) model to Microsoft from anugunj/cvt-w24-384-22k**

@NielsRogge @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
CodeT5 tokenizer.model_max_length is 1000000000000000019884624838656
Author: khoda81
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.12.2
- Platform: Windows-10-10.0.22000-SP0
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False
```


### Who can help?

@patrickvonplaten


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("formermagic/codet5-large")
print(tokenizer.model_max_length)
```
This prints `1000000000000000019884624838656`.

### Expected behavior

```shell
To print 512.
```


Comments:
Hey @khoda81, 

The bug seems to be related directly to the model of @mozharovsky and not really to a canonical model. However, also not that T5 does **not** have a maximum length. You can use T5 with lengths as long as you want until your memory errors out. T5 uses relative positional embeddings so that there is no static weight that will give you an "out-of-index" error. 

Also see: https://github.com/huggingface/transformers/issues/5204

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix a typo `relative_postion_if_large` -> `relative_position_if_large`
Author: stancld
Pull request: yes

# What does this PR do?

This PR fixes a minor typo in `T5` and `WavLM` model codes.

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@patil-suraj 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Predicting incorrect loss when eval data size is not a multiple of batch size
Author: ajindal1
Pull request: no

## Environment info

- `transformers` version: 4.18.0.dev0
- Platform: Linux-5.4.0-96-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.13
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.12.0.dev20220411+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.4.1 (cpu)
- Jax version: 0.3.5
- JaxLib version: 0.3.5
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help

@sgugger

## Issue:
When the input data size is not a multiple of batch_size, the loss calculated seems wrong to me. As mentioned in this line https://github.com/huggingface/transformers/blob/69233cf03be5fbce0492f3997e139c4d05499e27/src/transformers/trainer.py#L2469 The loss is repeated batch_size times which does not makes sense for the last input which is not divisible by the batch_size. This also leads to the failure of HF test case (tests/trainer/test_trainer.py::TrainerIntegrationTest::test_evaluate) when I am running this on my device.

## To reproduce

Steps to reproduce the behavior:
1. Install pytest
2. RUN: pytest tests/trainer/test_trainer.py::TrainerIntegrationTest::test_evaluate

Error:
FAILED tests/trainer/test_trainer.py::TrainerIntegrationTest::test_evaluate - AssertionError: 0.517515242099762 != 0.41851458 within 7 places (0.09900066256523132 difference)

## Expected behavior

The test should pass.

Comments:
No, the evaluation loss is properly computed thanks to this line actually. Repeating it the number of times then truncating to the length of the dataset [here](https://github.com/huggingface/transformers/blob/924484ee4a6ebc79426d27eef31a1ee7d13cbb9a/src/transformers/trainer.py#L2551) makes the final evaluation loss the proper average of all losses.

As for the test not passing, I think you are running it on 2 GPUs? It's only intended to work on one.

Thank you for the quick reply. Yes, I was running the code on 2 GPUs and it works fine on 1 GPU. May I ask why is it intended to work on 1 GPU?

The batch size is actually wrong in that case. Pushing a fix!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Resuming Language Model Pre-training spends too much time skipping data
Author: murthyrudra
Pull request: no

# 🚀 Feature request

Speed-up data skipping process when resuming from a checkpoint

## Motivation

Hi, currently resuming a language model pre-training from a checkpoint spends too much time skipping data.
<img width="1641" alt="Screenshot 2022-04-13 at 1 51 58 PM" src="https://user-images.githubusercontent.com/14203368/163404033-1232510a-efe7-4d9b-b4b0-699d49be234e.png">

While we can use `--ignore_data_skip` , the model would be trained on the already seen data. This would be problematic when we are training on huge corpus where I want the model to be trained on data not encountered to train till now.

If anyone has found a simpler solution, please let me know.


Comments:
Hello! Could you please provide information on your environment, on the script used, etc? Thanks.

Hi, Thanks for the reply.
Please find the details

- `transformers` version: 4.18.0
- Platform: Red Hat Enterprise Linux release 8.5 (Ootpa)
- Python version: 3.8.11
- PyTorch version (GPU?):  1.9.1 (A100)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:  No. I am currently training on a single GPU

Please find the script used to train the model. I am using the language model training script from [here](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py). I am training on a dataset containing `117 M` sentences

```bash
TRANSFORMERS_CACHE=/tmp/ PYTORCH_TRANSFORMERS_CACHE=/tmp/ PYTHONIOENCODING=utf-8 python src/lm/run_mlm.py \
    --model_type bert \
    --config_overrides="max_position_embeddings=512" \
    --remove_unused_columns False \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 8 \
    --train_file train.txt \
    --validation_file dev.txt \
    --line_by_line \
    --do_train \
    --do_eval \
    --preprocessing_num_workers 64 \
    --pad_to_max_length \
    --evaluation_strategy steps \
    --num_train_epochs 1 \
    --output_dir ./models/bert \
    --report_to tensorboard \
    --cache_dir /tmp/ \
    --logging_steps 10000 \
    --save_steps 10000 \
    --save_total_limit 2 \
    --tokenizer_name ./models/bert_tokenizer
```

cc @sgugger 

I don't have a simpler solution.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Why do we need to expand the token_type_ids?
Author: CaoYiwei
Pull request: no

in modeling_bert.py, foward function of bertembeddings, line224, we can just write like this:
```python
        if token_type_ids is None:
            if hasattr(self, "token_type_ids"):
                token_type_ids = self.token_type_ids[:, :seq_length]
            else:
                token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)
```

instead of:
```python
        if token_type_ids is None:
            if hasattr(self, "token_type_ids"):
                buffered_token_type_ids = self.token_type_ids[:, :seq_length]
                buffered_token_type_ids_expanded = buffered_token_type_ids.expand(input_shape[0], seq_length)
                token_type_ids = buffered_token_type_ids_expanded
            else:
                token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to initialize BigBird Encoder-Decoder model with weights of full_attention Transformer Encoder model like BERT
Author: tetsu9923
Pull request: no

Hi, I am trying to train summarization model with [run_summarization.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization.py).
As a starting point, I want to initialize BigBird Encoder-Decoder model with weights of full_attention Transformer Encoder model (e.g. BERT, [SPECTER](https://huggingface.co/allenai/specter))

What should I do?
Thanks.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
A tool to generate this library with (a) specific model(s)
Author: dinhanhx
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

A tool that allows users (mostly researchers) to get a specific version of (a) transformer model(s) depending on framework, hugging face version.

For example, I want only the source code of BART model for PyTorch in the latest version of HuggingFace, but I don't want to break other HuggingFace feature. The tool should remove any files related to other models, and any frameworks such as TensorFlow. I could run `gettransformers bart pytorch`. This command should generate for me a huggingface/transformers library that only has BART model written with PyTorch.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

To let the researchers obtain less line of codes that they can reuse. 

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md -->

I propose the feature.


Comments:
Hey @dinhanhx! The transformers library is made to have models be independent of each other, so that when modifying a model file, you're certain that it will not break other models. You can therefore feel free to modify the code of BART for your use-case, and rest-assured that it will not break other models.

If you'd like to create a brand new model from BART and tweak it to your use-case while keeping BART intact, I recommend you use the `add-new-model-like` command line feature. You can read more about it [here](https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
update vilt model: remove index param in torch.meshgrid
Author: chengjuzhou
Pull request: yes

# What does this PR do?

remove index argument in torch.meshgrid in ViLT model which would lead to 
`Exception has occurred: TypeError
meshgrid() got an unexpected keyword argument 'indexing'`






Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16534). All of your documentation changes will be reflected on that endpoint.

Hi,

Thanks for your PR. However, reading the [documentation of torch.meshgrid](https://pytorch.org/docs/stable/generated/torch.meshgrid.html), the PyTorch team is planning to change the default behaviour in the future to indexing="xy" to match the behaviour of Numpy. So I don't think we can merge this PR.

Instead, we could add a warning, telling users to have PyTorch 1.10 installed or higher, similar to #16756.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Trainer: Support scheduler that reduce LR on loss plateau.
Author: KarelDO
Pull request: no

# 🚀 Feature request

The HuggingFace Trainer currently only supports learning rate schedulers where the learning rate follows a fixed trajectory. Schedulers that adapt the learning rate to the state of the training procedure are (as far as I know) not supported.

As en example, take [ `ReduceLROnPlateau`](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html) from `torch.optim.lr_scheduler`. This scheduler reduced the learning rate when a given metric is stuck on a plateau. In its `step` method, this scheduler requires an input. This input, typically the validation metric, is used to decide whether or not to decrease the learning rate.

This behaviour could be supported by giving the scheduler access to the training/validation metrics. This would most likely involve changing [this piece of code](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/trainer.py#L1468).

I believe that [`TrainerState`](https://huggingface.co/docs/transformers/v4.17.0/en/main_classes/callback#transformers.TrainerState) holds the training/validation metrics in its `log_history` attribute. If the scheduler would have access to `TrainerState`, it would be easy to create a 'stateful' scheduler.

## Motivation

This change would allow user to easily use more complex learning rate schedulers together with the `Trainer` class.



Comments:
Might be of interest to @sgugger 

This is not a high-priority feature on our list, as no Transformers pretraining or fine-tuning is making use of such schedulers in the literature. If you want to make a PR with this, I'm happy to have a look though.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add type hints OpenAIGPT
Author: TristanBilot
Pull request: yes

This PR adds type annotations for both pytorch and tensorflow implementations of OpenAIGPT model.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @TristanBilot, I'm sorry I didn't see this PR until now! It looks like those models have had type hints added by other PRs. Also, this PR is quite hard to review because of the rebase, which has made the list of "files changed" really large. Can you check the `OpenAIGPT` model files to see if we still need a PR for this, and if we do, can you make a fresh one based on the latest version of `main`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Remove `pos` from _build_network_inputs
Author: tanmoyio
Pull request: yes

@NielsRogge 
https://github.com/huggingface/transformers/issues/15971


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16122). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Rocketknight1 is it mergeable ?

Hi, I don't know much about this issue unfortunately. Does it relate to the issue from @NielsRogge ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix CodeParrot training script
Author: loubnabnl
Pull request: yes

This PR fixes some features in the training script of CodeParrot:

* use `Pytorch` implementation of `AdamW` instead of `transformers` implementation
* add shuffling of the sequences in the batches
* fix error in weight decay for LayerNorm
* change the tracked loss to the average over batches instead of the main worker loss + compute average over accumulated steps manually for wandb/tensorboard plot
* update requirements

cc @lvwerra

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Quick question about efficiently initializing model parameters
Author: FrederickXZhang
Pull request: no

Hi,

First of all, thanks for your great work on building such an amazing pytorch package!
I am trying to modify the [modeling_bart.py](https://github.com/huggingface/transformers/blob/v4.11.3/src/transformers/models/bart/modeling_bart.py) file, specifically [BartDecoderLayer](https://github.com/huggingface/transformers/blob/v4.11.3/src/transformers/models/bart/modeling_bart.py#L331). I am planning to add a new cross-attention layer which would attend to the graph nodes.  The following snippet is adapted from the original codes. 
```
        self.graph_attn = BartAttention(
            self.embed_dim,
            config.decoder_attention_heads,
            dropout=config.attention_dropout,
            is_decoder=True,
        )
```

My question is about how to **efficiently** initialize these parameters to be the same as the parameters of original BartDecoder Cross-attention layer. By "efficiently", I mean some simple and fast way for the initialization process. A brutal-force way I could easily think of right now is to download [pytorch_model.bin](https://huggingface.co/facebook/bart-large/blob/main/pytorch_model.bin) and then manually assign values by doing key-matching. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.