noamaanMulla-03/transformers-issues huggingface dataset:
Cached sin and cos matrices for rotary at GPT-J model initialization for faster generation
Author: kurumuz
Pull request: yes

# What does this PR do?

Makes generations faster by caching sincos matrices for rotary for the max sequence length at model initialisation so it won't be done every other forward.

around ~%15 percent speedup overall
Tested on A100-SXM4-80GB
Benchmarks:
1 token forward average runtime out of 100 iterations with cached sin cos:
0.023421470640283642s

1 token forward average runtime out of 100 iterations without cached sin cos:
0.02738137301433869s

10 generations with 1 token context and 40 tokens generated without sincos caching:
1.410405054409057s

10 generations with 1 token context and 40 tokens generated with sincos caching:
1.2199230638332665s

Test script:
```
torch.manual_seed(123)
input_ids = torch.randint(0, 100, (1, 1)).cuda().long()

iterations = 11
with torch.no_grad():
    for i in range(iterations):
        if i == 1:
            #start counting time after tier 1 due to pytorch warming up
            t = time.perf_counter()
        outputs = model.generate(input_ids, max_length=input_ids.shape[-1] + 50, min_length=input_ids.shape[-1] + 50, do_sample=True, use_cache=True)
        #outputs = model.forward(input_ids)
print((time.perf_counter() - t) / (iterations - 1))
```

Models:
- GPT-J



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19991). All of your documentation changes will be reflected on that endpoint.

Mmm, now it looks like the tests are not running for some reason. Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)? And/or push an empty commit?

Hi @kurumuz Thank you for the PR. I left a few comments, especially for the shape and the case where `self.rotary_dim = None`.
It is not very clear to me why the (changed) shapes of `sin` and `cos` doesn't cause any issue.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`ViTHybrid`] fix last `accelerate` slow test
Author: younesbelkada
Pull request: yes

# What does this PR do?

FIxes the last test that I forgot to run on multi gpu setup ! Sorry for the multiple iterations

Fixes: 
1- https://github.com/huggingface/transformers/actions/runs/3650247864
2- in `test_model_parallelism` the test splits the model into different sub-modules, with respect to the class attribute `_no_split_modules`. Sometimes a manual device assignment is needed to avoid errors such as `tensors not on the same device`
3- Similar to https://github.com/huggingface/transformers/pull/19912 (check the last modification)

cc @sgugger @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add missing image transforms to init
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds functionality in the image transforms library to the init to allow direct imports e.g.:
`from transformers import center_crop`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[docs] transformers/quicktour rendering issue
Author: scott-vsi
Pull request: no

There is a rendering issue in https://huggingface.co/docs/transformers/quicktour#trainer-a-pytorch-optimized-training-loop
step 5.

The markdown looks like [this](https://github.com/huggingface/transformers/blob/9e56aff58a742b48fc8edea8d28d5b80330efbcc/docs/source/en/quicktour.mdx?plain=1#L445-L451)

```python
>>> def tokenize_dataset(dataset):
...     return tokenizer(dataset["text"])


>>> dataset = dataset.map(tokenize_dataset, batched=True)
```


but `dataset = ...` renders as a separate triple-indented quote


Comments:
Transfering to the transformers repo as it's more relevant there cc @sgugger 

Should be fixed by the PR mentioned above.

Probably not worth looking at more, but just by way of comparison, there is multiple returns on this page that seem to work (again referencing `dataset.map`):

https://huggingface.co/docs/transformers/training#prepare-a-dataset
https://github.com/huggingface/transformers/blob/74330083b5eea68eadf25bcbad3bcbb094f60c57/docs/source/en/training.mdx?plain=1#L50-L54

I have no idea if it matters, but when viewed as markdown, there are some sections of the quickstart that don't render right. E.g., see the

```
```py >>> pt_batch = tokenizer( ... ["We are very happy to show you the...
```

in this [section](https://github.com/huggingface/transformers/blob/9e56aff58a742b48fc8edea8d28d5b80330efbcc/docs/source/en/quicktour.mdx#autotokenizer)



Thanks for raising this issue! 

It isn't as important when viewed as Markdown because we use a special syntax (`<frameworkcontent>`) to generate the PyTorch and TensorFlow code blocks in our docs.
Fix rendering issue in quicktour
Author: sgugger
Pull request: yes

# What does this PR do?

Fixes #20700 I think the two new lines is what makes the problem happen.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ok so the extra line is what causes the issue, but then `black` is unhappy with the formatting.

Ok, so I just split the content in two blocks at the end, since `doc-builder` wants no more than one empty lines and `black` wants two :man_shrugging:
Added resources on albert model
Author: JuheonChu
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->
Co-author: @Adia Wu <wua@dickinson.edu>
Fixes #20055 


## Before submitting
- [o] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [o] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [o] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [o] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [o] Did you write any new necessary tests?


## Who can review?

@stevhliu @younesbelkada 


Comments:
Thank you @younesbelkada ! Would you mind if I ask you how I can pass the 
ci/circleci: tests_pipelines_tf tests?

I tried
`pip3 install --upgrade pip`
`pip3 install --upgrade tensorflow`

Thanks for your PR! Could you focus it solely on the new resources added? There are multiple changes that are not desired.

Thank you! Will try! 
So, I deleted those undesired behaviors, and now I will look for more resources to add!



Do you mind if I open a new Pull Request in order to contain only meaningful commits?

Yes please, that'd be great!
Albert resource
Author: JuheonChu
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # 20055


## Before submitting
- This PR adds resources on ALBERT model based on the materials outlined in #20055.


## Who can review?
@stevhliu 

Co-authored by: @Adia Wu


Comments:
Closing this issue and reopening the issue at [Issue 20697](https://github.com/huggingface/transformers/pull/20697).
Replace assertions with value errors on distilbert model
Author: JuheonChu
Pull request: yes

# Replace assertions with ValueErros on distilbert model

This PR is made to check if the valid checks made from #20433 pass for all cases.

Co-author: @batese2001


To: @younesbelkada 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you so much!


Thank you so much for your guidance and help!
Replace assertions with ValueErros on distilbert model
Author: JuheonChu
Pull request: yes

Co-author: @batese2001
This is an extended PR from [here](https://github.com/huggingface/transformers/pull/20432)  for new valid-checks to improve _code quality._

This raises exceptions on Multi-head Attention models for following specified conditions on the _Distilbert_ model.

This successfully replaces assertions with ValueErrors with the _Distilbert_ model related to #12789.

Would you be open to me changing assertions if I encounter other ones?

To: @younesbelkada

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This is a new PR of my own based on your suggestion from [here](https://github.com/huggingface/transformers/pull/20375). files which improved to pass all the validity checks. 
Can I ask for your further suggestions for this PR to be merged? 
Any comments will be very appreciated! 

Below mentions of PR #12789 is a related PR regarding the replacement of assertions with raising exceptions with conditions that are contrary to the pre-defined conditions.


Hi @JuheonChu ! 👋
Thanks for the PR ;) will review it asap and let you know!

Thank you @younesbelkada  for providing us with valuable suggestions! We will make changes and make anotehr PR! 
Sincerely appreciate it :)

No worries @JuheonChu ! 
As I made you some suggestions, actually you can directly continue on this PR, let me guide you through this step by step: 
Step 1: Go to "file changed" (top right of the github UI of this PR):
<img width="921" alt="Screenshot 2022-11-25 at 22 17 47" src="https://user-images.githubusercontent.com/49240599/204059724-181f2dbc-9e09-4031-bb8c-3499b3ab12ce.png">

Step 2: For each suggestion, click on "Add suggestion to batch", to add each suggestion
<img width="921" alt="Screenshot 2022-11-25 at 22 18 59" src="https://user-images.githubusercontent.com/49240599/204059801-2e1f1a1a-cf9b-43a9-8e9b-bf8d43ea4b84.png">

Step 3: Once you have added all the suggestions (make sure to add all of them), a pop up bar will appear on the top-right corner, and you can just click to it, and the suggestions will be pushed ;) 

<img width="1467" alt="Screenshot 2022-11-25 at 22 19 43" src="https://user-images.githubusercontent.com/49240599/204059837-7f8bf1fc-4c63-4234-8dae-48b52b38eb20.png">

This way no need to open a new PR each time we do a suggestion ;) ! Let me know if anything is unclear! 

Due to the reformatting of jupyter files, I created a new PR [here](https://github.com/huggingface/transformers/pull/20463) which passes all the validation checks. I apologize for any inconvenience, but do you mind if you can check  [here](https://github.com/huggingface/transformers/pull/20463)?

To: @younesbelkada 

Sure! I propose to close this PR as we moved everything on the other PR ;)
Does `prune_heads` really speed up during inference?
Author: CaffreyR
Pull request: no

### System Info

- `transformers` version: 4.22.1
- Platform: Linux-5.13.0-48-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.13
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@LysandreJik

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi I have a code here, based on your examples. I try to use `bert` to test inference speed
The problem is no matter whether I pruned the model, the time seems to remain the same 

The code is belows

```

from transformers import AutoTokenizer, AutoModel
import torch
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
device='cuda'
prune_heads = {}
prune_heads[0] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[1] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[2] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[3] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[4] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[5] = [0,1,2,3,4,5,6,7,8,9]

prune_heads[6] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[7] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[8] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[9] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[10] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[11] = [0,1,2,3,4,5,6,7,8,9]

'''Whether to prune'''
# model.prune_heads(prune_heads). 

inputs = tokenizer("Hello world!", return_tensors="pt").to(device)
model=model.to(device)
model.eval()

import time
cnt=0
for i in range(3):
    outputs = model(**inputs)


for i in range(10):
    torch.cuda.synchronize()
    start = time.perf_counter()
    outputs = model(**inputs)
    torch.cuda.synchronize()
    end = time.perf_counter()
    print(i,":",end-start)
    cnt += (end-start)
    # print(outputs)
print(cnt)

```

### Expected behavior

The problem is no matter whether I pruned the model, the time seems to remain the same 

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep issues for bugs and feature requests only. I am not aware of any place in the doc where we advertise head-pruning as a mean to speed up inference. I think you will need to look at converting your model to ONNX or quantize it for that.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
changes for mbart_causal_lm
Author: amankhandelia
Pull request: yes

# What does this PR do?

This PR add FlaxMBartForCausalLM which was previously part of #19831, as discussed in [here](https://github.com/huggingface/transformers/issues/19897#issuecomment-1294648919), I am raising a separate PR for it.

Reason I want to add this model is, that it is a prerequisite to Donut model.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@sanchit-gandhi 

Comments:
@sanchit-gandhi please have a look when you have some time to spare, for some reason test did not ran. can you check the same and please re-initialize the test pipeline

Might need to fix style according to https://huggingface.co/docs/transformers/pr_checks#code-and-documentation-style

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Very different results on inference between mps and cpu for same input
Author: forrestdavis
Pull request: no

### System Info

transformers version: 4.22.2
Platform: macOS-12.5-arm64-arm-64bit 
Python version: 3.9.13
Huggingface_hub version: 0.10.0
PyTorch version (MPS?): 1.13.0.dev20221005 (True)


### Who can help?

@patil-suraj, @patrickvonplaten, @LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am running the following code: 

```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

model = GPT2LMHeadModel.from_pretrained('gpt2').to(torch.device('mps'))
inputs = tokenizer('the man who is tall',
                   return_tensors='pt').to(torch.device('mps'))

print(inputs)
outputs = model(**inputs).logits
print(outputs[0,0,:])

print()
print()

model = GPT2LMHeadModel.from_pretrained('gpt2').to(torch.device('cpu'))
inputs = tokenizer('the man who is tall',
                   return_tensors='pt').to(torch.device('cpu'))

print(inputs)
outputs = model(**inputs).logits
print(outputs[0,0,:])
```

What I see as output is:

```
{'input_ids': tensor([[1169,  582,  508,  318, 7331]], device='mps:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1]], device='mps:0')}
/opt/homebrew/Caskroom/miniforge/base/envs/mapi/lib/python3.9/site-packages/torch/_tensor_str.py:115: UserWarning: The operator 'aten::nonzero' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
  nonzero_finite_vals = torch.masked_select(
tensor([-14.9901, -14.6213, -17.5936,  ..., -21.4744, -21.1240, -14.7532],
       device='mps:0', grad_fn=<SliceBackward0>)


{'input_ids': tensor([[1169,  582,  508,  318, 7331]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])}
tensor([-33.1021, -31.8638, -35.0600,  ..., -38.2193, -38.8318, -32.7428],
       grad_fn=<SliceBackward0>)
```

### Expected behavior

I expect the output to be the same (or at least much closer). Am I missing something obvious here? 

Comments:
Interesting! @pcuenca could you take a look here? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
T5 should not use teacher-forcing when under evaluation
Author: ghost
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-6.0.2-76060002-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@patrickvonplaten 



### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce behaviour:

1. Find a task where the same token in the target sequence is repeated multiple times, e.g. IOB Tagging.
1. Train a t5 model on a task
3. Evaluate by predicting
4. Evaluate using generate and giving the same input_ids

### Expected behavior

The prediction task during evaluation should produce identical results to generate using num beams=1 ie greedy decoding.

Due to the nature of the task above, the target sequence has the form ((a|b|c){n})+, meaning sequences of target id repetitions, e.g. `000011110000`, due to teacher forcing, the model can learn to repeat the input token and quickly minimize the loss except for the boundary cases of `01` and `10`.

This is not visible when evaluating the model (e.g. via `.eval()`), and due to teacher forcing, the model gives excellent results. However, when using `.generate` with the same inputs, the model behaves horribly.

This is similar to 
#12488

Comments:
Please use the [forums](https://discuss.huggingface.co/) for discussions like this one as we keep issues for bugs and feature requests in the library.

Then perhaps the ability to turn off teacher-forcing could be listed as a feature for the model?

I don’t see how using teacher forcing in an auto regressive model during evaluation is not a bug.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.