noamaanMulla-03/transformers-issues huggingface dataset:
Assistance Exporting git-large to ONNX 
Author: gracemcgrath
Pull request: no

Hello! I am looking to export an image captioning Hugging Face model to ONNX (specifically I was playing with the [git-large](https://huggingface.co/microsoft/git-large) model but if anyone knows of one that might be easier to deal with in terms of exporting that is great too)

I'm trying to follow [these](https://huggingface.co/docs/transformers/serialization#exporting-a-model-for-an-unsupported-architecture) instructions for exporting an unsupported architecture, and I am a bit stuck on figuring out what base class to inherit from and how to define the custom ONNX Configuration since I'm not sure what examples to look at (the model card says this is a transformer decoder model, but it looks like i that it has both encoding and decoding so I am a bit confused)

I also found [this](https://github.com/huggingface/notebooks/blob/main/examples/onnx-export.ipynb) notebook but I am again not sure if it would work with this sort of model. 

Any comments, advice, or suggestions would be so helpful -- I am feeling a bit stuck with how to proceed in deploying this model in the school capstone project I'm working on. In a worst-case scenario, can I use `from_pretrained` in my application? 

Comments:
Questions around conversion to ONNX should go in the [optimum repo](https://github.com/huggingface/optimum) as  this is where the feature is actually implemented :-)

> Questions around conversion to ONNX should go in the [optimum repo](https://github.com/huggingface/optimum) as this is where the feature is actually implemented :-)

Thank you!! I will post this there.
Update tiny model creation script
Author: ydshieh
Pull request: yes

# What does this PR do?

Main changes:
- **Better error message**, including allowing to save the traceback to the report files
- Add `UNCONVERTIBLE_MODEL_ARCHITECTURES`, so for those models could not be converted to tiny versions, the code will skip them (and we have cleaner reports)
- Add a method `build_tiny_model_summary`, which will produce the entries we might want to add `tests/utils/tiny_model_summary.json` (for pipeline testing purpose)
  - (well, we might want to remove this file in the future) 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Difference in the architecture of openai whisper and huggingface whisper
Author: hannan72
Pull request: no

I was comparing the whisper-medium which I got from OpenAI directly (from https://github.com/openai/whisper.git) with the HuggingFace whisper-medium.
In the decoder part of the model, both has 24 decoder blocks but there is a difference in the block architecture between openai's and huggingFaces.
As it is represented below, In OpenAI whisper decoder, there is a GELU activation function between two linear layer (at mlp block) but in HuggingFace whisper decoder, there isn't that GELU activation function between fc1 and fc2 block.

![image](https://user-images.githubusercontent.com/8229163/224065417-aa2d7a8f-b385-4e18-975b-7638914ffa44.png)

So based on such difference, is the huggignface whisper model works similar to openai's one?
And is it a totally a difference between these two whisper-medium models?
@sanchit-gandhi 



Comments:
Hey @hannan72!

We register the activation function here (`activation_fn` in the state dict):
https://github.com/huggingface/transformers/blob/fdf84096565b8d2e15de35ac0cd86818c4b12adb/src/transformers/models/whisper/modeling_whisper.py#L468

And then apply it directly after the first feedforward layer (`fc1`):
https://github.com/huggingface/transformers/blob/fdf84096565b8d2e15de35ac0cd86818c4b12adb/src/transformers/models/whisper/modeling_whisper.py#L556

If you check the config for Whisper, you'll see that this activation function defaults to GELU:
https://github.com/huggingface/transformers/blob/fdf84096565b8d2e15de35ac0cd86818c4b12adb/src/transformers/models/whisper/configuration_whisper.py#L106

So the two are entirely equivalent 👍 OpenAI just register the GELU in a sequential block, we register it standalone. But both apply it in the same place.

Thank you a lot @sanchit-gandhi for your clear answer!
add progress bar to the sharded model download status
Author: stas00
Pull request: no

### Feature request

`from_pretrained('bigscience/bloom')` is taking forever the first time until it's cached (~350GB) - I thought that perhaps with 72 shards it'd be awesome to have an overall progress bar (in addition to the each shard download progress bar) to know where things stand and how many hours the coffee break should last.

Thank you!


Comments:
Nested tqdm bars in the console are unreadable though, so not sure how to fix that.

I didn't know that. I remember it definitely worked in the past. Unless you're referring to something I'm not aware of when you say it's unreadable. Is it because the outside tqdm line will be so far from line 72 that it won't be seen as it'd scroll past visible area? 

If that's the case then perhaps this would work:

1. switch to show and erase individual shard download progress as soon as it completed
2. only keep the overall progress bar 

so that there are only 2 lines dedicated to the progress updates.

But if none of this is doable nicely, perhaps at least using `desc` to number each shard's tqdm as in "43/72" would at least give some indication. Though it won't really help that much since one can't tell how much time it took to download the previous x entries.



Note that the description should already contain the name of the file, which ends with 0043-of-0072 normally.

I can have a look at what adding an overall progress bar would look like, but I don't have any control on the per-file progress bar, as it's issued by huggingface_hub. I could deactivate it entirely (so solution 2.) but I don't have control over it's closing.

> Note that the description should already contain the name of the file, which ends with 0043-of-0072 normally.

not for me:

![snapshot_49](https://user-images.githubusercontent.com/10676103/224100313-6f4c44e1-b7cd-4e5f-90f3-bd61d34e8c97.png)

> I can have a look at what adding an overall progress bar would look like, but I don't have any control on the per-file progress bar, as it's issued by huggingface_hub. I could deactivate it entirely (so solution 2.) but I don't have control over it's closing.

Understood! Thank you for looking, Sylvain!

Oh maybe you have an older version of huggingface_hub?

oh, I didn't know - you're correct - updating to 0.13 did add the filenames - that's much better. Thank you for that, Sylvain.


Can you try the PR mentioned above? I got confused and nested progress bars do appear nicely in the console. It's in notebooks that the result is messy.
Add a progress bar for the total download of shards
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a feature requested in #22047 and fixes a small bug I encountered while testing it.

**The feature**: a new progress bar is added when loading a sharded checkpoint that gives the overall progress.

**The bug**: when passing along `force_download=True`, the files were not downloaded if cached, because an early return in `cached_file` returned the cached file.

Fixes #22047 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm not observing any changes with this branch:

I tried:
```
python -c 'import sys; from transformers import AutoModel; AutoModel.from_pretrained(sys.argv[1], revision="sharded")' t5-11b
```

after deleting the cache and not getting a new overall progress bar. 

Am I doing something wrong or have some wrong dependencies?

Woops, wrong check for the file being cached. Can you try again?

It works great! Thank you, Sylvain!

As you can see it moves the outer progress bar with the inner bar, so it doesn't matter how many shards there are as I was concerned with 72 bloom shards.

![snapshot_50](https://user-images.githubusercontent.com/10676103/224163116-03d9ce60-959f-4064-9dd6-8324946cc06b.png)

![snapshot_51](https://user-images.githubusercontent.com/10676103/224163559-c4ce666f-8f15-4d76-89b6-2c6554a9dbb8.png)
Fixed docstring formatting
Author: koullouros
Pull request: yes

# What does this PR do?
Fixes the formatting in docstring for whisper model.
Fixes #22052 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Issue #22052 
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
 @sgugger, @stevhliu and @MKhalusova


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22067). All of your documentation changes will be reflected on that endpoint.
Edit the docstring of `image_processing_donut` to match code
Author: vermouthmjl
Pull request: yes

# What does this PR do?

It changes the list of arguments in the the docstring of the class `DonutImageProcessor`, for the current docstring does not match the list of parameters in the code.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Documentation: @sgugger
Donut: @amyeroberts @alaradirik 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

@vermouthmjl If running `make style` didn't resolve the `check_code_quality` checks, make sure the most recent formatting libraries are installed using `pip install -e .[quality]`
Fix gradient checkpointing bug in Speech2Text
Author: KMFODA
Pull request: yes

This PR fixes a bug that a user can encounter while using generate and models that use gradient_checkpointing.

Fixes Issue https://github.com/huggingface/transformers/issues/21737

cc @younesbelkada or @gante

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Sorry fixed that

@KMFODA your fork probably needs to pull from `main` :p
Fix small typo in flan-ul2.mdx
Author: kevin51jiang
Pull request: yes

# What does this PR do?

Saw small typo while reading. Not sure why the last line shows as changed; used GitHub's web UI and just modified "Resources". 

Thanks for making thie library!



<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->



## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @sgugger, @stevhliu and @MKhalusova


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Force-merging this without the tests as I'm 99.9% sure everything will be fine, but for future PR, there is an issue with your CircleCI permissions, the tests won't run.
You will need to refresh your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?
Broken link in Documentation 
Author: datavistics
Pull request: no

### System Info

@sgugger 

Where I started
https://huggingface.co/docs/transformers/v4.26.1/en/main_classes/text_generation#transformers.GenerationMixin.contrastive_search

What doesnt exist
https://huggingface.co/docs/transformers/v4.26.1/en/main_classes/generation_strategies
https://huggingface.co/docs/transformers/main/en/main_classes/generation_strategies

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Go to the links above

### Expected behavior

The pages would exist.

Comments:
Hey @datavistics -- the problem is not present on `main`, but rather in the previous release (which we can't fix unless we make a new release) 

See these docs: https://huggingface.co/docs/transformers/main/en/main_classes/text_generation

I still get the broken link here: https://huggingface.co/docs/transformers/en/main_classes/text_generation#transformers.GenerationMixin.contrastive_search

Which links to v4.26.1 in one of the links.

Should be fixed by #22078 on main.
Fix gradient checkpointing bug in Speecht5
Author: KMFODA
Pull request: yes

This PR fixes a bug that a user can encounter while using generate and models that use gradient_checkpointing.

Fixes Issue https://github.com/huggingface/transformers/issues/21737

cc @younesbelkada or @gante

Comments:
Sorry fixed that

_The documentation is not available anymore as the PR was closed or merged._

@KMFODA it is possible that this is the wrong branch -- the diff doesn't contain changes to speecht5 :)

Yes the diff seems to have changed a bit, can you please double check 🙏 @KMFODA ?

Sorry yes I think the branches were mixed up. This should now introduce the changes to SpeechT5 and fix the formatting issues @gante spotted in [modeling_speech_to_text.py](https://github.com/huggingface/transformers/pull/22080/commits/4da5f4baaf4eddb71787ba4dbbbd3791953e481a).

Perfect, thank you for the changes @KMFODA 💛
handle numpy inputs in whole word mask data collator
Author: dwyatte
Pull request: yes

# What does this PR do?

Adds support to `DataCollatorForWholeWordMask` to work on numpy arrays as inputs. I added tests for all variants (np, pt, tf), but only tf had the bug which is now fixed.

Fixes https://github.com/huggingface/transformers/issues/22009

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@gante 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@Rocketknight1, can you have a look too plz? You have been working with these :) Then we can tag Sylvain, after you approve too
Fix imports of TF MobileViT
Author: sgugger
Pull request: yes

# What does this PR do?

Small cleanup in the main init.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

It's only the type hints that are wrong, not the actual init.

> It's only the type hints that are wrong, not the actual init.

Yeah, just figure it out after I posted. Deleted it but you are too fast to answer

Failures are related to Hub being down, so no blocker to merge.
Revert "[GPT2] Propose fix for #21080"
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#21853

A few PT/TF and PT/Flax cross tests started to fail after #21853. Revert that PR for now. We need to look what went wrong (and why it is not reported in the PR CI)

```
FAILED tests/models/gpt2/test_modeling_tf_gpt2.py::TFGPT2ModelTest::test_pt_tf_model_equivalence - AssertionError: 3.225274 not less than or equal to 1e-05 : outputs.last_hidden_state: Difference between torch and tf is 3.225274085998535 (>= 1e-05).
FAILED tests/models/encoder_decoder/test_modeling_tf_encoder_decoder.py::TFGPT2EncoderDecoderModelTest::test_pt_tf_model_equivalence - AssertionError: 0.3753911 not less than or equal to 1e-05 : outputs.logits: Difference between torch and tf is 0.3753910958766937 (>= 1e-05).
FAILED tests/models/vision_encoder_decoder/test_modeling_tf_vision_encoder_decoder.py::TFViT2GPT2EncoderDecoderModelTest::test_pt_tf_model_equivalence - AssertionError: 0.42845678 not less than or equal to 1e-05 : outputs.logits: Difference between torch and tf is 0.42845678329467773 (>= 1e-05).
```

A job run with failed tests is [here](https://app.circleci.com/pipelines/github/huggingface/transformers/59606/workflows/fd5cb028-f3df-4ac6-83ae-b0cc68396af8/jobs/728991)



Comments:
Tested locally - all good now. Will merge once CI is green too (already discussed offline)
SearchSummarizationPipeline - 'object has no attribute' error
Author: arun-ar
Pull request: no


I am facing the below error while initializing SearchSummarizationPipeline. I am using latest haystack version and python 3.10.   


```
pipe3 = SearchSummarizationPipeline(summarizer=summarizer,retriever=retriever,generate_single_summary=False,return_in_answer_format=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\wing\AppData\Local\Programs\Python\Python310\lib\site-packages\haystack\pipelines\standard_pipelines.py", line 424, in __init__
    self.pipeline.add_node(component=summarizer, name="Summarizer", inputs=["Retriever"])
  File "C:\Users\wing\AppData\Local\Programs\Python\Python310\lib\site-packages\haystack\pipelines\base.py", line 424, in add_node
    component_definitions[name] = component._component_config
AttributeError: 'SummarizationPipeline' object has no attribute '_component_config'
```

Comments:
How to resume training with a different learning rate or else TrainingArguments？
Author: Bardbo
Pull request: no

I tried the following settings, which will be covered by the learning rate of checkpoint

```
training_args = TrainingArguments(
    learning_rate=5e-4, # larger than before 5e-5
）
...
trainer.train(resume_from_checkpoint=True)```

Comments:
Please use the [forums](https://discuss.huggingface.co/) for surch questions as we keep issues for bugs and feature requests only.The resume training functionality is intended in case of instance crash and you should use the same hyperparameters.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
OOM when running causal language modelling sample
Author: rmc135
Pull request: no

The example at:

https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling

...mentions taking half an hour on a K80. I'm using an M40 24GB, which I would reasonably believe has sufficient VRAM, but it dies with OOM at startup. I used the exact commandline shown on the page.

Dropping -per_device_train_batch_size and --per_device_eval_batch_size from 8 to 4 succeeds, but it's still using ~15GB of VRAM

Unsure whether the WikiText-2 dataset has changed, or transformers has changed, or the model has changed, or there's something different about VRAM usage (eg default data type size) between the K80 and M40 24GB.

I'm just beginning with transformers fine-tuning which is why I was running the example commandline. Apologies if I'm missing something obvious.

Ubuntu 22.04 LTS
Python 3.10.6 running under venv
transformers 4.27.0.dev0-py3.10.egg

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
getting hidden_states in a causal menner
Author: amit-sofer
Pull request: no

### Feature request

i want to use the roberta model in the following way:
given a list of N tokens, i want the model to compute a hidden_state for each of the N tokens in a causal way, meaning the first token hidden_state is computed based only on the first token, the second hidden_state is computed based on the first two tokens, the third hidden_state is computed based on the first three tokens and so on.
additionally, i want a CLS token that his hidden_state will be computed based on all the input tokens.
there seems like there is no flag or input which will make this. or is there?



Comments:
You should use the [forums](https://discuss.huggingface.co/) for a question like this, as we keep issues for bugs and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.