noamaanMulla-03/transformers-issues huggingface dataset:
Fix switch transformer mixed precision issue
Author: timlee0212
Pull request: yes

# What does this PR do?

Fix the issue of dtype mismatch when using mixed precision training for switch transformer model. 

Fixes #27219 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] **[No Update Required]** Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] **[No New Test Required]** Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@younesbelkada 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27220). All of your documentation changes will be reflected on that endpoint.

Hmm per my understanding it seems we do not have CI tests for mixed precision training. 
I think that we could easily adapt the current existing `test_training` : https://github.com/huggingface/transformers/blob/af3de8d87c717c4bb090f037d0d89413c195a42f/tests/test_modeling_common.py#L581 test to also cover the mixed precision case (bf16 should work on our CI runners) - happy to work on that on a separate PR!

@younesbelkada Yes please! 

@younesbelkada  I used Accelerate for mixed precision training. I think it should use torch.cuda.amp as the default backend.
[`Docs` / `SAM` ] Reflect correct changes to run inference without OOM 
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/27266

One needs to always wrap the model forward pass to avoid OOM issues for SAM forward pass. In the orignal modeling code they force-use `torch.no_grad()` context manager in the forward pass: https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py#L53  - but we avoid that since we also support SAM fine-tuning. 

Check out my comment here for more details: https://github.com/huggingface/transformers/issues/27266#issuecomment-1792508065

cc @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27268). All of your documentation changes will be reflected on that endpoint.
[Docs] Model_doc structure/clarity improvements
Author: MKhalusova
Pull request: yes

This PR streamlines pages in model_doc, so that a user who has seen at least one of these pages, should find it easier to navigate throughout any other model_doc page. These are the changes:

1. Individual pages now have (with a few exceptions) the same structure: overview, followed by author acknowledgement, usage tips and/or examples, resources (other docs, notebooks, blog posts, etc.), API reference (autodoc). Previously, some pages did not have the same order which could make searching through them confusing.
2. Usage tips now have their own section with a header. Previously, they often have been buried within the overview section, now one can find them faster.
3. In cases where a model has the same architecture as another model, a visible `<Tip>` has been added with a link to API reference of the original model. 
4. “Documentation resources” and “resources” were combined into “Resources”
5. For models that support TF and JAX, a `<framework>` tag was added so that a user could collapse sections irrelevant to them.
6. Removed disclaimers like “This model has been recently added. If you see something strange, file a GitHub Issue” for models that have been in the library longer than 1 year.

The PR aims to:
- Make it easier to find usage tips and examples
- Make it easier to find API reference for models that are based on another model’s architecture
- Possibly reduce the bloat by letting the users collapse framework-specific sections
- Streamline the model_doc pages structure


Comments:
_The documentation is not available anymore as the PR was closed or merged._

If @ArthurZucker is busy, maybe @amyeroberts could take a look?
GPT-J specific half precision on CPU note
Author: MKhalusova
Pull request: yes

This PR adds a note to the GPT-J model doc indicating that the half precision on CPU example is specific to the model, and doesn't generally apply. Related to https://github.com/huggingface/transformers/issues/21989

Comments:
_The documentation is not available anymore as the PR was closed or merged._

My bad, I misunderstood the interaction in the issue. Tried the example, and it does indeed not work. I'll do another pass.

Did another take. It should now be clear that half-precision works on CUDA devices only. 

Btw, not sure if this is relevant but I tried this example without explicitly sending to device on a GPU, and it threw the same error.
Depth estimation task guide
Author: MKhalusova
Pull request: yes

This PR adds a zero-shot depth estimation task guide that covers inference with a pipeline, as well as manually, for monocular depth estimation supported by DPT and GLPN.

Comments:
PR with images: https://huggingface.co/datasets/huggingface/documentation-images/discussions/64

_The documentation is not available anymore as the PR was closed or merged._
Example of pad_to_multiple_of for padding and truncation guide & docstring update
Author: MKhalusova
Pull request: yes

This PR adds a minor update to the docs as previously it was not clear that `pad_to_multiple_of` has to be used with `padding=True`.
Based on https://huggingface.slack.com/archives/C027NLU6CE9/p1679325764920509


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for iterating!
Support loading pre-trained LayoutLM models with device_map argument
Author: leloykun
Pull request: yes

# What does this PR do?

Loading pre-trained LayoutLMv2 & LayoutLMv3 models with, say `LayoutLMv3Model.from_pretrained(..., device_map="auto")`, currently fails with error message: `ValueError: LayoutLMv3ForTokenClassification does not support `device_map='auto'`. To implement support, the modelclass needs to implement the '_no_split_modules' attribute.`

This PR just implements the `_no_split_modules` attribute to the LayoutLMv2 & LayoutLMv3 model classes as suggested by the error message.


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@NielsRogge, @ArthurZucker 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[AMD] Enable nightly ci
Author: mfuntowicz
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26440). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Skip part of the dataset when training with trainer
Author: young-chao
Pull request: no

When I used the trainer to train, I encountered some bad data that caused the loss to increase abnormally. I wanted to temporarily skip this part of the data. I directly skipped "self.optimizer.step()" in some steps but it did not take effect. , the trajectory of loss is still consistent with before. I want to know how to skip some steps appropriately.

Comments:
![image](https://github.com/huggingface/transformers/assets/34190033/816ba98c-6417-4983-971f-00f28d6879ab)


Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Timm to ViT Model conversion fix #26219
Author: prabhuteja12
Pull request: yes

A problem with timm to Vit converter code was identified in #26219 

This is caused by a rather strict index based parsing of the timm model identifier which fails for several cases. This PR does two things:

1. Fixes that issue by using a regex parser for patch and image size. 
2. Fixes ViT configuration differences (ViT Small, missing Vit-Tiny, Vit-giant/gigantic). 



 @amyeroberts, @ArthurZucker


Comments:
@rwightman do you agree with this change?

Just wanted to check in if a review has been possible. 

It looks like an improvement on the original w/ better coverage, but there are less 'hard coded' ways of grabbing the needed dims and params from the timm model, avoiding model dependent ifs, etc

Can you please give me more details on the less hard coded ways? The simplest way I could think of was to use regex. 

@prabhuteja12 all of the values you need should be extracted without relying on the model names, use combination of model/submodule attributes and parameter shapes. All needed widths, depths, patch/img sizes, can be extracted that way.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Chat Template kwargs
Author: Rocketknight1
Pull request: yes

This PR adds an extra `chat_template_kwargs` attribute, so that the behaviour of templates can be modified without rewriting the whole thing. I factored out the messy LLaMA default system message stuff in a cleaner way using these, and I'm going to need it for the templates of some other checkpoints on the Hub as well!

This is implemented in a backward-compatible way, so no existing code should break because of it.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26185). All of your documentation changes will be reflected on that endpoint.

That's a fair point! I noticed when I was browsing the Hub that several models had chat templates that needed variable substitutions that don't work cleanly with the current template. [Here's](https://huggingface.co/PygmalionAI/pygmalion-6b) a good example - the `<CHARACTER>` variable needs to be filled in somehow, which would force the user to rewrite the template for every character.

Thinking about it more, though, maybe for a model like that extra information would need to be supplied e.g. as extra keys in the conversation inputs instead?

~I'm going to resurrect this PR - I think template kwargs are the right way to handle another thing I want to add: The ability to include a 'generation prompt' in the template.~

Scratch that, I think it makes more sense as a separate argument to `apply_chat_template` - I'll make a new PR!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
translate autoclass_tutorial to chinese
Author: jiaqiw09
Pull request: yes

# What does this PR do?

Part of  #26803 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? _not necessary_


## Who can review?
@stevhliu

Comments:
@stevhliu

hi, here is another pr. I will fix merge conflict later.

Besides, I think final insturction of `AutoModelForTokenClassification` and `TFAutoModelForSequenceClassification` is a bit repetitive.  I think it's better to reduce some introduction and just show differences.

Best

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27269). All of your documentation changes will be reflected on that endpoint.
translate run_scripts.md to chinese
Author: jiaqiw09
Pull request: yes

# What does this PR do?

Part of  #26803 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? _not necessary_


## Who can review?
@stevhliu

Comments:
@stevhliu

hi,

here is another PR of run_scripts.md, and I will fix merge conflict later.

Best

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27246). All of your documentation changes will be reflected on that endpoint.

@stevhliu 

reviews and merge conflict have been sloved.

best
translate peft.md to chinese
Author: jiaqiw09
Pull request: yes

# What does this PR do?

Part of  #26803 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? _not necessary_


## Who can review?
@stevhliu

Comments:
@stevhliu  

hi, here is another pr for peft.md translation.

And I just check the  _toctree.yml file, and I think I will first finish section `Tutorials` and `Developer guides`. I think it may be helpful for people who want to try `transformers`. Then I will continue to work on anther sections. When we finsihed all translation work with other contributors, I think it's better to look back and update some link info, small typos or wrong translation words.

Just try to do some work by the end of this year. Many people is using transformers as it's the trend for DL work, it's better to give more detailed docs with variety language. And I think more people will do contribution when they can get enough help from amounts of translated docs.

Best

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27215). All of your documentation changes will be reflected on that endpoint.

@stevhliu 

Hi, I have fixed problems.

And for `<br>`, I just think it is too compact to read easily. But I still delete this line.

Best
Escape special tokens
Author: imoneoi
Pull request: no

### Feature request

Can we introduce escape special tokens functionality in Tokenizer and TokenizerFast to ignore special tokens (treat them as plain text)?

### Motivation

`tiktoken` has an allowed special token option. This enables the processing of arbitrary user input, which may accidentally include special tokens.

### Your contribution

Nothing yet

Comments:
This is already supported by `split_special_tokens` and is a duplicate of the issue in `tokenizers`. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
request of num_proc for all files in load_dataset
Author: pphuc25
Pull request: no

### Feature request

adding num_proc for all files in load_dataset, and also download_config add datasets.DownloadConfig('num_proc=...')

### Motivation

With the ever-increasing volume of data in today's world, the process of downloading and extracting data has become excessively time-consuming. It is imperative to optimize and execute these tasks in a more efficient and expeditious manner.

### Your contribution

#26326 #26457

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix daily CI image build
Author: ydshieh
Pull request: yes

# What does this PR do?

This is currently failing due to `intel_extension_for_pytorch`. The post fix in the version like `intel_extension_for_pytorch==1.11.0+cpu` is no longer working 4 days ago. So I remove it.

Also update the version to `2.1.0`.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix tokenizer export for LLamaTokenizerFast
Author: mayank31398
Pull request: yes

# What does this PR do?
Currently, LLamaTokenizerFast is not exporting `add_bos_token` and `add_eos_token`. This will fix this issue.

Fixes # (issue)
https://github.com/huggingface/transformers/issues/23833
https://github.com/huggingface/transformers/pull/23855#issuecomment-1789796762

this is not saving add_eos_token and add_bos_token.
@sgugger @ArthurZucker Can you guys take a look at this?


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
- @ArthurZucker @sgugger

Comments:
@mayank31398 Could you add the same fix to [code llama's fast tokenizer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/code_llama/tokenization_code_llama_fast.py)? 

done @amyeroberts 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27222). All of your documentation changes will be reflected on that endpoint.

@amyeroberts can you merge this?
this is blocking us :)
Question: Resources to implement random & greedy sampling, beam search?
Author: EricLBuehler
Pull request: no

Hello everybody,

I want to implement random sampling, greedy sampling, and beam search, but cannot seem to find any useful resources on how to implement them.

Could you please provide some resources or code examples (perhaps from this repo) on how to implement the sampling strategies above?

Comments:
Hi @EricLBuehler - thanks for raising an issue! 

I can see that the issue is closed but for anyone in future visiting this thread, details on customising generation strategies can be found here: https://huggingface.co/docs/transformers/generation_strategies
Remove an unexpected argument for FlaxResNetBasicLayerCollection
Author: pingzhili
Pull request: yes

# What does this PR do?

Remove an unexpected argument `activation` which shouldn't be passed to FlaxResNetBasicLayerCollection.

Fixes #27257


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@amyeroberts Thanks!


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27272). All of your documentation changes will be reflected on that endpoint.