noamaanMulla-03/transformers-issues huggingface dataset:
Trainer for devided model
Author: yeonju7kim
Pull request: no

### Feature request

Trainer class settings for divided model.

If I use trainer.predict, it loads the model on the device in the code of the trainer package.
It's hard to separately load the model manually.

I think it'll be better to implement to avoid loading the model twice.

### Motivation

https://huggingface.co/docs/accelerate/usage_guides/big_modeling
I read the above. Then I divided the LLM and loaded the model parts on the different GPUs.

My code used Trainer for training.
Because of the line below, the already loaded model loads again here on args.device.
https://github.com/huggingface/transformers/blob/21dc5859421cf0d7d82d374b10f533611745a8c5/src/transformers/trainer.py#L3072-L3076

I manually erased the line to use Trainer with the divided LLM. 
Is there any other way to use Trainer with a divided model? 

### Your contribution

I'm a novice for trainer class. I'm afraid that my question seems stupid.

Comments:
cc @muellerzr 

No, you cannot use the big model inference API to do training in this way (hence training in the name). You can use methods like DeepSpeed or FSDP to split the work between multiple GPUs or perform offloading during training, which the trainer already supports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update resuming with FSDP to work with different number of nodes
Author: jmzeng
Pull request: no

### Feature request

It would be great if it's possible to resume a FSDP run on single node to two or three nodes. When I try it currently, it would give the following warning and OOM:

`Didn't find an RNG file for process 8, if you are resuming a training that wasn't launched in a distributed fashion, reproducibility is not guaranteed.`

Would it be possible to add a flag to Trainer to redistribute the model if a particular RNG is missing? 

### Motivation

This would be a great feature for testing on a single machine and then scaling up the run on multiple machines. 

### Your contribution

I'm currently exploring an approach for this. 

Comments:
cc @muellerzr @pacman100 

cc @pacman100 

Hello @jmzeng, what is the FSDP config and launch command(s) along with the library versions? 

You can save the whole model ckpt instead of the sharded ckpts and then load it when changing the distributed setup from 1 nodes to 3 nodes for example. Did you try that?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Backward pass error during OWL-ViT finetuning
Author: pasqualedem
Pull request: no

### System Info

Google Colab

### Who can help?

@ArthurZucker 
@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1GfspAHLGTMmzfNShAVebzM04s32q1LKb?usp=sharing

### Expected behavior

I was trying to finetune Owl-ViT, and I came across this backward pass error. I investigated and found out that the error is the operator "/=" which does an inplace operation. 
At lines:

- https://github.com/huggingface/transformers/blob/4e8929dcbb9040f54f52d898a600260f338ef54f/src/transformers/models/owlvit/modeling_owlvit.py#L1296
- https://github.com/huggingface/transformers/blob/4e8929dcbb9040f54f52d898a600260f338ef54f/src/transformers/models/owlvit/modeling_owlvit.py#L1297

Comments:
@pasqualedem 

Would you like to open a PR to fix it ? 🤗 

Could you share your scripts for owlvit finetuning? Thanks @pasqualedem
T5Tokenizer: Different decoding behaviour depending on the tokenizer method used
Author: sorenmulli
Pull request: no

### System Info

- `transformers` version: 4.36.1
- Platform: Linux-6.1.55-1-lts-x86_64-with-glibc2.38
- Python version: 3.11.5
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1


- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```py
from transformers import T5TokenizerFast
tokenizer = T5TokenizerFast.from_pretrained("google/flan-t5-base")
tokens = ['▁', '?', '▁', '?']

ids = tokenizer.convert_tokens_to_ids(tokens)
# [3, 58, 3, 58]

tokenizer.decode(ids)
# '??'
tokenizer.convert_tokens_to_string(tokens)
# '? ?'
tokenizer.decoder.decode(tokens)
# '? ?'
```

### Expected behavior

I expected these two methods to yield same result: `'? ?'`.

I do not understand the result `'??'` and failed myself to find the logic where this space is removed; I guess it must be in `tokenizers`.

In advance, thank you for all help :heart: :hugs: 

Comments:
Also get same results with other `t5` models like `t5-base`; haven't tried other tokenizers though.

When using a non-fast tokenizer, I also get the same result. Here I am using `tokenizer.sp_model`instead of `tokenizer.decoder` in the last step. 

There must be a punctuated-based post-processing which I fail to see (and might be expected behaviour), as this also happens with tokens `.`. `.`, `!` but not `a` 

Hey! You should checkout the [clean_up_tokenization_spaces](https://huggingface.co/docs/transformers/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.clean_up_tokenization_spaces) attribute:
```python 
tokenizer.decode(ids, clean_up_tokenization_spaces=False)
```
is what you are looking for

Thanks! Then this is intended default behaviour :)

I think it then makes sense for me to disable this cleaning for my T5 model to get same result as sentencepiece but I cannot completely figure out if I want this cleanup in some other cases :thinking: 

Yep I have no clue either as to why this was always done but it's recurring issue 😅
[`Llava` / `Vip-Llava`] Add SDPA into llava
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, adds SDPA into Llava-family

This makes generation faster through torch sdpa for llava-like models

Also closes: https://huggingface.co/llava-hf/llava-1.5-7b-hf/discussions/9


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28107). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove warning if `DISABLE_TELEMETRY` is used
Author: Wauplin
Pull request: yes

In https://github.com/huggingface/transformers/issues/27564 I did some cleaning in the environment variables. I added a warning if `DISABLE_TELEMETRY` was set to encourage using `HF_HUB_DISABLE_TELEMETRY` instead. However this warning is not necessary for at least two reasons:
- `DISABLE_TELEMETRY` is already well understood and parsed by `huggingface_hub`. No need to handle it specifically in `transformers`. If in the future we want to deprecate it and/or handle it differently, everything would have to happen in `huggingface_hub` directly.
- Also, as highlighted in https://github.com/huggingface/huggingface_hub/issues/1917, keeping `DISABLE_TELEMETRY` in addition to our custom `HF_HUB_DISABLED_TELEMETRY` is also beneficial if this variable become a standard with other libraries. In any case, we do not benefit from not handling it.

Therefore this PR removes the deprecation warning + let `huggingface_hub` handle the environment variables by itself. It removes any custom code from `transformers` about this topic.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28113). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the quick review @amyeroberts!
More TF fixes
Author: Rocketknight1
Pull request: yes

The TF `build()` PR brought back an old issue where TF would latch onto the first concrete shape it saw, which would then become the model's save signature. We avoid it by hitting `self._set_save_spec()` with flexible shapes ASAP when models are created.

This PR also replaces a few more instances of `build()` with `build_in_name_scope()` in our tests. This should hopefully fix the CI issues (cc @ydshieh)

Comments:
Quick re-ping about this one @amyeroberts! (it's a very small PR, but I'd like to get it in so the CI stops upsetting Yih-Dar)
[docs] General doc fixes
Author: stevhliu
Pull request: yes

Cleans up a few things:

- Fused AWQ benchmark [table](https://huggingface.co/docs/transformers/main/en/quantization#fused-awq-modules) broken because there wasn't a blankline between the title and table
- tidies up the new sections added in the GPU inference doc
- removes the `NerPipeline` from internal discussion [here](https://huggingface.slack.com/archives/C02GLJ5S0E9/p1702480089513399) (basically it's identical to the `TokenClassification` pipeline and it may cause confusion)
- removes `perf_train_tpu.md` because it is empty and doesn't add any value to the docs

Comments:
Hi @amyeroberts! The CI error is:

```
Exception: The following objects are in the public init so should be documented:
 - NerPipeline
```

Would this be fixed by removing `NerPipeline` from the following?
- https://github.com/huggingface/transformers/blob/0d63d17765f954ba2b050c1d8be0001e952b7830/src/transformers/pipelines/__init__.py#L82

- https://github.com/huggingface/transformers/blob/0d63d17765f954ba2b050c1d8be0001e952b7830/src/transformers/__init__.py#L957

@stevhliu Yes, however this would stop NerPipeline being importable from the top level of transformers, which would be a breaking change.

Instead, we can make an exception for it in the doc checks by adding it to [DEPRECATED_OBJECTS](https://github.com/huggingface/transformers/blob/e6cb8e052a74313c2b2440c43df26303d379df71/utils/check_repo.py#L906) (which it effectively is if TokenClassificationPipeline supercedes it) 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28087). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix a typo in tokenizer documentation
Author: mssalvatore
Pull request: yes

# What does this PR do?

Fixes a typo in tokenizer documentation. For some methods, such as `tokenize()`, the description currently reads "Converts a string in a sequence of tokens, using the tokenizer." I believe what is meant is "Converts a string INTO a sequence of tokens".


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? (N/A)


## Who can review?

@ArthurZucker 
@sgugger 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28118). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix weights not properly initialized due to shape mismatch
Author: ydshieh
Pull request: yes

Currently, if there is some weight shape mismatched between the model and the checkpoint, and if `ignore_mismatched_sizes=True`, that/those weight(s) won't get initialized by the model's `_init_weights` method, and could get crazy values like `1e37`.

This could make the training gets nan loss value from the beginning and won't have any progress.

One example is by running `src/transformers/modeling_utils.py` (add `ignore_mismatched_sizes=True`).

We usually set `ignore_mismatched_sizes=True` when we want to perform classification tasks using an existing model but to another task having different number of targets.

This PR aims to fix this issue.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28073). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

As this touches the core file, request 2 reviews 🙏 

Guess I have to add some tests for this case. Change to draft for now
Fix indentation error - semantic_segmentation.md
Author: rajveer43
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

this PR removes the indentation error in code segment of sementic_segmentation.md file


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Documentation: @stevhliu and @MKhalusova






Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28117). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Heapify `BeamHypotheses`
Author: Wovchena
Pull request: yes

# What does this PR do?

Heapify `BeamHypotheses`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR. @gante

Comments:
cc @gante 

Hey @Wovchena 👋 

Thank you for opening the PR! We are planning, however, to change the structure of beam search, for better support with `torch.compile` and vectorization. As such, we are avoiding merging changes related to beam-search

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
AutoModel find_adapter_config_file does not use revision
Author: edbeeching
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-6.2.0-36-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No



### Who can help?

@younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have a (private) repo `alignment-handbook/zephyr-7b-sft-lora` with two model revisions `main` a peft adapter and `main.merged` the adapter weights merged with the based model. 
If I try to load the merged model with the following:
```
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("alignment-handbook/zephyr-7b-sft-lora", revision="main.merged")
```

I get an error related to revisions on the base model:
```
OSError: main.merged is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at 'https://huggingface.co/mistralai/Mistral-7B-v0.1' for available revisions.
```

I think the cause of this is `revision` kwarg is not passed to `find_adapter_config_file` [here](https://github.com/huggingface/transformers/blob/cf32c941350cb296e4c2c9e26a9274291d515e90/src/transformers/models/auto/auto_factory.py#L505C48-L505C48).


### Expected behavior

The correct merge model should be loaded.

Comments:
Hi @edbeeching you need to pass revision through `adapter_kwargs` as such:

```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("alignment-handbook/zephyr-7b-sft-lora", adapter_kwargs={"revision":"main.merged"})
```
So that we cover also the usecase where adapter weights and base model weights are on different revisions


Hi, as discussed on slack. The model I am loading is not an adapter. It is merged, the problems is that find_adapter_config_file is not using the model revision. Making PR with a fix now.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
TypeError: ResNet.__init__() got an unexpected keyword argument 'out_indices'
Author: wuxiaolianggit
Pull request: no

Hello, may I ask you a question? How to solve this problem about “TypeError: ResNet.__init__() got an unexpected keyword argument 'out_indices‘”？@ @vanpelt @tmm1 @pvl @tmc @arfon 

Comments:
Hi @wuxiaolianggit, thanks for raising this issue. 

So that we can help you, could you make sure to fill out the [bug report template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and include: 

* Minimal code reproducer
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output

Okay, thank you very much for your reply@amyeroberts 

I also encountered this error when I tried to load the `"timm/vit_small_patch14_dinov2.lvd142m"` model from Timm, this is how I called `timm.create_model`:
```python
timm.create_model(
                model_name="timm/vit_small_patch14_dinov2.lvd142m",
                features_only=False,
                pretrained=True,
                in_chans=3,
                out_indices=(2, 5, 8, 11),  # vit-s has total 12 blocks, I want those four level of features to be used as fpn for VitDet purpose
                norm_layer=FrozenBatchNorm2d,
            )
```

The `transformers-cli env` command return 
```bash
- `transformers` version: 4.35.0
- Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.10
- Python version: 3.8.18
- Huggingface_hub version: 0.19.1
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 1.11.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
```

The error I got is :
```bash
__init__() got an unexpected keyword argument 'out_indices'
```

@Goooyi  - the error encountered is coming from the `timm` library and is unreleated to `transformers`. It's coming from the fact [VisionTransfomer](https://github.com/huggingface/pytorch-image-models/blob/ef72c3cd470dd67836eebf95ec567199c890a6a2/timm/models/vision_transformer.py#L389) doesn't accept `out_indices` as an input argument. If you think this is an error or a feature that should be supported I'd suggest opening a new issue in that repo.  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.