noamaanMulla-03/transformers-issues huggingface dataset:
Update some torchscript tests after #24505
Author: ydshieh
Pull request: yes

# What does this PR do?

Need to update the logic in some torchscript tests after #24505

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger Thanks for the suggestion. I actually copied from the common test file. However, there is 

```python
            model_buffers = list(model.buffers())
            for non_persistent_buffer in non_persistent_buffers.values():
                found_buffer = False
                for i, model_buffer in enumerate(model_buffers):
                    if torch.equal(non_persistent_buffer, model_buffer):
                        found_buffer = True
                        break

                self.assertTrue(found_buffer)
                model_buffers.pop(i)
```
there (which I didn't see when working on this PR). This uses the `values`.

So I am going to copy the above block (and therefore keeping using dict). Let me know if you have other opinions instead.

If you end up using the values, no worries.

Copy the mentioned block to all `_create_and_check_torchscript` definition.

Let's not change this by looking if the model use persistent or non-persistent buffuer: just keep the logic in the common test file.
Removal of deprecated vision methods and specify deprecation versions
Author: amyeroberts
Pull request: yes

# What does this PR do?

* Removes a bunch of properties, methods and logic which had deprecation warnings for a few versions ago.
* Adds some specific version deprecations for methods that didn't have a specified version for deprecation


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @sgugger Just wanting to double check it's OK to remove these deprecated methods now before I merge in
Fix ESM models buffers
Author: sgugger
Pull request: yes

# What does this PR do?

Apparently the keys are important to be loaded from the state dict even if we create them at init.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24576). All of your documentation changes will be reflected on that endpoint.
save all custom files in the meanwhile
Author: JaheimLee
Pull request: no

### Feature request

Many models in your huggingface hub have their custom files, like [mpt](https://huggingface.co/mosaicml/mpt-7b-instruct). However, your `save_pretrained` func can't save them all. Is it possible to save all of them at the same time?

### Motivation

It's pretty useful when you fine-tune the model and want to save it in a new directory.

### Your contribution

no

Comments:
Hi @JaheimLee, thanks for raising this issue. 

Which files specifically are being referred to when you say "custom files"? 

> Hi @JaheimLee, thanks for raising this issue.
> 
> Which files specifically are being referred to when you say "custom files"?

Like the model files which is not merged into transformers. For example:
https://huggingface.co/mosaicml/mpt-7b-instruct/blob/main/configuration_mpt.py
https://huggingface.co/mosaicml/mpt-7b-instruct/blob/main/adapt_tokenizer.py
https://huggingface.co/mosaicml/mpt-7b-instruct/blob/main/modeling_mpt.py
...

The `save_pretrained` function indicates a reference to the original repo so that you get the updates they push there automatically. You can still use your model with `from_pretrained` and push it to the Hub, and it will always work and keep the latest code version from their side.

If you really want the modeling files to make changes, you can find them all in the cache: `~/.cache/huggingface/hub/models--mosaicml--mpt-7b-instruct`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[DO NOT MERGE] Add Crop Transformation
Author: amyeroberts
Pull request: yes
Labels: name: WIP

# What does this PR do?

Abstracts out cropping logic to be a more generic `crop` function which other, more specific cropping functions e.g. `center_crop` can call. 

Motivation: 
* The output of the CLIP feature extractor changed after #17628. This was due to a difference in how the `top` and `left` coordinates were calculated resulting in some values being off by one. 
* The original CLIP feature extractor matched the original implementation
* Having a more generic `crop` method enables each image processor to have its own center_crop logic with minimal code replication. 

[BEFORE MERGING]: Verify this doesn't have large impact on any popular CLIP dependant pipelines

Fixes #22505


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22608). All of your documentation changes will be reflected on that endpoint.
Check all objects are equally in the main `__init__` file
Author: ydshieh
Pull request: yes

# What does this PR do?

Add one more check in `check_repo.py` to further ensure objects are in the main `__init__`. (using our pytorch objects as the reference)

### Actions to take
(Do you agree @sgugger ?)

- The following should be added to the main `__init__`:

```bash
TFAutoModelForAudioClassification should be defined in the main `__init__` file.
TFAutoModelForMaskGeneration should be defined in the main `__init__` file.
TFAutoModelForMaskedImageModeling should be defined in the main `__init__` file.
```

- A fix is required:
```bash
TF_AutoModelForSemanticSegmentation should be defined in the main `__init__` 
file.
```
due to the mistake
```
TF_AutoModelForSemanticSegmentation = auto_class_update(
    TFAutoModelForSemanticSegmentation, head_doc="semantic segmentation"
)
```

- The following (the **pytorch** one) should be **removed** from the main `__init__`:
```bash
TFBertLayer should be defined in the main `__init__` file.
FlaxBertLayer should be defined in the main `__init__` file.
FlaxBigBirdLayer should be defined in the main `__init__` file.
TFLxmertEncoder should be defined in the main `__init__` file.
TFLxmertXLayer should be defined in the main `__init__` file.
TFMPNetLayer should be defined in the main `__init__` file.
TFMobileBertLayer should be defined in the main `__init__` file.
FlaxRoFormerLayer should be defined in the main `__init__` file.
TFSegformerLayer should be defined in the main `__init__` file.
TFViTMAELayer should be defined in the main `__init__` file.
```



Comments:
_The documentation is not available anymore as the PR was closed or merged._

We cannot remove things from the main init, as it would be a breaking change. The `BertLayer` and the likes should never have been made public like this but it's too late now. And we shouldn't add the corresponding TF/Flax objects either, so this defeats the purpose of this new check.

I can add exceptional cases list though and not to touch those problematic entries.

Sure!
Fix annotations
Author: tony9402
Pull request: yes

# What does this PR do?

This PR is minor. Just fixed wrong annotations.
All models were tested and confirmed based on Pytorch model.

I think all the codes I modified are made based on the template.
Please check if it is okay to modify the annotation in the template code.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
fix peft ckpts not being pushed to hub 
Author: pacman100
Pull request: yes

# What does this PR do?

1. Currently, when ckpts are saved every `save_steps` and `push_to_hub=True`, PEFT ckpts aren't being pushed to hub.
Reason:
every `save_steps` , the ckpts are being written to local `checkpoint-xx` folders. Now, in the trainer's function `_push_from_checkpoint` , it will copy the model files from the latest ckpt folder `checkpoint-xx` to the `output_dir`  which gets pushed to the Hub. Note that the modeling_files doesn't have `ADAPTER_WEIGHTS_NAME`  or `ADAPTER_SAFE_WEIGHTS_NAME`  or `ADAPTER_CONFIG_NAME`  leading to them not being pushed.

This PR fixes it. 

cc @younesbelkada 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Udate link to RunHouse hardware setup documentation.
Author: BioGeek
Pull request: yes

# What does this PR do?

The [hardware setup](https://runhouse-docs.readthedocs-hosted.com/en/main/rh_primitives/cluster.html#hardware-setup) link gives a 404 error. Replaced with a [link](https://runhouse-docs.readthedocs-hosted.com/en/latest/api/python/cluster.html#hardware-setup) that points to the latest version of the RunHouse documentation.


## Before submitting
- [ X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).



## Who can review?

Documentation: @dongreenberg, @sgugger, @stevhliu and @MKhalusova




Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks!! I'll update the one in Accelerate too.
Token Alignment
Author: akesh1235
Pull request: no

`data`
> DatasetDict({
    train: Dataset({
        features: ['input', 'output'],
        num_rows: 4500
    })
    test: Dataset({
        features: ['input', 'output'],
        num_rows: 500
    })
})

**# input (in-correct sentence)**
`data['train'][0]['input']`
**>>** 'We are meet sunday 10am12pmET in Crown Heights Brooklyn New York'
**# output (correct sentence)**
`data['train'][0]['output']`
**>>** 'We meet Sundays 10am-12pmET in Crown Heights, Brooklyn, New York.'

**I Want to align the output tokens with input**

```
`# tokenize both inputs and targets
def tokenize_fn(batch):
    # tokenize the input sequence first
    # this populates input_ids, attention_mask, etc.

    tokenized_inputs = tokenizer(
    batch['input']
    )
    
    labels_batch = tokenizer.tokenize(batch['output']) # original targets

    aligned_labels_batch = []
    for i, labels in enumerate(labels_batch):
        word_ids = tokenized_inputs[i].word_ids()
        aligned_labels_batch.append(align_targets(labels, word_ids))  # align_targets is another user defined function which is been called here

    # recall: the 'target' must be stored in key called 'labels'
    tokenized_inputs['labels'] = aligned_labels_batch

    return tokenized_inputs`
```
```
data.map(
  tokenize_fn,
  batched=True,
  remove_columns=data['train'].column_names,
)
```

When this user defined function is mapped to every records of train and test batch am getting following error:

**1.** **raise DatasetTransformationNotAllowedError(
   3457         "Using `.map` in batched mode on a dataset with attached indexes is allowed only if it doesn't create or remove existing examples. You can first run `.drop_index() to remove your index and then re-add it."**

**2.** **TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]**

Comments:
Looks like this should go on the Datasets repo, t doesn't seem linked to Transformers :-)

> Looks like this should go on the Datasets repo, t doesn't seem linked to Transformers :-)

Could you help me in anyway to fix this?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
feat: add warning for using use_pretrained_backbone with from_pretrained
Author: CreatlV
Pull request: yes

# What does this PR do?

This PR adds a warning when using `from_pretrained` and you also have `use_pretrained_backbone` enabled. `from_pretrained` will load its weights after the pretrained backbone is initialized meaning some weights may be unexpectedly overridden. I am not sure if this case is general enough to warrant a warning. The more general case would be if any weights are loaded before the `from_pretrained` weights. An additional feature would be to allow selectively loading weights using `from_pretrained`, but I understand if that use case is too esoteric. 

```
model = DetrForObjectDetection.from_pretrained(
    "facebook/detr-resnet-50",
    num_labels=len(categories.keys()),
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
    num_queries=20,
    backbone="resnet50",
    use_pretrained_backbone=True,
    use_timm_backbone=True,    
)
```

For the scenario above, the `use_pretrained_backbone` will not have any effect as the `facebook/detr-resnet-50` weights will take precedent. 

## Who can review?

Perhaps this is the area of @sgugger or @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23690). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[RWKV] Inference memory leak unless use_cache=False is specified
Author: rsbf
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): 2.13.0-rc0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help?

@sgugger 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Memory usage rapidly climbs when running a stripped-down version of the RWKV overview example [here](https://huggingface.co/docs/transformers/model_doc/rwkv#overview), in a loop:

```python
from transformers import AutoTokenizer, RwkvModel

model = RwkvModel.from_pretrained("sgugger/rwkv-430M-pile")
tokenizer = AutoTokenizer.from_pretrained("sgugger/rwkv-430M-pile")


for _ in range(1000):
        inputs = tokenizer("This is an example.", return_tensors="pt")
        # Feed everything to the model
        model(inputs["input_ids"]).  #  <--- memory leak 
```

Passing `use_cache=False` to the forward step solves this, though it's not clear why, since the cached state 'should' be bounded to 5 entries. 



### Expected behavior

Stable memory usage

Comments:
This is because you are not executing your forward pass under a `torch.no_grad`, so the gradient history blows up via the state of the outputs. Either do this or manually detach the states to avoid this memory use. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to check word ids for BartTokenizer? 
Author: akesh1235
Pull request: no

one single words is getting splitted to multiple tokens, and i tried to check word_ids by using **t.word_ids()** 
i got an Error  
**ValueError: word_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).**

Comments:
As the error indicates, you need to use `BartTokenizerFast` to use this method.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
About Tokenizer 
Author: Yu-xm
Pull request: no

### System Info

### Who can help?

@ArthurZucker @younesbelkada @ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

tokenizer = BertTokenizer.from_pretrained(gpt2_path)

### Expected behavior

I see the following code:

tokenizer = BertTokenizer.from_pretrained(gpt2_path)

This code uses BertTokenizer to read GPT2 related files. What is the difference between the above code and the following two codes?

tokenizer = BertTokenizer.from_pretrained(bert_path)
tokenizer = AutoTokenizer.from_pretrained(gpt2_path)

Comments:
Please use the [forums](https://discuss.huggingface.co/) for questions around the library as we keep issues for bugs and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
AutoTokenizer Encode Error
Author: congyingxia
Pull request: no

### System Info

For the LlamaTokenizer, I can get correct encoding result when directly loading from LlamaTokenizer. But the results are incorrect when using AutoTokenizer.  Another issue is loading the AutoTokenizer much slower than directly loading the LlamaTokenizer. It take around 4 mins to load the tokenizer from the path when using AutoTokenizer, while it only takes one second if directly using the LlamaTokenizer. 

### Who can help?

@ArthurZucker

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Python version: 3.8.16
transformers version: 4.28.1

Follow the given example:

```
from transformers import LlamaTokenizer, AutoTokenizer

model_path = 'openlm-research/open_llama_7b_700bt_preview'
str = ' is embarassed, because Samantha made snide comments about the shirt Rebecca was wearing.'

tokenizer1 = LlamaTokenizer.from_pretrained(model_path)
tokenizer2 = AutoTokenizer.from_pretrained(model_path)

ret1 = tokenizer1.encode(str, add_special_tokens=False)
ret2 = tokenizer2.encode(str, add_special_tokens=False)

print(ret1)
print(ret2)

```


### Expected behavior

ret1: [322, 2661, 285, 14363, 31844, 906, 23982, 985, 3668, 483, 4309, 562, 266, 13803, 15136, 393, 7732, 31843]
ret2: [31822, 322, 2661, 285, 14363, 31844, 906, 23982, 985, 3668, 483, 4309, 562, 266, 13803, 15136, 393, 7732, 31843]

ret1 is the expected output and ret2 is an error result from AutoTokenizer. AutoTokenizer add an additional token, 31822 (which is a space token), to the encoding results.


Comments:
Hey! You are using an old version of the tokenizer. You should be using the one available [here](https://huggingface.co/huggyllama/llama-7b). This issue was already fixed. 

AutoTokenizer has to convert the slow tokenizer to a fast one, which takes of course a lot of time since the model was not saved on the shared repo. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Speed up TF tests by reducing hidden layer counts
Author: Rocketknight1
Pull request: yes

A lot of our slow TF tests are caused by TF compilation. TF compilation isn't really affected by layer width at all - the main thing is just the number of operations it has to build a graph for. By reducing the number of hidden layers, compilation gets much faster, (hopefully) without interfering with test coverage at all.

Comments:
Would be nice if you can show the timing for one model (before v.s. after) 🙏 . Thanks.

_The documentation is not available anymore as the PR was closed or merged._

@ydshieh testing locally BERT went from 510 seconds -> 220 seconds

I don't know if it was @sgugger either - a lot of this code is really old! I see `tf.tuple()` in there, and even I had to look up the TF 1.x docs to remember what that was supposed to do, lol

I know he is probably not the one to decide use `5`, but he might know the history :-)