noamaanMulla-03/transformers-issues huggingface dataset:
Custom training for LayoutLMv3 for document answering task
Author: NevilleMthw
Pull request: no

### System Info

- `transformers` version: 4.32.0
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hello,

I am not having a specific type of bug but I would like to know if there is any sort of guide towards a custom dataset training for LayoutLMv3, I am following through this link but I don't see how someone with another dataset can follow: https://huggingface.co/docs/transformers/tasks/document_question_answering#evaluation

Is there any standard for the JSON structure? Is there a specific way to import the data so that we can use the existing functions?

Thanks.

### Expected behavior

Ability to fine tune with custom dataset instead of Docvqa dataset.

Comments:
See my demo notebook here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb.

It's equivalent for LayoutLMv3, just make sure to prepare the data in the right format

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Implementation of Git-Rebasin to Huggingface models
Author: NamburiSrinath
Pull request: no
Labels: name: New model

### Feature request

Hi, 

I am not sure if this is even feasible/appropriate request in Huggingface. Recently, there was an ICLR paper titled "Git Rebasin" (https://arxiv.org/abs/2209.04836) which demonstrates a method to effectively merge models.

But the authors have implemented it only for plain NN and Resnet kind of architectures. It would be really great if this feature is supported for Huggingface models as there'll be lots of instances where we might need to merge the models and a naive way of adding is not the best way.   

### Motivation

This feature is helpful especially in federated learning where 2 different people train 2 models separately and would like to combine their models. 

In academic circles as well, this is very much useful as there'll be lots of instances where effective model merging is critical! 

### Your contribution

I've raised PR's in subsequent Github repos

1. https://github.com/samuela/git-re-basin/issues/13#issue-1910375545
2. https://github.com/themrzmaster/git-re-basin-pytorch/issues/8#issue-1910392068

Not sure if this is feasible based on some of the previous discussions as the paper is based on "Permutation Invariance" of model weights and an architecture like Llama might have various possibilities thus difficulty in the method.

I would be happy to contribute if there's some help!

Comments:
Bring back `set_epoch` for Accelerate-based dataloaders
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR brings back the `set_epoch` logic and solves the last remaining regression from the accelerate integration. Resuming training should now be exactly identical. (Introduced due to an incomplete implementation in https://github.com/huggingface/transformers/pull/24028)

Linked with https://github.com/huggingface/accelerate/pull/2057

Fixes # (issue)

fixes https://github.com/huggingface/transformers/issues/26541


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik @ArthurZucker 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Bump`flash_attn` version to `2.1`
Author: younesbelkada
Pull request: yes

# What does this PR do?

FA-2.0 seems to cause several issues such as https://github.com/huggingface/transformers/issues/26697 / https://github.com/huggingface/transformers/issues/27056 

As discussed offline, I propose to bump the FA-2 version to at least 2.1 to overcome these issues in the future

cc @ArthurZucker 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Llama FA2] Re-add _expand_attention_mask and clean a couple things
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR cleans the attention mask converter a bit more, corrects some docstrings and removes outdated comments and deprecates `_expand_attention_mask` to fix optimum.

Comments:
@ArthurZucker could you give this a quick review? It'd make the Bart FA PR much easier to continue and should also fix the better transformers problem with optimum

_The documentation is not available anymore as the PR was closed or merged._

Of course!
Add AMD daily CI workflow file
Author: ydshieh
Pull request: yes

# What does this PR do?

AMD daily CI workflow file. There are something to do with the runners and report channels, but nothing is big.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Correct docstrings and a typo in comments
Author: lewis-yeung
Pull request: yes

## What does this PR do?

Correct docstrings of the following methods:

- `TrainingArguments.set_save`
- `TrainingArguments.set_logging`

## Who can review?

@stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27047). All of your documentation changes will be reflected on that endpoint.
TensorBoard integration on huggingface
Author: omermazig
Pull request: no

### System Info

I'm running a somewhat changed version of [this](https://github.com/huggingface/notebooks/blob/main/examples/video_classification.ipynb) tutorial, and somewhere last month.

As the tutorial explains, it used to be that when I ran `push_to_hub`, a TensorBoard was created and appeared under the model page on huggingface:

![image](https://github.com/huggingface/transformers/assets/95534441/af510d37-4641-4ad0-8669-7da18b64e90b)


But lately I've noticed that it has stopped generating that tab. I have two runs from the 20 and 24 of Aug, where the tab appears in the former and doesn't appear in the latter:

https://huggingface.co/omermazig/videomae-base-finetuned-kinetics-finetuned-nba-data-8-batch-5-epochs-dataset_10_classes
https://huggingface.co/omermazig/videomae-base-kinetics-finetuned-nba-binary-data-2-batch-50-epochs-399-train-vids-multilabel

You can see that the transformer's verion updated from `4.31.0` to `4.32.0`.  All runs before these two had a `Training Metrics` tab, and all runs after them didn't.

Is it possible that the integration was broken on `4.32.0`? How can I further investigate that?

Thanks!

### Who can help?

@Rocketknight1

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run this notebook:

https://github.com/huggingface/notebooks/blob/main/examples/video_classification.ipynb

### Expected behavior

Creation of a `Training Metrics` tab on huggingface

Comments:
cc @Rocketknight1 our TF expert!

Hmmm indeed it seems like there is an issue with `runs` not being saved. I don't think that's related to TensorFlow but rather to the `Trainer`. 

@pacman100 @muellerzr could you please give this issue a look? Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This issue has gone stale even though it still needs to be addressed, so I'm commenting to bump it as instructed 

Thanks, this will be solved with #27022 as long as you do `trainer.push_to_hub()`
Ported Dinov2 to flax
Author: ifeherva
Pull request: yes

# Ported the Dinov2 model to jax/flax

This PR adds the dinov2 model in flax. It is based on the vit flax port but uses the existing pytorch dinov2 as base.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

@sanchit-gandhi @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25579). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @ifeherva! Given you made a great start on this PR, it would be super nice if you were able to see it to completion! On hand to help with any questions/queries 🤗 Of course if you are busy, there's no pressure to finish this. In this case, we can open it up to the community to see if anyone is able to finish the integration so that this work is merged into main
Skip unfixable failing tests on RoCm
Author: fxmarty
Pull request: yes

A bug in nn.Conv2d in PyTorch 2.0.1 on RoCm systems make the output of TVLT significantly diverge vs CPU / Nvidia GPUs. The issue is fixed on nightly and on 2.1.0 RC.

Some slow tests may be affected as well, thus marking as draft for now. Adding relevant tensors / code to reproduce the issue just in case.
[conv2d_mi210.zip](https://github.com/huggingface/transformers/files/12652047/conv2d_mi210.zip)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26231). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Custom positional embeddings for whisper
Author: bcl213124314
Pull request: no
Labels: name: wontfix

### Feature request

Add the ability to pass custom positional encodings from the generate() call of the ConditionalGeneration model down to the forward() of the WhisperEncoder.

### Motivation

Currently the processor pads the audio input to 3000 mel frames (30 s). Encoding all 3000 frames is the main computational cost of transcribing short audio segments which are a lot shorter than 30 s. The only place where this fixed number of  3000 frames is currently hard coded into the model is:
```
embed_pos = self.embed_positions.weight
hidden_states = inputs_embeds + embed_pos
```

in the forward method of the whisper encoder, since embed_position.weight has a fixed shape. 

Custom positional embeddings would allow the inference of shorter inputs, decreasing the computational cost of the encoding.
Of course that is not what the model was originally trained with and, depending on the custom positional encodings one might use, this changes the output of the model dramatically.
Short tests have shown that just setting 
`embed_pos = self.embed_positions.weight[0:inputs_embeds.shape[1]]`
leads to reasonable predictions.
Better engineered schemes to generate embeddings for shorter inputs might lead to even better results.



### Your contribution

This is a pretty small change and I have implemented it [here](https://github.com/bcl213124314/transformers/tree/encoder_pos_embeddings)

Comments:
You should have a look at #25744 😉 This is not planned 

Thanks for the link. Sorry for not searching enough.

No worries 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
example code problem
Author: iplayfast
Pull request: no
Labels: name: Feature request

while trying the example from https://huggingface.co/amazon/MistralLite This is the result.

python examplecode.py 
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
  File "/home/chris/ai/text-generation-webui/amazonmistral/examplecode.py", line 8, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_id,
  File "/home/chris/anaconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
    return model_class.from_pretrained(
  File "/home/chris/anaconda3/envs/textgen/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3076, in from_pretrained
    config = cls._check_and_enable_flash_attn_2(config, torch_dtype=torch_dtype, device_map=device_map)
  File "/home/chris/anaconda3/envs/textgen/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1265, in _check_and_enable_flash_attn_2
    raise ValueError(
ValueError: The current architecture does not support Flash Attention 2.0. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/new


Comments:
This is fixed by #26464 and will be in the next release
Revert "add exllamav2 arg"
Author: ArthurZucker
Pull request: yes

Reverts huggingface/transformers#26437 in order to have a proper deprecation cycle cc @SunMarc 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I think we can open an issue and can just do a follow up PR no?

Let's give @SunMarc some more time not sure if he'll get through this before next week and let's not forget about this! 🤗 

OK sounds great!
Fix no split modules underlying modules
Author: SunMarc
Pull request: yes

# What does this do ?

This PR fixes `_get_no_split_modules` introduced in #26949. Basically, we don't check the children modules if they appear in `_no_split_modules`. By doing that, we can decide to not split a children module (can be a `PretrainedModel`) without having to check if its `_no_split_modules` is set or not. This is particularly useful as we don't necessarily want to set it (small model, can't be split ...). 

This PR along with this [one](https://github.com/huggingface/transformers/pull/27089) should also fix the issues in the CI 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`core`/ `gradient_checkpointing`] Refactor GC - part 2
Author: younesbelkada
Pull request: yes

# What does this PR do?

Extends https://github.com/huggingface/transformers/pull/27020 by further simplifying the GC enable / disable mechanism. We can simply iterate over all submodules of the `PreTrainedModel` and check for the attribute `gradient_checkpointing`. 

Some models had `supports_gradient_checkpointing` attribute set to `True` whereas they actually don't. So this PR fixes that as well.

Some models were also calling `torch.utils.checkpointing.checkpoint` instead of `self.gradient_checkpointing_func`, this PR fixes it.

Also `gradient_checkpointing` is now private to avoid exposing it as a public attribute

cc @ArthurZucker 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ArthurZucker @LysandreJik - as discussed offline now this PR reverts back the previous behaviour (i.e. if a user sets `module.gradient_checkpointing = True` in a module that supports it, everthing should work fine) + I have set `gradient_checkpointing_func` as a private attribute. This PR is ready for review
fix detr device map
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR solves an issue that #26849 introduced. For `DeformableDetrForObjectDetection` and `DetaForObjectDetection`, we can't initialize the model on `meta` device as we modify the weights during the initialization. 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yes, not every `ForObjectDetection` models needs this. For example, the `TableTransformerForObjectDetection` don't modify the weights at initialization. I will add a comment.
add exllamav2 arg
Author: SunMarc
Pull request: yes

# What does this PR do ?
This PR adds the possibility to choose exllamav2 kernels for GPTQ model. This PR follows the [integration](https://github.com/PanQiWei/AutoGPTQ/pull/349 ) of the kernels in auto-gptq and the [integration]((https://github.com/huggingface/optimum/pull/1419)) in optimum.

- [x]  Merge after the optimum PR is merged 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> What about changing disable_exllamav2 to disable_exllama_v2 ?

I've named it that way in auto-gptq, so let's keep `disable_exllamav2`

> Can you also add few lines in the documentation adding some expected speedups and how to use this argument? You can copy paste the table you shared in the auto-gptq PR

I added a link to the optimum benchmark and I will update the benchmark in a follow-up PR . It will be easier to maintain this way.
Provide alternative when warning on use_auth_token
Author: Wauplin
Pull request: yes

This PR updates the warning message when a user uses `use_auth_token` instead of `token`. In some cases, the deprecation warning states that `use_auth_token` is deprecated without mentioning `token` as an alternative. This PR fixes this.

Opened this issue after some users got confused they wouldn't be able to authenticate anymore (see [Forum](https://discuss.huggingface.co/t/the-use-auth-token-argument-is-deprecated-and-will-be-removed-in-v5-of-transformers/53943)). cc @radames who pinged me there.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Simply added 4 words, nothing more :wink: 

Well, 3 reviews so I'll merged! :smile: Thanks everyone :)

thanks for the prompt response @Wauplin
Resize failing with torch and np objects
Author: rafaelpadilla
Pull request: no

### System Info

`transformers.__version__: '4.34.1'`

### Who can help?

@rafaelpadilla @amyeroberts 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python 
import torch
from transformers.image_transforms import resize

# dummy torch image
img = torch.zeros((3, 100,100), dtype=torch.uint8)
# any new size
new_size = (20, 20)
resized_img = resize(img, new_size)

>>> Traceback (most recent call last):
>>>   File "<stdin>", line 1, in <module>
>>>   File "/home/rafael/anaconda3/envs/hf/lib/python3.11/site-packages/transformers/image_transforms.py", line 326, in resize
>>>     do_rescale = _rescale_for_pil_conversion(image)
>>>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>   File "/home/rafael/anaconda3/envs/hf/lib/python3.11/site-packages/transformers/image_transforms.py", line 139, in _rescale_for_pil_conversion
>>>     elif np.allclose(image, image.astype(int)):
>>>                             ^^^^^^^^^^^^
>>> AttributeError: 'Tensor' object has no attribute 'astype'. Did you mean: 'dtype'?
```

### Expected behavior

`resize()` should be flexible and accept any image type  (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`) as in [here](https://github.com/huggingface/transformers/blob/main/src/transformers/image_transforms.py#L289).

However, it fails if image type is torch.

The reason occurs because the `_rescale_for_pil_conversion()` is being called [here](https://github.com/huggingface/transformers/blob/main/src/transformers/image_transforms.py#L326) and assumes that the input `image` is `np.array`.



Comments:
Hei! This seems to be quite straightforward. I can work on this if help is needed.

Hi @daniilgaltsev ,

Thanks for your interest in fixing it. 

I had just spoken with @amyeroberts , one of our core maintainers, and she explained me that this is actually not a bug.
In fact, the image transformations functions like `resize`, `normalize`, `center_crop`, etc should work with numpy arrays. So, this is just a very small docstring and typing fix. 

In [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/image_transforms.py#L289), we just needed to replace
```python
image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
```
by
```python
image (`np.ndarray`):
```
If you still want to contribute, please, feel free to take it 🤗 
Also, please, tag your PR with `#Close #26986`.