noamaanMulla-03/transformers-issues huggingface dataset:
the accuracy issue of left padding and right padding
Author: hijkzzz
Pull request: no

### System Info

transformers v4.38.2
docker container:  nvcr.io/nvidia/pytorch:23.12-py3

### Who can help?

@ArthurZucker 
@younesbelkada

### Information

The output results of the left and right pading are inconsistent

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# any llama2 model
modelname = "OpenLLMAI/Llama-2-7b-sft-model-ocra-500k"
model = AutoModelForCausalLM.from_pretrained(modelname).cuda()

# left pad
inputs={'input_ids': torch.tensor([[    1,  7251,   727, 29901, 29871],
        [    2,     2,     1, 29871, 29896]]).cuda(), 'attention_mask': torch.tensor([[1, 1, 1, 1, 1],
        [0, 0, 1, 1, 1]]).cuda()}
# right pad
inputs2={'input_ids': torch.tensor([[    1,  7251,   727, 29901, 29871],
        [    1, 29871, 29896,     2,     2]]).cuda(), 'attention_mask': torch.tensor([[1, 1, 1, 1, 1],
        [1, 1, 1, 0, 0]]).cuda()}

# baseline
output = model(**inputs)
output2 = model(**inputs2)

output2.logits[1][:3] - output.logits[1][-3:]
tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0020, -0.0010, -0.0001,  ...,  0.0006,  0.0013,  0.0007],
        [ 0.0025,  0.0040, -0.0005,  ...,  0.0025,  0.0015,  0.0008]],
       device='cuda:0', grad_fn=<SubBackward0>)

# fixed positions
position_ids = inputs['attention_mask'].long().cumsum(-1) - 1
position_ids2 = inputs2['attention_mask'].long().cumsum(-1) - 1

output = model(**inputs, position_ids=position_ids)
output2 = model(**inputs2, position_ids=position_ids2)

output2.logits[1][:3] - output.logits[1][-3:]
tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 9.3555e-04, -8.1062e-06, -8.5831e-05,  ...,  7.9441e-04,
          5.7936e-04,  4.6229e-04]], device='cuda:0', grad_fn=<SubBackward0>)
```
### Expected behavior

no accuracy issue

Comments:
related issue: https://github.com/OpenLLMAI/OpenRLHF/issues/217

A duplicate of #25921 and #25420 I am going to close this, feel free to read this great comment: https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535 
TLDR: it is expected when you pad inputs
Convert SlimSAM checkpoints
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR extends the conversion script of SAM (Segment Anything) to also support [SlimSAM](https://github.com/czg1225/SlimSAM/tree/master) checkpoints.

SlimSAM is a compressed (pruned) version of SAM, claims to outperform FastSAM and MobileSAM. Looks cool! Below are the SAM-vit-base (top) versus the SlimSAM-77-uniform (bottom) results.

<img width="1254" alt="Screenshot 2024-01-07 at 20 59 40" src="https://github.com/huggingface/transformers/assets/48327001/588bdec0-86b8-4669-8e27-88232a379243">

see https://github.com/czg1225/SlimSAM/issues/4


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28379). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix OneFormer `post_process_instance_segmentation` for panoptic tasks
Author: nickthegroot
Pull request: yes

# What does this PR do?

Fixes the `post_process_instance_segmentation` method of OneFormer. 

When specifying both `task_type="panoptic"` and `target_size` in `post_process_instance_segmentation`, the `i` variable would be overloaded by an enumeration. This would cause line 1265 (specifying `target_size`) to blow up and throw `IndexError: list index out of range`. This PR fixes the variable overloading in the two places it could occur.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten @NielsRogge

> ☝️  Based on the original reviewers for OneFormer.

Comments:
> Thanks for fixing!
> 
> Could you add a test for `post_process_instance_segmentation` as well?

@amyeroberts added in a3f370b! Mind taking another look?
4.36 transformers got wrong _save_checkpoint  with deepspeed. work with previous versions
Author: tszdanger
Pull request: no
Labels: name: solved

### System Info

transformers: 4.36.0
python: 3.10
deepspeed:  0.9.4                   

Traceback (most recent call last):
  File "/home/xxxx/xxxx/src/./xxxx.py", line 363, in <module>
    main(args)
  File "/home/xxxx/xxxx/src/./xxxx.py", line 352, in main
    run_training(args, train_dataset, eval_dataset, len(tokenizer))
  File "/home/xxxx/xxxx/src/./xxxx.py", line 339, in run_training
    trainer.train(resume_from_checkpoint=args.resume_from_checkpoint)
  File "/home/xxxx/miniconda3/envs/xxxx/lib/python3.10/site-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
  File "/home/xxxx/miniconda3/envs/xxxx/lib/python3.10/site-packages/transformers/trainer.py", line 1914, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/xxxx/miniconda3/envs/xxxx/lib/python3.10/site-packages/transformers/trainer.py", line 2274, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/home/xxxx/miniconda3/envs/xxxx/lib/python3.10/site-packages/transformers/trainer.py", line 2383, in _save_checkpoint
    os.rename(staging_output_dir, output_dir)
FileNotFoundError: [Errno 2] No such file or directory: '/home/xxxx/tmp-checkpoint-100' -> '/home/xxxx/checkpoint-100'



**When Using deepspeed with Trainer, 4.36 just have the wrong code, see below:**


https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2352C13-L2352C83

You may need a PR for this.


### Who can help?

@muellerzr @pacman100

Like below. 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

quite simple, just with the simplest code example.

trainer = Trainer(
      model=model,
      args=training_args,
      train_dataset=train_data,
      eval_dataset=val_data,
  )
  print("Training...")
  trainer.train(resume_from_checkpoint=args.resume_from_checkpoint)





### Expected behavior

no error would be fine

Comments:
See this PR: https://github.com/huggingface/transformers/pull/28009
It can help overcome this issue.

Glad to see this issue has been fixed :) 

> See this PR: #28009 It can help overcome this issue.

it not worked!

@jiezhangGt could you provide a reproduer / make sure you are using the latest version of main?
Permission Denied Error When Saving Checkpoint During Training
Author: Tanman2001
Pull request: no

### System Info

- `transformers` version: 4.37.1
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.1
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- PyTorch version (GPU?): 2.2.1+cpu (False)

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Follow exactly the Image Classification example in the HuggingFace docs: https://huggingface.co/docs/transformers/tasks/image_classification
2. Error thrown at `trainer.train()` 
![TrainingError](https://github.com/huggingface/transformers/assets/32078791/728d7ed0-8f3c-4ef5-8a0f-9ec5e2f403a6)


### Expected behavior

Should train normally.

Issue and possible solution identified on forum earlier this year: https://discuss.huggingface.co/t/permission-issues-when-saving-model-checkpoint/70109/2

Removing (or commenting out, like pictured below) lines 2417-2420 of transformers/trainer.py resolves issue without further errors, but may have complications for other tasks.

![TrainerCommentedOut](https://github.com/huggingface/transformers/assets/32078791/3292819d-83e9-40bb-8ddd-a5da2eecff7b)

Comments:
Fix grad_norm unserializable tensor log failure
Author: svenschultze
Pull request: yes

# What does this PR do?
Fixes Issue #29207 

@muellerzr 
@pacman100 

Comments:
The issue seems to be that deepspeeds get_global_grad_norm() unexpectedly returns a tensor instead of a float.
This fixes that issue. For Non-Deepspeed training, we already call item(), so there is no fix required.

Actually, this seems to be in direct conflict with https://github.com/huggingface/transformers/pull/29158

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29212). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@muellerzr In this case, which PR should stand?

> Actually, this seems to be in direct conflict with #29158

That other PR seems to affect the `else` clause where, whereas this is in the `if` part of the clause, so it might be okay

This one should stand and be okay, the other PR will need to change, so this is fine to me
`spectrogram` function may silently produce incorrect values for certain inputs
Author: xenova
Pull request: no
Labels: name: WIP

### System Info

- `transformers` version: 4.36.0.dev0
- Platform: Linux
- Python version: 3.8.3
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.3.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@sanchit-gandhi @hollance

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code for reference:

https://github.com/huggingface/transformers/blob/62ab32b2997846526cdffd629c72c47bc7b4f215/src/transformers/audio_utils.py#L451-L457

If `power=None` and `mel_filters != None`, then we will be performing `np.maximum` between 1 array of real numbers and 1 array of complex numbers. Since complex numbers do not have an ordering, `np.maximum` will only look at the real parts (for some reason, that is how it is implemented):
```py
>>> import numpy as np
>>> np.maximum(np.array([1,2,3]), 5j)
array([1.+0.j, 2.+0.j, 3.+0.j])
```

As a result, the function will continue without throwing an error.

Possible fixes:
1. If `power=None`, still perform `np.abs()`
2. Throw an error if this combination of arguments is provided


It should be noted that although I haven't found a processor which uses the function with these inputs, it may cause an issue in future, or for users of the spectrogram function directly.

### Expected behavior

An error should be thrown, or the absolute value should be taken if power is None.

Comments:
Feel free to open a PR for a fix 😈 

cc @ylacombe would you be able to take a look at protecting against this edge case?
Avoid edge case in audio utils
Author: ylacombe
Pull request: yes

# What does this PR do?

Throw an error if the spectrogram is of complex values but `mel_filters` are passed.


Fixes #27772

cc @sanchit-gandhi @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28836). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add link to free code-along on HuggingFace
Author: filipsch
Pull request: yes

This PR adds a link to a free resource for those wanting to get acquainted with HuggingFace without any of the configuration hassle and with an accompanying video to not get stuck.

This PR fixes a typo or improves the docs.

Comments:
Hey @NielsRogge, any chance you can have a look? Thanks

Hi @filipsch, thanks for your PR and the awesome video, however we don't accept resources from third-parties in our documentation. :( feel free to share about it on social media, which we could amplify.

Kind regards,

Niels 

@NielsRogge understood! Thanks for considering it!
Convert PyTorch Model to Hugging Face model
Author: Freemanlabs
Pull request: no

I have looked at many resources but I still seem to have issues/
I have a "model.pt" file which I got from finetuning a BERT model (with additional custom layesr added). I want to upload this to the hugging face hub. How can I convert the `.pt` model to files/model that can be used on hugging face hub?

Comments:
Hi, we have guide which explains how to share custom PT models:, which should help with this. https://huggingface.co/docs/transformers/custom_models

Thank s for your response. However, I do not seem to grasp the resource. It looks like it requires implementing the code from the start. In my own case, I already have a `.pt` file.

I am not sure what part of the docs handles this.

I will appreciate any further help.

Hi @Freemanlabs , it would be best to use the [forum](https://discuss.huggingface.co/) for this question. We use issues for bug reports and feature requests. Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Freemanlabs @patil-suraj Were you able to Convert a PyTorch Model (model.pt) to Hugging Face model? I am also trying to also do the same thing, and If so, i would appreciate any help.
[New Model] UDOP: Unifying Vision, Text, and Layout for Universal Document Processing
Author: WaterKnight1998
Pull request: no
Labels: name: New model

### Model description

We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 9 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark (DUE).

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation
UDOP Paper: https://arxiv.org/abs/2212.02623
UDOP Repo: https://github.com/microsoft/UDOP

UDOP Model Weights: https://huggingface.co/ZinengTang/Udop/tree/main

Comments:
@NielsRogge as you implemented Donut, you might be interested :)

Let's hope they open-source :)

@NielsRogge they added the code here https://github.com/microsoft/i-Code/tree/main/i-Code-Doc

Hi @NielsRogge, can I help in this implementation?

@NielsRogge here you have the weights: https://huggingface.co/ZinengTang/Udop/tree/main

@WaterKnight1998 Is the model accessible now?

> @WaterKnight1998 Is the model accessible now?

No, the PR from @raghavanone was closed. @NielsRogge is working on opening a PR with a refactor of UDop code as it was not very good.

I saw he has a branch for this: https://github.com/NielsRogge/transformers/tree/add_udop
Mra models have `nan` in output last_hidden_states
Author: zucchini-nlp
Pull request: no

### System Info

When adding tests for batching equivalence ([PR](https://github.com/huggingface/transformers/pull/29297)), it was found that Mra model has `nan` in outputs. I found the exact place where this happens, after the SparseDenseMatmul in [this line](https://github.com/huggingface/transformers/blob/0ad770c3733f9478a8d9d0bc18cc6143877b47a2/src/transformers/models/mra/modeling_mra.py#L429), tensors start containing `nan` values. 

When running tests for Mra, some configurations have to be tweaked as noted in [this comment](https://github.com/huggingface/transformers/pull/24513#issuecomment-1667010570)

### Who can help?

@amyeroberts , tagging you here for tracking

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Model tests

### Expected behavior

Find out why we are having `nan`

Comments:
Thanks for opening @zucchini-nlp and digging into this in depth. 

In particular, linking to the relevant comment from the tests. Based on that, I don't think there's really much we can do here - having `nans` was a compromise for MRA in order to have a faster running test suite. With that in mind, I think we can close the issue, and just skip the batching test for MRA, linking to this issue in the reason.
FIX [`Generation`] Fix some issues when running the MaxLength criteria on CPU
Author: younesbelkada
Pull request: yes

# What does this PR do?

Hey @zucchini-nlp @gante

In TRL we do have a class that call `generate` with `MaxLengthCriteria` and running the CI on main seems to cause some issues after https://github.com/huggingface/transformers/pull/29116 😱 
Not sure how to repro apart from running the failing CI and I did not flagged anything alarming outside my CI but to be on the safe zone I propose to init the `torch.full` tensor in bool directly, I assume it's fine according to the type hint

Here is the traceback:
```bash
self = [<transformers.generation.stopping_criteria.MaxLengthCriteria object at 0x7f652ccf8810>]
input_ids = tensor([[31373,   995, 19277],
        [31373,   995, 27455],
        [31373,   995,  4562],
        [31373,   995, 26964]])
scores = None, kwargs = {}, is_done = tensor([False, False, False, False])
criteria = <transformers.generation.stopping_criteria.MaxLengthCriteria object at 0x7f652ccf8810>

    @add_start_docstrings(STOPPING_CRITERIA_INPUTS_DOCSTRING)
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> torch.BoolTensor:
        is_done = torch.full((input_ids.shape[0],), False, device=input_ids.device)
        for criteria in self:
>           is_done = is_done | criteria(input_ids, scores, **kwargs)
E           RuntimeError: "bitwise_or_cpu" not implemented for 'Float'
```

https://github.com/huggingface/trl/actions/runs/8058401333/job/22011132917



Comments:
cc @zucchini-nlp @gante what do you think about these changes ? 🙏 I can also spend some time on trying to repro with a minial snippet but if these changes look good to you perhaps we can move forward with it

PS: and for some reason I can't request review @zucchini-nlp 🤯 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29317). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Okay for me, but it's weird that max_length criteria was returning a float tensor

@zucchini-nlp `torch.full(xxx)` returns a float tensor by default unless you don't explicitly cast it or init the tensor with the desired dtype. The thing I am not sure about is that why it didn't get flagged on other tests
🌐 [i18n-KO] Translated `semantic_segmentation.md` to Korean
Author: jungnerd
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다! -->
# What does this PR do?

Translated the `semantic_segmentation.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에, 이 아래에 리뷰를 요청할 팀원들을 멘션해주세요! -->
Team PseudoLab, may you please review this PR? @0525hhgus, @kihoon71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? @sgugger, @ArthurZucker, @stevhliu -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26515). All of your documentation changes will be reflected on that endpoint.