noamaanMulla-03/transformers-issues huggingface dataset:
Add bounding box converter
Author: qubvel
Pull request: yes
Labels: name: WIP

# What does this PR do?

Intorduces `output_bbox_format` parameter for `postprocess_for_object_detection`.
Adds the `convert_boxes` function to convert boxes from one format to another, e.g. from YOLO to Pascal VOC.

```python
boxes = convert_boxes(
    boxes, input_format="relative_xcycwh", output_format="absolute_xyxy", image_size=target_sizes
)
```

 - works with `np.array`, `torch.tensor`, `list`, `tuple` (could be easily extended to `jax` and `tf`)
 - works with single box shape=(4,) , image boxes shape=(N, 4), multiple images boxes List[(N, 4)]
 - supports boxes with extra metadata, e.g. `box=[x_min, y_min, x_max, y_max, color, class, ...]`
 - supports format aliases. e.g. `yolo`, `coco`
 
Supported input/output bounding box formats:
    - `absolute_xyxy` (aliases: `pascal_voc`, `xyxy`): [x_min, y_min, x_max, y_max]
    - `absolute_xywh` (aliases: `coco`, `xywh`): [x_min, y_min, width, height]
    - `absolute_xcycwh`: [center_x, center_y, width, height]
    - `relative_xyxy` (aliases: `albumentations`): [x_min, y_min, x_max, y_max] normalized to [0, 1] by image size
    - `relative_xywh`: [x_min, y_min, width, height] normalized to [0, 1] by image size
    - `relative_xcycwh` (aliases: `yolo`, `xcycwh`): [center_x, center_y, width, height] normalized to [0, 1] by image size

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@amyeroberts could you please look at this draft, looks a bit overсomplicated, probably we should drop some functionality in favor to simplification.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30852). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Adding the WIP label so the stalebot doesn't bother
[`ConvertSlow`] make sure the order is preserved for addedtokens
Author: ArthurZucker
Pull request: yes

# What does this PR do?
@zucchini-nlp caught a nice bug: the order of the user defined tokens and control tokens was not preserved

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31902). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<img width="342" alt="image" src="https://github.com/huggingface/transformers/assets/48595927/1d6cd40f-8481-48ea-9e17-da92ac54818f">
tests are failing because this is the sentencepiece.model that was uploaded
fix: Fixed the `1st argument` name in classmethods
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Updated the first argument of classmethods to `cls` as per the recommended naming convention of PEP8


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts

Comments:
[Quickfix] Update deprecated method `_register_pytree_node` to `register_pytree_node`.
Author: jacklanda
Pull request: yes

# What does this PR do?
Nothing too big, this PR aims at updating the deprecated method `_register_pytree_node` to `register_pytree_node`.

Fixes #31903


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts, @muellerzr and @SunMarc

Comments:
> Hi @jacklanda, thanks for opening this PR!
> 
> Unfortunately, we can't make this change as-is, as we officially support pytorch >= 1.11 and in [v1.11.0 register_pytree_node hasn't yet been added](https://github.com/pytorch/pytorch/blob/v1.11.0/torch/utils/_pytree.py).
> 
> We can try and do something, where we track the version it's introduced, and use the public function in this case e.g.
> 
> ```
> import ....
> 
> if is_torch_greater_or_equal_than_xxx():
>     register_pytree_node = _torch_pytree.register_pytree_node
> else:
>     register_pytree_node = _torch_pytree._register_pytree_node
> ```

Thanks for reviewing!
I just found this consideration has been involved in the current code.
So there is no need to continue this PR right now.
[Warning] Using deprecated function calling `torch.utils._pytree._register_pytree_node`.
Author: jacklanda
Pull request: no

In the latest version of pytorch, the old method `torch.utils._pytree._register_pytree_node` seems to be dropped soon. However, transformers ([1](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/generic.py#L339) and [2](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/generic.py#L482)) has not follow up this tiny update yet.

This cause redundant warnings as follows:

<img width="1709" alt="image" src="https://github.com/huggingface/transformers/assets/54089835/78d5a683-3d47-469e-a811-628bcde9bb60">


Comments:
Fix fx tests with inputs_embeds
Author: fxmarty
Pull request: yes

Some more tests were failing following https://github.com/huggingface/transformers/pull/31840, CI was not triggered, this time making sure locally that `pytest tests/models/ -s -vvvvv -k "test_torch_fx"` passes.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31862). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The remaining 2 failures in `torch_job` are irrelevant I believe.

@fxmarty Any update on this? This is still causing failures on `main` 

Thanks a lot @amyeroberts, could you give it a second look?
adding user defined tokens #30824
Author: itazap
Pull request: yes

Fixes #30824 #30947


## Tasks
- [ ] fix converter to handle user_defined_symbols
- [ ] create necessary flags for user_defined_symbols
- [ ] update docs
- [ ] test slow == fast
- [ ] add test without sentencepiece


## Who can review?
- @ArthurZucker

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30929). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

We can close this as #31305 superseeds it right?
[fix] AttributeError in is_flash_attn_greater_or_equal
Author: SeongBeomLEE
Pull request: yes

# What does this PR do?

Changed the local variable version to pkg_version. (#31887)

Thanks.

Comments:
@SeongBeomLEE Thanks, it is fixed in https://github.com/huggingface/transformers/pull/31446 as I hit the same issue resolving conflicts with gemma2 softcap.
Installing pydantic>=2.4.0 would cause error and fail to load model
Author: Luodian
Pull request: no

### System Info

The error would be as
```
Failed to import transformers.generation.utils because of the following error (look up to see its traceback): 'FieldInfo' object has no attribute 'required'
```

I spent a lot of time to spot it's the version change from the installation of upgrading `gradio` and it caused `pydantic==1.10.7` to `pydantic==2.4.2`.

I downgraded to 1.10.7 then could run smoothly as usual.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
!pip install pydantic==2.4.2
from transformers import FuyuForCausalLM, AutoTokenizer, FuyuProcessor, FuyuImageProcessor
```

### Expected behavior

```sh
Failed to import transformers.generation.utils because of the following error (look up to see its traceback): 'FieldInfo' object has no attribute 'required'
```

Comments:
Hi @Luodian, thanks for raising this issue! 

I looks like we might have to pin pydantic in our setup.py - WDYT @ydshieh?

Hi @Luodian 

We do have `"pydantic<2",` in out `setup.py`

Related https://github.com/huggingface/transformers/pull/24596

Not sure how much we can do with `pydantic 2` so far.

oh know that, thanks! I think the `pydantic>2` comes from upgrading `gradio`.

Eventually, if `pydantic>=2` is used by many libraries, we might consider to update the requirement  (as long as not so many things breaking 😄 )

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
🚨 fix(SigLip): remove spurious exclusion of first vision output token
Author: transmissions11
Pull request: yes
Labels: name: run-slow

Looks like the first token is spuriously excluded from the average pooling because this code was copy & pasted from `modeling_clip`, which does prepend a `[CLS]` token. However SigLip has no such special token, so it would seem we are needlessly excluding a token from being included?

p.s. why not use the `pooled_output` from the model's map head vs averaging the last hidden states? 

Comments:
There's another PR which suggested to use the pooled output rather than averaging the patch tokens: https://github.com/huggingface/transformers/pull/30373.

The reason I went with the latter was because of [this paper](https://arxiv.org/abs/2205.01580) (follow-up paper by the ViT authors which showed that it works better than the CLS token). However averaging patch tokens might work better for ViTs pre-trained using (self-)supervised image classification. For contrastive models like CLIP and SigLIP, perhaps it makes more sense to use the pooled output. I'm happy to change it, although we would need to make it backwards compatible

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@transmissions11 Any update on this?

As this is a breaking change, wondering whether we should just go for the pooled output anyway? It would however break any existing fine-tunes so we might need to add a flag for backwards compatibility.

@NielsRogge It might be breaking but the previous logic was wrong -- there's no reason to exclude the first token. We don't keep backwards compatibility for bugs, and so I'm not sure it makes sense to have backwards compatibility here for the token. 

If we did add the pooled logic, then yes we should have a flag. I'd do this in a separate PR
Generate: fix `SlidingWindowCache.reset()`
Author: gante
Pull request: yes

# What does this PR do?

`RUN_SLOW=1 py.test tests/models/mistral/test_modeling_mistral.py::MistralIntegrationTest::test_compile_static_cache` is failing due to the `reset` function in the sliding window cache, which is incorrect 

(possibly the result of a bad merge conflict? this used to be the old code)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31917). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix fairseq wav2vec2-xls-r pretrained weights conversion scripts
Author: heatz123
Pull request: yes

# What does this PR do?
Fixes:  #19319

This PR fixes some bug on wav2vec2 fairseq weights conversion scripts, that [wav2vec2-xls-r-kind](https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec/xlsr) weights files fail to be loaded on this line.

https://github.com/huggingface/transformers/blob/bc21aaca789f1a366c05e8b5e111632944886393/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py#L249

This can be resolved by specifying fairseq task as `audio_pretraining`, and loading fairseq weights with the task context.

This change follows the way of fairseq library, which loads pretrained model weights via passing `task` argument on cli. 
Conversion of other non-finetuned weights works without any side effects. (tested with [wav2vec2-base](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt), [wav2vec2-conformer](dl.fbaipublicfiles.com/fairseq/conformer/wav2vec2/librilight/LL_relpos_PT_no_FT)) 

Referenced model weights are in the following url: https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sanchit-gandhi
@patrickvonplaten 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Tokenizer][bug] LLAVA 1.6 tokenizer problem
Author: lanking520
Pull request: no

### System Info

Any OS system that can run transformers

related issue: https://github.com/vllm-project/vllm/issues/6224

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

transformers (4.42.3) and got the issue

```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("llava-hf/llava-v1.6-34b-hf")
print(tokenizer.encode("<image>"))

print(tokenizer.vocab_size)
```

output

```
[64003]
64000
```

Can confirm transformers (4.40.1) does not have this issue

```
[64000]
64000
```

### Expected behavior

It is supposed to be the same?

```
[64000]
64000
```

Comments:
After trying this code on a few different versions, it looks like this got changed between 4.41.2 and 4.42.0.

Also, this issue only occurs for `llava-hf/llava-v1.6-34b-hf`. It works fine for `llava-hf/llava-v1.6-mistral-7b-hf` and `llava-hf/llava-v1.6-vicuna-7b-hf`.

**Update:** I think this is the same issue as #31713.

Answered in https://github.com/huggingface/transformers/issues/31713#issuecomment-2222139329

Closing because #31902 was merged
Docker: TF pin on the consistency job (DO NOT MERGE, BUILDING IMAGES ONLY)
Author: gante
Pull request: yes

# What does this PR do?

Building new images 😎 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31927). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Docker: TF pin on the consistency job
Author: gante
Pull request: yes

# What does this PR do?

A TF pin was missing in the consistency docker file :)

Related docker-building PR: https://github.com/huggingface/transformers/pull/31927

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31928). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add `cosine_with_min_lr` scheduler in Trainer
Author: liuyanyi
Pull request: yes

# What does this PR do?

Add `cosine_with_min_lr` scheduler in Trainer.

Fixes #26209 about min_lr.

When set the num_steps=100, num_warmup_steps=10, lr=0.2, min_lr=0.01. The learning rate looks like:

![image](https://github.com/huggingface/transformers/assets/11705038/4533d69f-859d-4266-a697-83c301e757d6)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@muellerzr and @pacman100


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29341). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
PEFT + ZeRO Phase 2 + Transformers doesn't output pytorch_model.bin
Author: cdoern
Pull request: no

### System Info

Python 3.11.9
Transformers 4.38.2
torch 2.3.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

All of the below steps occur when creating a trainer using a python script and running trainer.train()

1. load a bitsnbytes 4 bit quantized, pretrained model from HF.
2. put it on GPU (single A10 in this case)
3. create a trainer with peft config using lora, and init deepspeed with phase 2
3. trainer.train
4. output dir looks like: 
```
checkpoint-1  checkpoint-2  checkpoint-3  checkpoint-4  checkpoint-5
```
5. one of the checkpoint dirs has: 
```
adapter_config.json        global_step2  rng_state.pth            tokenizer.json      training_args.bin
adapter_model.safetensors  latest        special_tokens_map.json  tokenizer.model     zero_to_fp32.py
added_tokens.json          README.md     tokenizer_config.json    trainer_state.json
```


### Expected behavior

I believe there should be pytorch_model.bin for me to load. If LorA is somehow messing with where everything gets stored, this isn't documented anywhere. Training succeeds, but I am not merging or unloading anything as I am not sure where to grab the proper weights from since the info in the docs doesn't cover this.

Comments:
cc @younesbelkada 

Just looking for clarity on if this is normal and if so how to get the trained model in some sane format I can carry on with.

@younesbelkada sorry for the ping, but this is a pretty big blocker for some work I am doing. Is there a quick howto I can get on PEFT+Deepspeed if my understanding is lacking here?

Hi @cdoern 
Apologies for my late reply, in the latest PEFT / transformers we should support QLoRA + DeepSpeed on all stages: https://huggingface.co/docs/peft/accelerate/deepspeed see that documentation page for more details
You are using QLoRA meaning you are fine-tuning adapters on top of the full model (i.e. base model), it is expected that you only have `adapter_model.safetensors` file. Note you can load directly the adapters with `AutoModelForCausalLM.from_pretrained` (see https://huggingface.co/docs/transformers/peft for reference), or you can use `PeftModel.from_pretrained` from peft and call `model.merge_and_unload()` to merge the adapter model into the base model and save the merged model as a standalone model.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.