noamaanMulla-03/transformers-issues huggingface dataset:
Keypoints 0.0 are confusing ../transformers/models/detr/image_processing_detr.py which are fixed
Author: hackpk
Pull request: yes

# What does this PR do?

This PR will fix the keypoint 0.0 at this transformers/models/detr/image_processing_detr.py as suggested by @duckheada

To fix this, I had to edit this file in transformers library:
.conda/lib/python3.9/site-packages/transformers/models/detr/image_processing_detr.py

I changed this as suggested by: @duckheada
  
```python
    if annotations and "keypoints" in annotations[0]:
       keypoints = [obj["keypoints"] for obj in annotations]
       print("keypoints", keypoints) #TODO: remove
       keypoints = np.asarray(keypoints, dtype=np.float32)
       num_keypoints = keypoints.shape[0]
       keypoints = keypoints.reshape((-1, 3)) if num_keypoints else keypoints
       new_target["keypoints"] = keypoints[keep]
```
To this:
```python
    if annotations and "keypoints" in annotations[0]:
     keypoints = [obj["keypoints"] for obj in annotations]
     # Apply the keep mask here to filter the relevant annotations
     keypoints = [keypoints[i] for i in range(len(keypoints)) if keep[i]]
     # converting the filtered keypoints list to a numpy array and reshape it
     keypoints = np.asarray(keypoints, dtype=np.float32)
     num_keypoints = keypoints.shape[0]
     keypoints = keypoints.reshape((-1, 3)) if num_keypoints else keypoints
     new_target["keypoints"] = keypoints # We no longer apply keep mask here
```
Why?
To ensure that the filtering applied to the key points respects its original structure (number of keypoints per annotation). When you reshape keypoints with key points. reshape((-1, 3)), it loses the information about which keypoints belong to which annotation.

Here is what needed to be done (at least in my little hack-ish workaround):

Before reshaping the key points array, I had to apply the keep mask to retain only the annotations I was interested in. Only after this could I reshape the keypoints array to apply further operations.
Then, I applied the keep mask on the keypoints list before converting it into a numpy array and reshaping it. This ensured that I only keep the keypoints corresponding to the bounding boxes that satisfy the condition in the keep mask.

Fixes #26126 

## Who can review?

@ArthurZucker, @younesbelkada, and @amyeroberts 
Please let me know if I need to do anything else for this issue.


Comments:
Hi @hackpk, I am making the first review here :) 

Hi @hackpk , :) 

I opened a discussion in issue #26126 trying to replicate the error. Let's discuss it there, then I continue with the review here.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26250). All of your documentation changes will be reflected on that endpoint.
Add "Fill-in-Middle" pipeline
Author: tanaymeh
Pull request: yes

# What does this PR do?

This PR adds the Fill-in-Middle pipeline to 🤗 transformers. 

FIM objective was proposed in [Efficient Training of Language Models to Fill in the Middle](https://arxiv.org/abs/2207.14255). They showed that autoregressive language models can learn to infill text after applying a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end.

As discussed in #27059

## Who can review?
@sayakpaul @ArthurZucker 

Comments:
Hey! Were you waiting for a review or do you not have time to work on this? 

> Hey! Were you waiting for a review or do you not have time to work on this? 

@ArthurZucker I am working on this but I accidentally opened a wrong PR, fixing and opening a new one today.

Sorry for the inconvenience :)
transformers==4.35.2 compatibility issue with pytorch==1.7.1+cu110
Author: rabjab
Pull request: no

### System Info

transformers==4.35.2 
pytorch==1.7.1+cu110

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install pytorch (torch==1.7.1+cu110) (assume using nvidia 470 drivers)
2. pip install transformers
3. `from transformers import AutoTokenizer, AutoModelForCausalLM`

### Expected behavior

`No module named 'torch.utils._pytree'`

Previous versions of transformers package (i.e., 4.33.0) work fine.

Maybe something to look into re dependency... 
Maybe something to add to docs? @stevhliu 

Comments:
Hey! A few PRs aimed to fix this were opened #27803 and will address this

Thanks, @ArthurZucker - I'll close this since y'all are obviously on top of it already. Apologies for the noise.
[Seamless v1] Link to v2 docs
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Links the Seamless v2 docs in the v1 docs.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27827). All of your documentation changes will be reflected on that endpoint.
[Whisper] Fix doctest in timestamp logits processor
Author: sanchit-gandhi
Pull request: yes

Doctest for Whisper timestamp logits processor was reported as failing in https://github.com/huggingface/transformers/pull/27485#issuecomment-1810876813. This PR corrects the code example.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27795). All of your documentation changes will be reflected on that endpoint.
Flash Attention 2 support for RoCm
Author: fxmarty
Pull request: yes

As per title.

https://github.com/ROCmSoftwarePlatform/flash-attention has been bumped to `2.0.4` recently, and we expect it to be bumped to `2.1` soon. Meanwhile, this PR adds suport for FA2 on RoCm devices. It can be simplified once FA2 RoCm version is bumped to `2.1`.

Tests that are passing on A100 are also passing on MI210 (some are flacky on A100).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27611). All of your documentation changes will be reflected on that endpoint.

@LysandreJik @ArthurZucker @amyeroberts @younesbelkada WDYT?

Reviewing now
single word should be set to False
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes qn issue reported on the hub: 

```python
from transformers import AutoTokenizer, MT5Tokenizer
tokenizer = MT5Tokenizer.from_pretrained("google/mt5-base", extra_ids = 100)
```
(this is rarely tested) trigers this
```diff
        # for legacy purpose, we keep this. Will be removed and tests updated. (when `added_tokens_decoder` is not passed as kwargs)
        self._added_tokens_decoder = {}
        for i in range(len(extra_tokens)):
            self._added_tokens_decoder[len(self.sp_model) - 1 + extra_ids - i] = 
-AddedToken(f"<extra_id_{i}>", single_word=True, lstrip=True, rstrip=True, special=True, normalized=False)
+AddedToken(f"<extra_id_{i}>", single_word=False, lstrip=True, rstrip=True, special=True, normalized=False)
```

Comments:
restructure AMD scheduled CI
Author: ydshieh
Pull request: yes

# What does this PR do?

So far, the AMD scheduled CI is run as a single workflow with `mi210` and `mi250` both in it (each has ~500 jobs): see [here](https://github.com/huggingface/transformers/actions/runs/6999845840)

<img width="791" alt="Screenshot 2023-11-28 140000" src="https://github.com/huggingface/transformers/assets/2521628/b16b47f4-d5bf-426f-9eb2-e964e09455ee">

This causes 2 issues:

  - the workflow run page is too large to display (A unicorn image with `This page is taking too long to load.`)
  - the artifact produced by the runs of `mi210` and `mi250` are mixed (overwritten by each other), so the report might be inaccurate.

This PR restructure AMD scheduled CI to make  `mi210` and `mi250` run in 2 workflow run, so avoid the above 2 issues.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27743). All of your documentation changes will be reflected on that endpoint.
translate internal folder files to chinese
Author: jiaqiw09
Pull request: yes

# What does this PR do?

Part of  #26803 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? _not necessary_


## Who can review?
@stevhliu

Comments:
@stevhliu

Hi, here is another pr. I will fix merge conflic later.

And for file `trainer_utils.md`, I think the subtile "Distributed Evaluation" is repeated. Would you mind providing a useful subtitle? I can update both en file and zh file at this pr.

@stevhliu  

Hi, thansk for your comment and reviews. I just update these reviews and I aslo update `trainer_utils` in `en` folder. 

And I am little busy past few weeks and now I think I will be back for those works again.

Best

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27638). All of your documentation changes will be reflected on that endpoint.

Thanks again, and no worries about being busy. Feel free to take the time you need 🤗
[JAX] Replace uses of jax.devices("cpu") with jax.local_devices(backend="cpu")
Author: hvaara
Pull request: yes

An upcoming change to JAX will include non-local (addressable) CPU devices in jax.devices() when JAX is used multicontroller-style, where there are multiple Python processes.

This change preserves the current behavior by replacing uses of jax.devices("cpu"), which previously only returned local devices, with jax.local_devices("cpu"), which will return local devices both now and in the future.

This change is always safe (i.e., it should always preserve the previous behavior), but it may sometimes be unnecessary if code is never used in a multicontroller setting.

For a similar PR in `diffusers` see huggingface/diffusers#5864.

Comments:
/cc @sanchit-gandhi 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27593). All of your documentation changes will be reflected on that endpoint.
[i18n-jp] Translating `en/tasks` folder docs to Japanese 🇯🇵
Author: rajveer43
Pull request: no
Labels: name: WIP


Comments:
Is anyone working on this? I'm interested in contributing!
If no one is working on it, I would like to go from A (asr.md) to Z (zero_shot_object_detection.md) :)

@Yuki-Imajuku , the PR is already made for this issue, see #27098, 

you can work on #27392
How to export a Marian model in rust ?
Author: flutter-painter
Pull request: no

Most models based on Marian are also available in rust, such as : Helsinki-NLP/opus-mt-en-roa

Is it possible to do this using transformers ?
Did you asssit Helsinki-NLP in exporting the models to Rust ?

Comments:
They're models from https://github.com/guillaume-be/rust-bert you can probably ask there! cc @guillaume-be for info

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Faster generation using AWQ + Fused modules
Author: younesbelkada
Pull request: yes

# What does this PR do?

Introduces a new feature - fused module generation using `autoawq` library. Users need to specify modules that they want to fuse inside `fusing_mapping`. 

The API is as follows:

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, AwqConfig, TextStreamer

model_id = "TheBloke/Mistral-7B-OpenOrca-AWQ"
torch_device = "cuda" if torch.cuda.is_available() else "cpu"

quantization_config = AwqConfig(
    bits=4,
    do_fuse=True,
    fuse_max_seq_len=512,
)

model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config).to(0)
tokenizer = AutoTokenizer.from_pretrained(model_id)

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

prompt_template = """\

system
You are MistralOrca, a large language model trained by Alignment Lab AI. Write out your reasoning step-by-step to be sure you get the right answers!


user
{prompt}


assistant"""

prompt = "You're standing on the surface of the Earth. "\
        "You walk one mile south, one mile west and one mile north. "\
        "You end up exactly where you started. Where are you?"

tokenizer.pad_token = tokenizer.eos_token

inputs = tokenizer([prompt_template.format(prompt=prompt), prompt_template.format(prompt=prompt), prompt_template.format(prompt=prompt)], return_tensors="pt", padding=True).to(0)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

## Before this PR:

![before-fusing](https://github.com/huggingface/transformers/assets/49240599/6086e7ad-d00a-427b-a2ce-30ac53b2efe4)

## After this PR:

![fast-awq-generate](https://github.com/huggingface/transformers/assets/49240599/f0c1fd61-34a0-4c26-b04e-54c6df9554ba)

## TODOs:

- [x] Block save_pretrained if one creates a fused model
- [x] Fuse MLP
- [x] wait until dynamic batch size is resolved on auto-awq side : https://github.com/casper-hansen/AutoAWQ/pull/181
- [x] Force users to use `autoawq>=0.1.7`
- [x] Tests
- [x] Benchmarks
- [x] Docs

cc @amyeroberts @casper-hansen @SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27411). All of your documentation changes will be reflected on that endpoint.

Before moving forward with tests and advanced docs, I would love to have an early feedback of the API that is described in the PR description. cc @amyeroberts , whenever you have time, i would appreciate your feedback on this PR 🙏 Thanks! 

Thank you very much @amyeroberts for your extensive review! I addressed most of your comments and left some open questions - let me know what do you think! 🙏 

Thanks @amyeroberts @SunMarc for your great reviews!
@SunMarc I just want to get more clarification on this comment, otherwise good to merge IMO !
Generate: Update VisionEncoderDecoder test value
Author: gante
Pull request: yes

# What does this PR do?

#27351 fixes a bug in beam search: the prompt length was being included in the length penalty computation, and this penalty should only be applied on newly generated tokens (otherwise decoder-only models would often see a big penalty, as the prompt is part of the output)

This PR updates the test results to account for the bug fix. I've double-checked that reverting those changes produces the old results!

(All tests in `RUN_SLOW=1 py.test tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py -vv` pass)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27850). All of your documentation changes will be reflected on that endpoint.
[`ClipVision`] `accelerate` support for clip-vision
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, all accelerate tests pass

addresses: https://github.com/huggingface/transformers/pull/27662#discussion_r1415445563 for llava

cc @ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27851). All of your documentation changes will be reflected on that endpoint.
Move tensors to same device to enable IDEFICS naive MP training
Author: willemsenbram
Pull request: yes

# What does this PR do?

Fixes #27736

`logits`, `labels`, and `attention_mask` are expected to be on the same device. 
This PR moves these tensors to the same device, making IDEFICS training compatible with naive model parallelism.

See also #22561 


@amyeroberts @ArthurZucker @younesbelkada

Comments:
For the test I suggest to do it in a follow-up PR to test that for all architectures, through a test that runs on the daily CI
delete `delete_doc_comment_trigger.yml` workflow
Author: ydshieh
Pull request: yes

# What does this PR do?

delete `delete_doc_comment_trigger.yml` workflow as `huggingface/doc-builder/.github/workflows/delete_doc_comment_trigger.yml` no longer exists for security reason.

Comments:
Superceded by #27852

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27854). All of your documentation changes will be reflected on that endpoint.
⚠️ [VitDet] Fix test
Author: NielsRogge
Pull request: yes

# What does this PR do?

After #27729, the `test_forward_signature` test fails for `VitDetBackbone` given that it uses the order of `output_hidden_states` first followed by `output_attentions` in its forward signature. The test is introduced to make sure new backbones always use the same order.

We have a couple of options here:
- either make the forward signature consistent with other models (as currently done in this PR). Is a breaking change, but given that this model is only used as internal building block of ViTMatte, and it is explicitly [hidden from the docs](https://github.com/huggingface/transformers/blob/e739a361bc19000f619cf248707914844456835f/utils/check_repo.py#L990), this might be acceptable.
- either overwrite the test in test_modeling_vitdet.py.

Comments:
> this model is only used as internal building block of ViTMatte, and it is explicitly [hidden from the docs](https://github.com/huggingface/transformers/blob/e739a361bc19000f619cf248707914844456835f/utils/check_repo.py#L990), this might be acceptable.

I agree + this is still (somehow recent + I can't even find a checkpoint (`facebook/vit-det-base` doesn't exist) ==> good for me

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27832). All of your documentation changes will be reflected on that endpoint.

⚠️as requested
Make image processors more general
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR undoes https://github.com/huggingface/transformers/pull/26965 and instead makes image processors more general, by removing the hardcoded "shortest_side" arguments, allowing to pass the following to image processors:

```
size = {"height": ..., "width": ...}
size = {"shortest_edge": ...}
```
Ideally it should also allow for the following (cc @amyeroberts)
```:
size = {"longest_edge": ...}
size = {"shortest_edge": ..., "longest_edge": ...}
```
Image processors should not be limited to only `{"shortest_edge": ...}` for instance, as `CLIPImageProcessor` is at the moment. The `size` argument is now always a dictionary containing [one of these 4 possibilities](https://github.com/huggingface/transformers/blob/7293fdc5b9cc809c2aa2ceb84f903ad47e5c06f0/src/transformers/image_processing_utils.py#L663); hence it would be great to support them all.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27690). All of your documentation changes will be reflected on that endpoint.

Thanks for opening this PR! 

>Ideally it should also allow for the following (cc @amyeroberts)
> 
> size = {"longest_edge": ...}
> size = {"shortest_edge": ..., "longest_edge": ...}

As discussed on slack, enabling all image processors to accept `{"height": h, "width": w}` is a change I'm very much pro. Enabling all to accept `{"shortest_edge": x, "longest_edge": y}` isn't something I think we should as this behaviour isn't well-defined and varies between models. 

cc @ydshieh this is failing for KOSMOS-2 - it would be great to revert that change for KOSMOS-2 (i.e. remove the "use_square_size" attribute) since this is introduced for legacy behaviour of `size` (which used to be an integer and is now a dictionary).

It did but it's also very convenient not to have to use copied from when the only difference was this, so reverting IMO is not the best solution. Let's rather adapt / make it compatible 

Reverting #26965 means we well have to add `Kosmos2ImageProcessor` (while currently it is `CLIPImageProcessor` with `use_square_size=True`). Probably it will go without problem, but still kind of breaking changes.

@ydshieh would it also be possible to just set size to `{"height": ..., "width" ...}`? With this PR, image processors are made more general, making sure square sizes are supported

Hi, could you give more details on

>  set size to {"height": ..., "width" ...}

(where should I set this, for example.)

Do you mean on the config file?

Yes I meant in the `preprocessor_config.json`. Given that the model is only one month old, we could still update them. Alternatively, I've added a backwards compatibility in this PR, only for `CLIPImageProcessor`. Let me know what you think works best.

We can keep your backwards compatibility code, and I could try to open Hub PRs to update them.
Let's also hear what the core maintainers say of course.

It's required to merge #27718 

Merge now as full (slow) CI on 4 models touched by this PR pass

cc @younesbelkada we have a regression on this