noamaanMulla-03/transformers-issues huggingface dataset:
Adding hiera
Author: EduardoPach
Pull request: yes
Labels: name: New model

# What does this PR do?

Adds Hiera model from Meta as suggested in https://github.com/huggingface/transformers/issues/28993

GitHub Repo: https://github.com/facebookresearch/hiera/ 
arXiv: https://arxiv.org/abs/2306.00989

Model License recently changed to `Apache`

As the work in https://github.com/huggingface/transformers/pull/29339 is on hold and had a lot of things to change to meet the library standards I've opened this new PR

c.c. @amyeroberts 



Comments:
Thanks for opening this @EduardoPach! 

As there's an already open PR which the contributor gave us warning would be pending for a few weeks because of their exams and we acknowledged, the best thing and polite this to do it ping the contributor on their PR to ask if and when it's going to be picked back up. If they aren't able to work on it and get it to a reviewable state in the next e.g. 2 weeks, then I'd say it's free to work on (letting them know on the PR, of course). 

> Thanks for opening this @EduardoPach! 
> 
> 
> 
> As there's an already open PR which the contributor gave us warning would be pending for a few weeks because of their exams and we acknowledged, the best thing and polite this to do it ping the contributor on their PR to ask if and when it's going to be picked back up. If they aren't able to work on it and get it to a reviewable state in the next e.g. 2 weeks, then I'd say it's free to work on (letting them know on the PR, of course). 

Thanks for the heads up @amyeroberts! Didn't mean to be unpolite I just have a particular interest in the model and thought I could contribute, but will have that in mind for future contributions. 

Regarding the other PR that is already open I've messaged the author to check if he was interested in merging our works so he could keep working on it by himself or together. If you have any comments feel free to add. 

@amyeroberts tests are green. Not all checkpoints were converted though (only the tiny versions of the base model, image classification, and MAE model) and a `HieraForVideoClassification` could be added 

Closing this one as this is being worked on https://github.com/huggingface/transformers/pull/30356
fix: center_crop occasionally outputs off-by-one dimension matrix
Author: mattlbeck
Pull request: yes

# What does this PR do?

If required padding for a crop larger than input image is odd-numbered, the padding would be rounded down instead of rounded up, causing the output dimension to be one smaller than it should be.

Fixes #30922 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@amyeroberts 


Comments:
`center_crop` outputs wrong sized array if provided with odd-numbered dimensions smaller than requested crop size
Author: mattlbeck
Pull request: no

### System Info

transformers 4.40.1, python 3.12

### Who can help?

@amyeroberts 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers.image_transforms import center_crop
import numpy as np
small_even = np.ones((3,4,4))
small_odd = np.ones((3,3,3))
big_even = np.ones((3,10,10))
big_odd = np.ones((3,11,11))
print([center_crop(x, (8,8)).shape for x in [small_even, big_even, big_odd, small_odd]])
```

Result:
```
[(3, 8, 8), (3, 8, 8), (3, 8, 8), (3, 7, 7)]
```

### Expected behavior

All arrays should be cropped to the requested size of (8,8). However, odd-numbered dimensions that are smaller than the crop size and require padding result in an unexpected off-by-one output size.

Comments:
I believe the issue is more accurately caused by odd-numbered difference between original size and new size. Rounding up rather than down when calculating the padding fixes the above test cases.
Trainer.py module 'datasets' has no attribute 'Dataset'
Author: FrancoisMentec
Pull request: no

I'm trying to use a Trainer, but I get this error:
```
c:\users\francois\appdata\local\programs\python\python37\lib\site-packages\transformers\trainer.py in __init__(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, tb_writer, optimizers, **kwargs)
    287 
    288         if is_datasets_available():
--> 289             if isinstance(train_dataset, datasets.Dataset):
    290                 self._remove_unused_columns(self.train_dataset, description="training")
    291             if isinstance(eval_dataset, datasets.Dataset):

AttributeError: module 'datasets' has no attribute 'Dataset'
```
My guess is that `datasets.Dataset` should be replaced by `torch.utils.data.Dataset` but I haven't checked the source file. Maybe the person responsible for the `Trainer` development should look into that.

I'm using transformers version 3.2.0 btw.

Comments:
I'm not sure which `datasets` module is installed in your env, but the Hugging Face `datasets` definitely has a `Dataset` attribute. And no, this part is not using PyTorch `Dataset`.

Apparently, the `datasets` module wasn't even installed on my environment. But installing it just replaced the error by another one. It's upgrading PyTorch that fixed the issue, might be cool to be notified during the installation or the execution of transformers that we don't have the required PyTorch version for it to works. It feels awkward having to fixe dependencies myself, that's what a package manager like pip is usually used for. Maybe it's better handled by anaconda...

> Apparently, the `datasets` module wasn't even installed on my environment. But installing it just replaced the error by another one. It's upgrading PyTorch that fixed the issue, might be cool to be notified during the installation or the execution of transformers that we don't have the required PyTorch version for it to works. It feels awkward having to fixe dependencies myself, that's what a package manager like pip is usually used for. Maybe it's better handled by anaconda...

hi, could you please tell me which pytorch version you have been upgraded to solve this problem? I got the same problem.

@ericdoug-qi Can't remember, did you check you are using the latest version of PyTorch and Transformers? Otherwise, try anaconda.

may be you use the wrong moudule, try to run conmand "pip uninstall datasets" , then it can use  Hugging Face datasets

This solved it for me:
I checked the dependencies of `datasets`.
`pip show datasets` 
I uninstalled all dependencies and the `datasets` package. 
`pip uninstall aiohttp dill filelock fsspec huggingface-hub multiprocess numpy packaging pandas pyarrow pyarrow-hotfix pyyaml requests tqdm xxhash datasets`
Then I reinstalled `datasets`, which also installs the dependencies as well but with the proper versions that do not conflict. 
`pip install datasets`
Finally I restarted the kernel.
TST / Workflows: Get slack notifications for docker image build
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, as similarly as https://github.com/huggingface/peft/pull/1723 - I created a new slack channel to receive notifications on the docker image build status for all our docker images 

We do that in PEFT / diffusers (https://github.com/huggingface/diffusers/pull/7938)

Related to: https://github.com/huggingface/transformers/pull/30890

cc @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30891). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks so much for the review @ydshieh !
Fix swin embeddings interpolation
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes broken tests on main introduced by #30656

The PR mistakenly introduced interpolation to the patch embeddings, which shouldn't be done, and also adds a verification check which doesn't account for the padding in `_maybe_pad`


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30936). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ydshieh Yes, it's set to `False` by default. There's a few things happening here: 

* The interpolation logic was added for the patch embeddings (which shouldn't have happened)
* Swin actually already handles images of different sizes (providing they're divisible by 32), so in the pipeline tests, images different from those the model was trained on were being passed in 
* The additional interpolation logic added a check to verify the input images size if interpolation was `False` (as before). However, because swin can handle some images of different sizes, and `interpolate_pos_encoding` was `False` by default this new check was triggered and would fail
Implement kv cache sparsity like H2O with attention score
Author: HarryWu99
Pull request: no
Labels: name: Feature request

name: Cache

### Feature request

Hello!

It is a bit like [#26553](https://github.com/huggingface/transformers/issues/26553), which implement `SinkCache`. I would love to see some method of kv cache sparsity like **H2O** implemented, as proposed in [http://arxiv.org/abs/2405.04434](http://arxiv.org/abs/2405.04434).

The authors have release the code here: [https://github.com/FMInference/H2O](https://github.com/FMInference/H2O).

People can use it like:
```python
from transformers import AutoModelForCausalLM AutoTokenizer, H2O_Cache

cache = H2O_Cache(recent_length=512, HH_length=512)
gen_out = model.generate(**inputs, do_sample=False, max_new_tokens=3000, past_key_values=cache)
```

### Motivation

<img width="899" alt="image" src="https://github.com/huggingface/transformers/assets/63134210/468abafb-ab40-4818-9fe3-1c156c9f2e4d">

> Our approach is based on the noteworthy observation that a small portion of tokens contributes most of the value when computing attention scores.
> a KV cache eviction policy that dynamically retains a balance of recent and H2 tokens

### Your contribution

I would love to help implement this into transformers.

It is not only implement a `H2Ocache` in `src/transformers/cache_utils.py`, but also change the order of some code in `LlamaAttention#forward` function, so `Cache#update` can get the attention score, which some method of kv cache sparsity like [snapKV](http://arxiv.org/abs/2404.14469) and future work also need.

Comments:
cc @ArthurZucker @gante 

Hey @HarryWu99 👋 

Techniques that improve memory utilization with LLMs are always exciting! At first glance, it seems like a good candidate to be added to `transformers` with the API you showcased in your example. Two additional points for consideration:
1. Benchmarks need to be run before merging, to confirm the implementation is working as expected;
2. You mentioned changes in `LlamaAttention.forward`, to use the attention scores. We may need a new function for that, like `Cache.post_process()`, we may need to iterate on the design throughout the PR.

If you're happy with these two points, we'd be happy to take your PR and guide you in the process 🤗 

(P.S. your first link is to the DeepSeek-V2 paper, I'm assuming you meant the H2O paper :) )
add interpolate_pos_encoding for Detr and test
Author: justyx404
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

- Add interpolate position embedding to Detr model #30579
  ![image](https://github.com/huggingface/transformers/assets/17937555/5e77728b-8680-4754-b71b-228036d38cb0)



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@amyeroberts

Comments:
It's a bit awkward due to Detr using sine pos embedding by default which doesn't need to be interpolated. As for as the assert in the test I made, I'm only making sure the shape stays the same as using sine pos embedding but essentially they are just the object class + bounding box prediction dim.
If this form is okay, I can either `make fixup` the rest detr models in this PR or submit a separate one.

totally understandable after attempting to make it work. and agree that users won't really be needing this. will close it then.
AttributeError: module 'transformers_modules.Qwen-72B-Chat.tokenization_qwen' has no attribute 'QWenTokenizer'
Author: pydaxing
Pull request: no

### System Info

AttributeError: module 'transformers_modules.Qwen-72B-Chat.tokenization_qwen' has no attribute 'QWenTokenizer'

### Who can help?

AttributeError: module 'transformers_modules.Qwen-72B-Chat.tokenization_qwen' has no attribute 'QWenTokenizer'

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

AttributeError: module 'transformers_modules.Qwen-72B-Chat.tokenization_qwen' has no attribute 'QWenTokenizer'

Transformers==4.34.0

### Expected behavior

AttributeError: module 'transformers_modules.Qwen-72B-Chat.tokenization_qwen' has no attribute 'QWenTokenizer'

Comments:
Hi @pydaxing, thanks for raising an issue! 

So that we can best help you, could you make sure to follow the issue template and provide: 
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output
* A minimal code reproducer
* The full error traceback

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, how did you solve it? I have the same bug

I have the same bug, how did you solve it?
Fix inhomogeneous shape error in example
Author: Zantares
Pull request: yes

# What does this PR do?

Fixes #30000 , which has reported an inhomogeneous shape error in the QA example with `numpy` >= 1.24.0:
```
batch = {k: np.array(v) for k, v in batch.items()} ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 2 dimensions. The detected shape was (64, 384) + inhomogeneous part.
```


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sanchit-gandhi 

Comments:
@ArthurZucker can you help to review this PR? Thanks!
Question about Multi-GPU inference (greedy generation with stopping criteria)
Author: sfc-gh-reyazda
Pull request: no
Labels: name: Generation

Hi,

I am curious about the implementation of the gpu-syncing on the generation pipeline when using multiple GPUs. I have been reading the code for the `_greedy_search` function and noticed at one part of the generation loop, we are checking for the each device (peer) to see if it is finished with the generation (potentially reaching a stopping condition) and then it skips the new-token generation: https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L2444. However, I am wondering why we need to even run the forward of the model (https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L2436) if we are not using the output logits anymore.
Do you think it would have been better that we moved this condition at the beginning of the generation loop before preparing the input and running the forward? 

Also, another issue that I can think of by delaying the skipping of the generation iteration for this peer that is finished is that there is a chance that the rest of the devices finish the post-processing and therefore `_has_unfinished_sequences` returns `True`, in which case we run into a deadlock as the already finished peers are running the forward still and the system hangs for any communication that is happening in the model!

Any feedback is appreciated :)

Thanks,
Reza

Comments:
cc @gante 

Hey @sfc-gh-reyazda 👋 

The answer is here: https://huggingface.co/docs/transformers/v4.39.3/en/deepspeed#generate 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
All-zero embedding distorts StaticCache.get_seq_length() output
Author: poedator
Pull request: no
Labels: name: Cache

### System Info

StaticCache.get_seq_length() method detects cache use status based on non-zero cache values. [link](https://github.com/huggingface/transformers/blob/8c12690cecbb97e187861e386f7a0ac790e4236c/src/transformers/cache_utils.py#L414)
I just stumbled upon 2 cases of models that have all-zeros embedding for `token_id==1` which distorts the output:
- [JackFram/llama-160m](https://huggingface.co/JackFram/llama-160m/discussions/15)
- [JackFram/llama-68m](https://huggingface.co/JackFram/llama-160m/discussions/7)

No blame here, just a combination of design decisions with unpredictable results. But there should be a more robust `get_seq_length()` method for StaticCache. I wonder if this is in the roadmap somewhere?

### Who can help?
 cc @ArthurZucker @younesbelkada 

### how to observe
```
model_name = "JackFram/llama-68m"
model = transformers.AutoModelForCausalLM.from_pretrained(model_name)
model.model.embed_tokens.weight
>>>Parameter containing:
>>> tensor(
        [[-0.0095,  0.0167,  0.0140,  ...,  0.0002,  0.0112,  0.0059],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],    # here it is, all zeros
        [-0.0194,  0.0022, -0.0326,  ...,  0.0332, -0.0052, -0.0284],
        ...,
)
input_ids=tensor([[   1,  450, 6635, 3290]])
model.forward(input_ids=input_ids, ...
...
model.model.layers[0].self_attn.past_key_value.get_seq_length()
>>>tensor(3)  # THIS SHOULD BE 4
self.tree.engine.model.model.layers[0].self_attn.past_key_value.key_cache[0,0,:,0]
>>> tensor([ 0.0000, -1.5942, -0.8042,  1.0514,  0.0000,  0.0000,  0.0000,  ...
```

Comments:
`StaticCache.get_seq_length()` should not even exist. We ought to rely on `cache_positions` 

Roadmap is to deprecate all calls to `get_seq_length`, cc @gante 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update build ci image
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Update script!
New workflow:
1. always checkout `push-ci-image`
2. do your thing
3. push a commit with [build-ci-image]
4. check the build
5. push a commit with [push-ci-image]
6. close the pr and delete the branch. 
7. DONE

Comments:
I need this to merge #30932 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30933). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Looks good to me. Just to comment that, by `delete the branch` you mean the remote branch I believe.

BTW, this verifies the images could be built successfully. But this doesn't verify that if those new images would make the CircleCI green or red. I am somehow concerned this part.

Yeah I'm pinning the corresponding timm version 


Actually I should probably build to push a different image than latest, otherwise it still affects everyone else 😅 

> Actually I should probably build to push a different image than latest, otherwise it still affects everyone else 😅

Yeah. Ideally, we should come up a way to build/push the images (somewhere) + run CircleCI. Only this is green, we could push images officially (that will be used by all users).
Reboot Agents
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Reboot agents in transformers!

cc @LysandreJik

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30387). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

You can also check my mini autogpt repo for reevaluating outputs which had low confidence.
This may help with some smaller models against hallucination