noamaanMulla-03/transformers-issues huggingface dataset:
Fix attn mask for static cache
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #30400. 

It was found that when static cache returns key and values if "length=max_length", the zeros are not masked out. That is why the generation starts returning gibberish at larger max_new_tokens, and is more expressed in "sdpa" attention which just falls back to its internal causal mask.

All slow tests in the models are passing for me locally.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30414). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Also Let's add tests in generation common or integration for llama at least. 




Ah i did not see it is supposed to be calculated here. The main error for teh code block in linked issue was in [this line](https://github.com/huggingface/transformers/blob/7bfe57777df3836b28198e47e5846cebf4a72591/src/transformers/models/llama/modeling_llama.py#L1073), where we skip all together the causal mask since it's batch_size=1 and no zeros in attention mask. 

That is for sdpa attn but eager also had errors, even though not so big. Let me check with these new info and tests ofc :)

Thanks

@ArthurZucker I found that indeed th causal mask was being calculated correctly and the gibberish output is only when we use SDPA and batch_size=1. Can be solved by removing the following so that we always rely on our own attn mask, which cannot be simply causal if static cache is used.

I am just not sure, if this line is crucial for Flash Attention 2? 🤔 

```python
if self.config._attn_implementation == "sdpa":
            # For SDPA, when possible, we will rely on its `is_causal` argument instead of its `attn_mask` argument,
            # in order to dispatch on Flash Attention 2.
            if AttentionMaskConverter._ignore_causal_mask_sdpa(
                attention_mask, inputs_embeds=input_tensor, past_key_values_length=past_seen_tokens
            ):
                return None
```

@zucchini-nlp https://github.com/huggingface/transformers/pull/30437 -- this PR also fixes the issue, and I think that is the way to go :D 

The fix in this PR adds complexity outside the SDPA path and needs extra computations 

@gante I see, yeah the cropping of the cache seems a better solution
AttributeError: 'BertTokenizer' object has no attribute 'split_special_tokens'
Author: KeshavUnnikrishnanTR
Pull request: no

### System Info

System Info
Python version: 3.12.0
transformers version: 4.38.2
Platform: MacOS
scikit-learn version: 1.4.0

When loading a sklearn.feature_extraction.text.TfidfVectorizer model, got following error:
`AttributeError: 'BertTokenizer' object has no attribute 'split_special_tokens'`
 
I had noticed this error from other issues too:
https://github.com/flairNLP/flair/issues/1796#issue-674666035
I use 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import pickle
from sklearn.feature_extraction.text import TfidfVectorizer

# Load the pickled vectorizer
with open("models.pickle", "rb") as f:
    vectorizer = pickle.load(f)
print(type(vectorizer))
# Use the vectorizer to transform some text
text = ["This is the first document.", "This is the second document."]
X = vectorizer.transform(text)
```

### Expected behavior

Expect no error for the transform step.

Comments:
Hi @KeshavUnnikrishnanTR, thanks for raising an issue! 

Without knowing what's in `"models.pickle"` it's very hard to know what's happening here. For security, we won't however accept the file to unpickle :) It would help to show the code how this object was created. 

From the linked issue, it appears this can happen from incompatible versions. How old is this file? Which version of transformers was used to create it? What happens if you create a new vectorizer, save it and reload it? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@amyeroberts can you please remove the model name from the issue here.

Hi @KeshavUnnikrishnanTR, sure, I've updated my comment. 

One thing to note for your own information, is that the model name will still be available if someone were to look at the history of the issue
fix jamba slow foward for multi-gpu
Author: SunMarc
Pull request: yes

# What does this PR do ?
This PR makes the slow forward of Jamba model compatible with multi-gpu setup with use_cache=True.  The issue is that the cache is not initialized on the right device. Not sure we want to refactor the cache since the `key_cache` and and `value_cache` are handled correctly. This is just specific to `ssm_states`. Moving the tensor manually should only impact the first forward also. After the first forward, the cache will be correctly initialized. 
Fixes https://github.com/huggingface/transformers/issues/30367 

Code: 

```python
import os
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig

prompt = "def quicksort(arr):\n"
tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-tiny-random")
tokenizer.pad_token = tokenizer.eos_token
config = AutoConfig.from_pretrained("ai21labs/Jamba-tiny-random")
# change the following to switch between fast and slow forward 
config.use_mamba_kernels=False 
model = AutoModelForCausalLM.from_pretrained(
    "ai21labs/Jamba-tiny-random",
    device_map="auto",
    config = config
)
print(model.hf_device_map)
print(model.config)
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(input_ids=inputs.to(model.device), max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0]))
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30418). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> could be changed but would need to accommodate for every layers? (or is the PKV handled by accelerate separately? )

Yes, we would need to accommodate for every layer but that can become quite complicated if we start to introduce layers that are offloaded to the disk and cpu. Hence, I think that it is not worth investing too much effort on that. The PKV is not handled by accelerate separately.
[SegGPT] Fix loss calculation
Author: EduardoPach
Pull request: yes

# What does this PR do?

This PR fixes https://github.com/huggingface/transformers/issues/30419 and ensures that the loss is being correctly calculated.

While working on this PR I not only noticed that `SegGptLoss` was broken, but it was being incorrectly calculated (shame on `SegGpt` contributor 😞). Proposed solution include:

- Passing `labels` to `SegGptModel` to correctly perform the `forward` pass when training in `In-Context Painting` style
- Changed `SegGptLoss` `forward` method and its docstrings accordingly so that the output is the same as the one obtained in the original implementation.

## Note
While running `test_modeling_seggpt` with `is_training = True` I found that `gradient_checkpointing` is also not working due to `type_token_semantic` parameter that is not used in the forward pass and is controlled by the `embedding_type` the model's `forward` and by default we use the `type_token_instance` just like the original implementation. Hence, we could probably move the `embedding_type` to config to allow `gradient_checkpointing` or remove it entirely as in the original implementation is not clear what is the use case for `type_token_semantic`


c.c. @amyeroberts 

Comments:
> Thanks for working on this!
> 
> Have we verified the loss value with that of the original model?

Yeap, also added some tests to make sure this won't be an issue again 🙂. 

Regarding `gradinet_checkpoint` while training, should this be resolved in this PR or a new one? It will probably involve re-uploading the checkpoint to the hub as we would need to take the `type_token_semantic` parameter from the weights

> Regarding gradinet_checkpoint while training, should this be resolved in this PR or a new one? It will probably involve re-uploading the checkpoint to the hub as we would need to take the type_token_semantic parameter from the weights

This should be done in a separate PR.
Add `paths` filter to avoid the chance of being triggered
Author: ydshieh
Pull request: yes

# What does this PR do?

We decide to remove the status check from the repository setting, i.e. no more `Check slow test status` as in the following screenshot.

Therefore, we don't need to trigger the workflow file all the time. This PR adds a `paths` filter. The workflow will be triggered only if there is a change matching `src/transformers/models/*/modeling_*.py`.

- For external contributors' PRs, this avoids to click  unnecessary  `Approve and run` most of the time.
- If a change to modeling files, the workflow will still be trigger even if it is not a new model PR.
  -  If it is from external contributor,  `Approve and run` would be shown and waits for click. 
    -  If it is NOT new model PR, reviewers could ignore it however. But if it is  clicked it (after review), it will be ✅  anyway
- If it is a new model PR, the slow tests will be triggered (external PRs need the button to be clicked + assume the required label  `single-model-run-slow` is added), and the reviewers can check it is ✅  or ❌ (which is easy to identify).


<img width="638" alt="Screenshot 2024-04-24 100723" src="https://github.com/huggingface/transformers/assets/2521628/397bfc97-9e77-42a2-8e12-704bb0e6db1b">


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30453). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`research_project`] Most of the security issues come from this requirement.txt
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Draft

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29977). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Neuron: When save_safetensor=False, no need to move model to CPU
Author: jeffhataws
Pull request: yes

# What does this PR do?

save_safetensor=True is default as of release 4.35.0, which then required TPU hotfix https://github.com/huggingface/transformers/pull/27799 (issue https://github.com/huggingface/transformers/issues/27578). However, when the flag save_safetensor is set to False (compatibility mode), moving the model to CPU causes generation of too many graphs during checkpoint https://github.com/huggingface/transformers/issues/28438. This PR disable moving of model to CPU when save_safetensor=False.

Fixes # 28438 (https://github.com/huggingface/transformers/issues/28438)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- trainer: @muellerzr and @pacman100

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29703). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Pin torch version to < 2.3
Author: amyeroberts
Pull request: yes

# What does this PR do?

Recent pytorch release breaks a bunch of tests on our CI 😢 

Pinning until resolved to avoid blocking for PR merges.

Example of failing run: https://app.circleci.com/pipelines/github/huggingface/transformers/90724/workflows/055d784f-ff15-412e-8033-9ebf8a5fa498/jobs/1184589

Comments:
cc @ydshieh Am I missing anything here?


LGTM. Not sure if we should change the dockerfile and 

examples/pytorch/_tests_requirements.txt

but the priority is to make CircleCI green I guess?

Thanks for the prompt fixes!

@ydshieh Good point about the examples - I've added a pin. 

Re the dockerfile, which one are we talking about? (`transformers-gpu`?). As you say - it's really for CI, so I think it's OK to leave. If people complain, we can pin, WDYT? 

Yes, leave those dockerfile is OK. Let's have daily CI run with torch 2.3 and see how it goes. I will check tomorrow and see what to do.

I am not sure, but I still see (at least in torch job)

```
torch==2.3.0
torchaudio==2.2.2
torchvision==0.17.2
```

maybe it's cache issue. I can take a look. But the fast way is to increase the cache version `cache_version` from `0.8.2` to anything new like `0.8.3`

Hmm, no cache is found. I am not sure why we still get torch 2.3. Let me check

@ydshieh OK, thanks for helping out with this! Shall I remove the increment cache commit? 


No, you can keep using the new version number it's fine.

if --no-deps not working, try change the order of install lines or try to reinstall torch after accelerate (not ideal, but if we just want to have green CI for now)

Woo hoo! Only failing on non-torch related issues. Closing as compatibility is fixed in #30459 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30460). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Casting to device inside of the Tokenizer
Author: Anna146
Pull request: no

# 🚀 Feature request

I think it will make sense if the tokenizer.encode() and in particular, tokenizer.encode_plus() accepting a string as input, will also get "device" as an argument and cast the resulting tensors to the given device. Otherwise, in case of encode_plus(), one has to loop through the output dict and manually cast the created tensors.

Comments:
You can just do:

```
from transformers import AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

device = "cuda" if torch.cuda.is_available() else "cpu"

text = "hello world"
encoding = tokenizer(text, return_tensors="pt").to(device)
```

Cool, thanks!

Suppose I am Using device_map="auto" instead of device Then How to Cast the Tokenizer?

@debanjanm did you find a solution?

@debanjanm valid question that I also had! It turns out you need to just specify `device="cuda"` in that case.

See the guide here for more info: https://huggingface.co/docs/accelerate/concept_guides/big_model_inference#the-devicemap
Enable fp16 on CPU
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR enables users to allow ops that can be done on CPU in FP16 to be done, without blocking them.

(Note: how well they work is TBD, at least the examples tests pass which have a threshold, but edge with caution)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30459). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
consistent job / pytest report / artifact name correspondence
Author: ydshieh
Pull request: yes

# What does this PR do?

Nothing big in this PR: just change some job `names` (or more precisely, job `id`) and artifact names.

The rule is simple: the `test report (directory) name` and the `artifact name` should correspond to the `job id` with a post-fix `test_reports`.

For example:

```yaml
jobs:
  run_models_gpu:  # (job id)
  ...
  steps:
    - name:  run_test
      run: pytest ... --make-reports=run_models_gpu_test_reports  # (pytest report)

      - name: "Test suite reports artifacts: run_models_gpu_test_reports"  # (artifact)
       ... 
       with:
          name: run_models_gpu_test_reports  # (artifact)
          path: /transformers/reports/run_models_gpu_test_reports  # (pytest report)
```

If there is matrix involved like `machine_type` and/or (model) folders, it should be

`${{ matrix.machine_type }}_run_models_gpu_${{ matrix.folders }}_test_reports`



Comments:
cc @younesbelkada , @SunMarc  and @fxmarty 

(not to ping you for review, but to let you know about this change)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30392). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I have verified the changes work for daily (schedule) ci.

For other CI like `push`, `nightly`, `past` and `amd`, they all need rework since we pass the 256 models (`daily-ci` is OK after #28773, but this hasn't applied to other CI).

I will rework other CI progressively.
[DynamicHead - Microsoft] Implementation request
Author: lsch0lz
Pull request: no
Labels: name: New model

### Model description

Hi there! 
I was wondering if anyone has tried to implement Microsoft's DyHead? If not, I would like to contribute the implementation by adding a new model to the library. Is there any interest in a PR for this model?

Here a short explanation from their official implementation:
_"In this paper, we present a novel dynamic head framework to unify object detection heads with attentions. By coherently combining multiple self-attention mechanisms between feature levels for scale-awareness, among spatial locations for spatial-awareness, and within output channels for task-awareness, the proposed approach significantly improves the representation ability of object detection heads without any computational overhead."_

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

GitHub Repository: [Link](https://github.com/microsoft/DynamicHead/tree/master)
Paper: [Link](https://arxiv.org/pdf/2106.08322.pdf)

Comments:
That would be really cool! However, I see they integrated DyHead only into frameworks such as Faster R-CNN and ATSS. Would that imply that you would also implement one of these frameworks?

Oh shoot! 
I wasn't checking whether all the parts I needed were already in the transformers library, I just wanted to add a new model. 
Is there some sort of list of models you want to implement where all components are already there?

Currently the only object detection model being added is RT-DETR at #29077. I think it'd be great to support more recent object detection models which top the rankings at https://paperswithcode.com/sota/object-detection-on-coco (although those are usually high performing but slower). cc also @qubvel
Non blocking support to torch DL's
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR brings in https://github.com/huggingface/accelerate/pull/2685 to transformers.

TL;DR Accelerate's dataloaders can optimize `send_to_device` when sending the data to CUDA directly using `non_blocking` safely. This PR allows the user to configure that as part of the `AcceleratorConfig` via `non_blocking=True`. 

It's also recommended to use `pin_memory=True`, so this will flag a warning letting the user know it's recommended if not enabled.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30465). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.