noamaanMulla-03/transformers-issues huggingface dataset:
Add new Model - SegGPT: Segmenting Everything In Context
Author: raghavanone
Pull request: no
Labels: name: New model

### Model description

Add new Model - SegGPT: Segmenting Everything In Context

The model code and weights are available. 

- Paper : https://arxiv.org/pdf/2304.03284.pdf
- Code : https://github.com/baaivision/Painter/tree/main/SegGPT/SegGPT_inference
- Weights : https://huggingface.co/BAAI/SegGPT/blob/main/seggpt_vit_large.pth

I will be implementing this model.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hey, would you mind adding a link to the paper, the code and the released checkpoints? 😉 

@ArthurZucker 

- [Paper](https://arxiv.org/pdf/2304.03284.pdf)
- [Code](https://github.com/baaivision/Painter/tree/main/SegGPT/SegGPT_inference)
- [Checkpoint](https://huggingface.co/BAAI/SegGPT/blob/main/seggpt_vit_large.pth)
[`cuda kernels`] only compile them when initializing
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #29130, from 1min to 6seconds

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29133). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I'll make sure of that before merging! Testing now!

```bash
FAILED tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelIntegrationTests::test_inference_object_detection_head - AssertionError: False is not true
FAILED tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelIntegrationTests::test_inference_object_detection_head_equivalence_cpu_gpu - AssertionError: assert False
FAILED tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelIntegrationTests::test_inference_object_detection_head_with_box_refine_two_stage - AssertionError: False is not true
```
failing. Logits do no match, failing on main as well

```bash 
FAILED tests/models/deta/test_modeling_deta.py::DetaModelIntegrationTests::test_inference_object_detection_head - AssertionError: False is not true
FAILED tests/models/deta/test_modeling_deta.py::DetaModelIntegrationTests::test_inference_object_detection_head_swin_backbone - AssertionError: False is not true
```
failing as well  on main. Merging

Yoso is alright
Add Fast model 
Author: raghavanone
Pull request: yes
Labels: name: WIP

Add Fast model 
Fix issue #26501

Comments:
FYI @rafaelpadilla to keep an eye on this ! 

Hi @raghavanone,

Thank you for your contribution :) 
Please, ping me when you're done for a first review.

@rafaelpadilla Request you to do a first review.

> Did a first pass on the code, looks good already, my main comment is to leverage the AutoBackbone API for this model as well. This would require adding TextNet as a separate, standalone model first, which implements a TextNetBackbone class as well as a TextNetForImageClassification (I saw the authors released checkpoints fine-tuned on ImageNet-1k).
> 
> Next, one could use the following in modeling_fast.py:
> 
> ```
> from transformers import AutoBackbone
> 
> class FastModel(FastPretrainedModel):
>      def __init__(self, config):
>         self.backbone = AutoBackbone.from_config(config.backbone_config)
> ```
> 
> This is used by models which also leverage backbones like DETR, Deformable DETR, MaskFormer, etc.

The TextNet checkpoint released does not have the classifier head with it. So if pulled out as separate module, then the module will only have the TextNet backbone, witch out any downstream models from it . Is it okay to have it that way ?

@raghavanone We can add both TextNetModel and TextNetForImageClassification. For the backbone, we'd effectively be wrapping the `TextNetModel` as a backbone to be loaded with the `AutoBackbone` api. For TextNetForImageClassification, if a checkpoint doesn't have weights for the classification head, then they will be randomly initialized with a warning. 

@NielsRogge  @rafaelpadilla @amyeroberts  I have changed the implementation to follow backbone style. Requesting for another round of review.

@raghavanone Thanks for the continued work on this model! Could you please split the addition of Fast and TextNet into two separate PRs. We can review TextNet first, and once approved and merged we can review this one. This prevents the diffs being too large for review.
Add support for Phi in Flax
Author: dfdx
Pull request: yes
Labels: name: WIP

name: Flax

# What does this PR do?

This is WIP PR aiming to add Flax support for Phi and Phi-2 models.

Progress so far:

- [x] Implement all modules (heavily inspired by Flax Llama implementation)
- [x] Check consistency with PyTorch (see temporary `_checks.py` file)
- [ ] Implement/check loading from PyTorch checkpoints
- [ ] Check JIT compatibility
- [ ] Add formal tests (instead of REPL-based checks)
- [ ] Remove temporary code fixes (e.g. absolute imports)

I'm currently stuck on loading pretrained models from PyTorch checkpoints and need some help from the community. 

According to the [model sharing](https://huggingface.co/docs/transformers/en/model_sharing) section, I expected the following to work out of the box:

```python
FlaxPhiForCausalLM.from_pretrained("microsoft/phi-2", from_pt=True)
```
but in practice it fails:


> OSError: Can't load the model for 'microsoft/phi-2'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'microsoft/phi-2' is the correct path to a directory containing a file named flax_model.msgpack or pytorch_model.bin.


Note that since I wasn't adding a new model but instead only adapting an existing one for a new framework, I did not generate model template as described in [the docs](https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model) and could miss something important. Could somebody with better understanding of the internal machinery point me to what needs to be done to load PyTorch checkpoints to the Flax model? 

Tagging @ArthurZucker @younesbelkada and @sanchit-gandhi as probably the most relevant people here. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. - Yes, [link](https://github.com/huggingface/transformers/issues/28237)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?





Comments:
Output logits differ for the same input text in a batch of size 1 with half precision on GPU
Author: zhukpm
Pull request: no

### System Info

Linux 20.04.1-Ubuntu x86_64 GNU/Linux 
Python 3.10.12
transformers==4.37.1
torch==2.1.2+cu121

GPU A100
NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

We run an inference with a CausalLM model, providing the same text, but in different batches. One of the batches is of size `1`, and the other - of size `> 1`. Output logits differ *slightly* for the same input sequence.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed


# MODEL_ID = 'mistralai/Mistral-7B-Instruct-v0.2'
MODEL_ID = 'facebook/opt-350m'


model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map='auto',
    return_dict=True
)
model.eval()

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = tokenizer.pad_token_id

batches = [
    ['hello, world'],
    ['hello, world', 'hello', 'world']
]

tokenized = [tokenizer(b, padding='longest', return_tensors='pt').to(model.device) for b in batches]

assert (tokenized[0]['input_ids'][0] == tokenized[1]['input_ids'][0]).all().item()

set_seed(0)
with torch.inference_mode():
    logits = [model(**t).logits for t in tokenized]

assert torch.allclose(logits[0][0], logits[1][0], atol=1e-3)

```

### Expected behavior

Output logits should be the same (at least very close to other) regardless of the batch size. 
Note that we observe this problem only with `torch.float16` and `torch.bfloat16` on GPUs. 

The code above works without errors
- on CPUs
- when using `float32`
- when comparing batches of sizes e.g. 2 and 3:
```python
batches = [
    ['hello, world', 'hello'],
    ['hello, world', 'hello', 'world']
]
```

So for some reason the problem occurs for half precision and `batch_size=1` only. 

I think that [this thread](https://discuss.huggingface.co/t/results-of-model-generate-are-different-for-different-batch-sizes-of-the-decode-only-model/34878) might be related somehow, but I'm not sure.

Comments:
This seems like a duplicate of https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535 

Yeah, it is. Thanks

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add FlashAttention-2 support for Mask2Former model
Author: DanieleVeri
Pull request: no
Labels: name: Feature request

### Feature request

Is it possible to add FlashAttention-2 support to the Mask2Former model? 

### Motivation

Since it is already availble for ViT, it would be great to have it on Mask2Former too.

Maybe the additional input masks to the decoder layer represent a major challenge?

### Your contribution

I could help by testing the implementations.

Comments:
Hey @ArthurZucker I can take this issue on.

feel free to open a PR
Image Feature Extraction docs
Author: merveenoyan
Pull request: yes

This PR adds task guide for image feature extraction.

Note that as of now the image feature extraction pipeline doesn't have a pooler output, so the result in the doc is the theoretical result :') This PR can be reviewed after that change is merged since this wouldn't work without `ViTPooler`. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28973). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@merveenoyan There's a PR to add the pool option - I'm just waiting for review #28985
Fix `attn_implementation` documentation
Author: fxmarty
Pull request: yes

As reported in https://github.com/huggingface/transformers/pull/26572#discussion_r1493074475, `attn_implementation` is wrongfully documented under `PretrainedConfig`, and is not under `AutoModel.from_config` & `PreTrainedModel.from_pretrained` as it should be.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29295). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@fxmarty I don't think this is quite right - if someone can pass in 

```
model = AutoModel.from_config(config, attn_implementation="foo")
```

they should also be able to pass it in for the config constructor and then pass that to the model

```
config = ModelConfig(attn_implementation="foo")
model = AutoModel.from_config(config)
```

@amyeroberts This is not what was suggested in https://github.com/huggingface/transformers/pull/26572#discussion_r1405904274 by @patrickvonplaten.

Happy to change that, but I think it should be in an other PR. This PR simply reflects in the documentation the current expected usage by users:

```python
model = AutoModel.from_config(cfg, attn_implementation="eager")
model = LlamaModel.from_pretrained("xxx", attn_implementation="eager")
```

while the following is illegal/not supported:
```python
from transformers import AutoModelForCausalLM, AutoConfig, LlamaForCausalLM

cfg = AutoConfig.from_pretrained("fxmarty/tiny-llama-fast-tokenizer")
cfg.attn_implementation = "eager"

model = LlamaForCausalLM(cfg)
```

> they should also be able to pass it in for the config constructor

Ultimately I agree with you, `PreTrainedModel.__init__` should in the future have a way to obey to a specified attn_implementation (currently not exposed to users, only `config._attn_implementation = "eager"` works).

@fxmarty OK, thanks for explaining and linking to the relevant comment! If it's something that causes a lot of confusion we can circle back on enabling it through the config creation
Fix llama sin_cached/cos_cached backward compatibility
Author: fxmarty
Pull request: yes

The `_sin_cached` & `_cos_cached` are never set in the init (compare to https://github.com/huggingface/transformers/blob/v4.37.2/src/transformers/models/llama/modeling_llama.py#L134-L136), which yields errors in external packages as backward compatibility is broken (e.g. in https://github.com/AutoGPTQ/AutoGPTQ/blob/6b55300dd83326504ee6e02b730fa4451adfa479/auto_gptq/modeling/_utils.py#L95-L96)

IMO this should be in a patch release.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29299). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@fxmarty The extent of the fix may depend on the following question: are the libraries downstream broken because a) of the lack of the tensors, or because b) the lack of the tensors AND their values?

The PR as it stands would fix a), but it probably wouldn't fix b). 

_________________________________________
Full story of how this came to be:
1. Static cache removed these tensors
2. [This PR](https://github.com/huggingface/transformers/pull/29109) added them, but a) NOT at init time (which this PR would fix as of the latest commit) and b) the contents and shape of the tensors are different 

Related PR: https://github.com/huggingface/transformers/pull/29198 (which tries to fix https://github.com/huggingface/transformers/issues/29173)

Note to ourselves: non-permanent buffers can't be treated as common variables for deprecation purposes 😬 

#29198 will add them at init time. 
We keep the values but we deprecate fast because the values are not longer updated. 

Was not aware https://github.com/huggingface/transformers/pull/29198 was a fix for that, nice! Note that with `self.register_buffer("_cos_cached", ...)` I had some torch.fx tests failing.

Feel free to comment over there
Use torch 2.2 for deepspeed CI
Author: ydshieh
Pull request: yes

# What does this PR do?

Use torch 2.2 for deepspeed CI docker image

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29246). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Skip failing test_save_load_low_cpu_mem_usage tests
Author: amyeroberts
Pull request: yes

# What does this PR do?

Related to #29043 and #28948

Fixes more failing model tests on main which weren't picked up by the test fetcher cc @ydshieh cc @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29110). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I also in favor of reverting the original PR #28948 - if that PR is not something urgent to be in `main`. Adding this many of skip is already a heavy burden, then clean it up later is another one.

(The test fetcher is indeed not ideal, but there is a trade-off and we can decide how to improve it.)

However, I am OK for this PR to be merged if we decide so - the job is already done (the time is spent) anyway.

@ydshieh @ArthurZucker I agree, let's revert for now. 

The main reason is I don't trust the testing suite at the moment. @hackyon opened another, more complete PR to address the failing models #29118 which believe were retrieved from explicitly testing `test_save_load_low_cpu_mem_usage` for all models. Importantly, there were more models covered there than this PR addresses - highlighting lack of coverage and failing tests not being run. 

However, because we're just about to release, let's err on the side of caution.
Tried to attach a forward hook to the self-attention layer but failed.
Author: ArgusK17
Pull request: no

### System Info

- `transformers` version: 4.38.1
- Platform: Linux-4.18.0-477.10.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.18
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I attempted to attach a forward_pre hook to the self attention layer but failed. The input was simply a empty token.  I'm wondering if this could be a bug. If not, perhaps this issue should be assigned to Feature request. XD

```python
from transformers import LlamaForCausalLM, LlamaTokenizer, GenerationConfig
import transformers
import torch

path="./llama_hf/7b"

tokenizer = LlamaTokenizer.from_pretrained(path, padding_side='left')
tokenizer.pad_token = tokenizer.eos_token
model = LlamaForCausalLM.from_pretrained(path, device_map="cuda", attn_implementation="eager")

def forward_pre_hook(module, inputs):
    print(inputs)
    return

module_hook = model.model.layers[0].self_attn  # You need to identify the correct submodule
hook = module_hook .register_forward_pre_hook(forward_pre_hook)

prompts=["test"]
inputs = tokenizer(prompts, return_tensors="pt", padding=True)
input_ids = inputs["input_ids"].cuda()
outputs=model(input_ids)

>>> ()

hook.remove()
```

### Expected behavior

I would greatly appreciate it if someone could take the time to explain why the forward_pre hook failed in this case.

Comments:
From [the doc of torch](https://pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_forward_pre_hook.html):
>The input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward. 

The forward is defined by:
```python 
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            cache_position=cache_position,
            **kwargs,
        )
```
Add compatibility with skip_memory_metrics for mps device
Author: SunMarc
Pull request: yes

# What does this PR do 
This PR makes mps device compatible with memory profiling integrated in Trainer (`skip_memory_metrics`=False). However, we can't get the information on the peak memory since` torch.mps.max_memory_allocated()` doesn't exist yet. 
Fixes #27181
Tested on mac and I get the following output for a simple Trainer [example](https://gist.github.com/SunMarc/ccd7704cb5d5668aa5d15f5d02c712bf):

`{'train_runtime': 7.8136, 'train_samples_per_second': 24.572, 'train_steps_per_second': 3.072, 'train_loss': 5.870065689086914, 'init_mem_cpu_alloc_delta': 950272, 'init_mem_gpu_alloc_delta': 512, 'init_mem_cpu_peaked_delta': 16384, 'init_mem_gpu_peaked_delta': 'Not available', 'train_mem_cpu_alloc_delta': 38354944, 'train_mem_gpu_alloc_delta': 2304, 'train_mem_cpu_peaked_delta': 16384, 'train_mem_gpu_peaked_delta': 'Not available', 'before_init_mem_cpu': 351813632, 'before_init_mem_gpu': 0, 'epoch': 3.0}`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29264). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `torch.compile` with `fullgraph=True` when `attention_mask` input is used
Author: fxmarty
Pull request: yes

As per title.

Fixes https://github.com/huggingface/transformers/issues/29190

Comments:
Let's consider using https://github.com/pytorch/pytorch/pull/120400 if this is accepted and released.

> I guess we don't have a choice?

The other choice would be:

```python
            is_tracing = (
                torch.jit.is_tracing()
                or isinstance(input_tensor, torch.fx.Proxy)
                or torch._dynamo.is_fullgraph_tracing())
            )
```

but `torch._dynamo.is_fullgraph_tracing` does not exist in PyTorch.

One other possibility is to always do `causal_mask = causal_mask.mul(~torch.all(causal_mask == causal_mask.min(), dim=-1, keepdim=True))` no matter what.

One other possibility is to not use `causal_mask = causal_mask.mul(~torch.all(causal_mask == causal_mask.min(), dim=-1, keepdim=True))` at all, but that means that we drop support for memory-efficient attention backend in Transformers cc @drisspg

One other possibility is to move the causal mask logic outside of the modeling code.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29211). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the fix!
Just wanted to share new API offerings from PyTorch (as of a couple days ago):
https://github.com/pytorch/pytorch/pull/119602

Summary:
- A more general flag is named `torch.compiler.is_compiling()`. This flag indicates whether a graph is traced/compiled via `torch.export()` or `torch.compile()`. The flag works even in non-strict mode (i.e. even without TorchDynamo).
- A more specific flag is named `torch.compiler.is_dynamo_compiling()`, it's stricter, because it's only set to True when TorchDynamo is used, so, in non-strict mode it would be False.

Cc: @mreso @khabinov

@kwen2501 Thank you! torch.export is not always using dynamo? Reading https://pytorch.org/docs/stable/export.html I thought so!