noamaanMulla-03/transformers-issues huggingface dataset:
Make `is_torch_bf16_available_on_device` more strict
Author: ydshieh
Pull request: yes

# What does this PR do?

The layernorm op is required to be supported in order for this function to return `True`.


### detail
Previously, the function `is_torch_bf16_available_on_device` check
```python
        x = torch.zeros(2, 2, dtype=torch.float16).to(device)
        _ = x @ x
```
With torch < 2.2 on CPU, this will give

> RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'

and `is_torch_bf16_available_on_device` returns `False`.

With torch 2.2, this doesn't fail and the function return `True`. However, many models use `LayerNorm` and this is still not supported by torch 2.2 on CPU. We then get many failures for fp16 tests on CircleCI (cpu only)



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28796). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, but we get the same set of tests to run as before. Nothing change. And if on GPU, fp16 test will be run anyway (it won't reach the line that I changed in this PR). The change here only affects CI on CPU (CircleCI)
Fix symbolic_trace with kv cache
Author: fxmarty
Pull request: yes

We should NOT trace models with 0-shaped concrete metas as we otherwise miss https://github.com/huggingface/transformers/blob/bb6aa8bc5ff8537f58c4b6ac80611101ba556226/src/transformers/modeling_attn_mask_utils.py#L162-L163 in the captured graph.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28724). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you @amyeroberts, added a test that fails on the current `main` due to the controlflow not being correctly captured.
rewrite trainer's save_model method get unexpected pytorch_model.bin file
Author: Chandler-Bing
Pull request: no

### System Info

- `transformers` version: 4.33.1
- Platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes, deepspeed zero2

### Who can help?

@pacman100 @muellerzr 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. I rewrite trainer's `save_model` method below
```
def save_model(self, output_dir=None, _internal_call=False):
        os.makedirs(output_dir, exist_ok=True)
        self.model.save_pretrained(output_dir)
```
2. training info
- deepened zero2
- 1 node, 8 GPUs
3. seems `save_pretrained` has default `max_shard_size=10GB`  so I expect 2 bin files each less than 10GB. however I get one 14GB  pytorch_model.bin. **why?**
4. I find that if  I didn't rewrite `save_model` , it behave normal. and the  execute code in trainer.py line 2784
```
elif self.is_deepspeed_enabled:
            # this takes care of everything as long as we aren't under zero3
            if version.parse(accelerate_version) <= version.parse("0.20.3"):
                raise ValueError("Install Accelerate from main branch")
            try:
                state_dict = self.accelerator.get_state_dict(self.deepspeed)
                if self.args.should_save:
                    self._save(output_dir, state_dict=state_dict)
            except ValueError:
                logger.warning(
                    " stage3_gather_16bit_weights_on_model_save=false. Saving the full checkpoint instead, use"
                    " zero_to_fp32.py to recover weights"
                )
                self._save(output_dir, state_dict={})
                # remove the dummy state_dict
                remove_dummy_checkpoint(self.args.should_save, output_dir, [WEIGHTS_NAME, SAFE_WEIGHTS_NAME])
                self.model_wrapped.save_checkpoint(output_dir)
```
`state_dict = self.accelerator.get_state_dict(self.deepspeed)` **what's the difference between  `self.accelerator.get_state_dict(self.deepspeed)` and `self.model.state_dict()` ?** 


### Expected behavior

says above  
thanks for answering

Comments:
Hello, this does not seem to be a direct issue, so would recommend you to ask on[ the forum ](https://discuss.huggingface.co/) 🤗
Adding Gradient Checkpointing and Flash Attention 2 implementation to VisionTextDualEncoderModel
Author: Syrinechen
Pull request: no
Labels: name: Feature request

name: Good Difficult Issue

VisionTextDualEncoderModel allows training any image and text encoders with a contrastive loss. Would be convenient to add gradient checkpointing as well as flash attention 2 to optimize training. 

Thank you 

Comments:
Does generate() support "Export to TorchScript"
Author: gongel
Pull request: no

### Feature request

Does T5 `model.generate` support "Export to TorchScript"
https://huggingface.co/docs/transformers/main/en/torchscript

I want to make a faster deployment and transform `model.generate` like this:
https://discuss.huggingface.co/t/t5-base-not-torchscriptable/11173

thanks all


Comments:
cc @gante


Hey @gongel 👋 The model foward pass probably can be serialized, but the full `model.generate` cannot. We are working on the serialization of `model.generate` as we speak, in the context of PyTorch dynamo.

Can I be of further assistance? :)

Thank you, I'm looking forward to it

Is there an issue tracking this? What's the status of `TorchScript`ing `model.generate`? Or can `generate` be a standalone function and call `TorchScript` module?

@ZisIsNotZis AFAIK no tracking issue. We are exploring generation speedups (which will likely include static shapes, i.e. should be TorchScript compilable) at the moment, to be released in ~3 months.

Is there any update here, I also met the same problem when i export model with model generate to TorchScript, is this support now? 

I think that #27931 might help for this and will kickstart the effort to support compile so a bit related (functional logits processors etc)
Add tip on setting tokenizer attributes
Author: Rocketknight1
Pull request: yes

This PR adds a quick tip to the chat template docs on setting tokenizer attributes (after some discussion on Slack)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28764). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix some docs and tensor device bug
Author: zspo
Pull request: yes

# What does this PR do?

Fixes
1. fix missing spaces in parameter descriptions
2. fix tensor device
3. add parameter description

## Who can review?

@ArthurZucker @amyeroberts 



Comments:
Hi @zspo, thanks for opening this PR and contributing to the repo! 

This PR contains a handle of unrelated changes. Let's split up the doc changes from the device placement ones.
GPT2 minicons surprisal:  IndexError: index out of range in self
Author: joyce9936
Pull request: no

### System Info

I am trying to calculating surprisal value by feeding in a txt file with about 5000 sentences. But there is a error message I encounter:  **IndexError: index out of range in self** Can anyone help with this issue? 


Thank you!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here is the code:
<img width="1337" alt="Screenshot 2024-01-31 at 9 35 06 PM" src="https://github.com/huggingface/transformers/assets/119527282/486ae5fc-2a9f-4f04-a518-99fba53a7775">

Here is the error message: 
<img width="1337" alt="Screenshot 2024-01-31 at 9 34 13 PM" src="https://github.com/huggingface/transformers/assets/119527282/8e50c012-276c-4a4d-810f-39a212280e55">

### Expected behavior

I would like to have the surprisal value for each word for the whole text file.

Comments:
Sorry but this seems unrelated to transformers and rather related to the `scorer` library.
[docs] fix some bugs about parameter description
Author: zspo
Pull request: yes

# What does this PR do?

Fixes
1. fix missing spaces in parameter descriptions
2. add parameter description

@amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add models from deit
Author: rajveer43
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

part of  #28301
- [ ] #28301 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@stevhliu 

 -->


Comments:
@stevhliu

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28302). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] Backbone
Author: stevhliu
Pull request: yes

This PR adds some updates to the backbone docs:

- have API references for `AutoBackbone`, `BackboneConfig`, `BackboneConfigMixin`, `TimmBackbone`, and `TimmBackboneConfig` in the docs so users can easily check them out
- include a list of supported backbones
- break up and move the content into `autoclass_tutorial.md` and `create_a_model.md`
- move initializing backbone from config to `create_a_model.md` which is more similar to the other examples we have there for creating something from their config
- update with loading a backbone by setting for example `config = MaskFormerConfig(backbone="microsoft/resnet50", use_pretrained_backbone=False)` - from #28214

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28739). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hmm not sure why the test is failing with `ERROR dummy.py - AttributeError: 'HfDoctestModule' object has no attribute 'getini'`. Any idea how to fix or if I can go ahead and merge?

we can merge but let's rebase on main it should fix it!
Add missing None check for hf_quantizer
Author: jganitkevitch
Pull request: yes

Adds None check for hf_quantizer that otherwise can blow up when `from_pretrained` is called with `quanitization_config=None`.


Comments:
@younesbelkada @ArthurZucker @poedator 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28804). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Added a test. PTAL – I'm not familiar with HF's testing guidelines, so happy to take suggestions.

@ArthurZucker the test seems unrelated & flaky?

Fixes #28831

@jganitkevitch thank you for the quick PR, as this is a bit sensitive we'll merge now!
Fix issues caused by natten
Author: ydshieh
Pull request: yes

# What does this PR do?

Don't really know what happened, but from the suggestion

> E   Failed to import NATTEN's CPP backend. This could be due to an invalid/incomplete install. Please uninstall NATTEN (pip uninstall natten) and re-install with the correct torch build: shi-labs.com/natten

I changed the installation command. It works, see https://app.circleci.com/pipelines/github/huggingface/transformers/83676/workflows/36ffade9-c7e6-49c2-b8f4-bb5fcb016a83/jobs/1079519



Fix current error we have

> __________ ERROR collecting tests/models/dinat/test_modeling_dinat.py __________
../.pyenv/versions/3.8.12/lib/python3.8/site-packages/natten/functional.py:28: in <module>
    from natten import _C
E   ImportError: /home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/natten/_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops10zeros_like4callERKNS_6TensorESt8optionalIN3c1010ScalarTypeEES5_INS6_6LayoutEES5_INS6_6DeviceEES5_IbES5_INS6_12MemoryFormatEE

Comments:
> might have nightly issues no?

sorry, what do you mean? You mean the daily CI? I haven't have the time to check, recently quite a lot of changes arriving

I mean we might want to change this for the daily ci as well 😉 

Sure, but let's unblock red CircleCI first (I'm  fine but you probably need it's green more than me need it 😄 )

Yep that's why I approved! 👍🏻 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28834). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix / skip (for now) some tests before switch to torch 2.2
Author: ydshieh
Pull request: yes

# What does this PR do?

Some changes are necessary to switch to 2.2. Others are flaky even in torch 2.1.

Comments:
Note, this PR doesn't unpin torch 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28838). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
PatchtTST and PatchTSMixer fixes
Author: wgifford
Pull request: yes

# What does this PR do?

Makes PatchTST and PatchTSMixer interfaces more consistent -- using similar parameter names for method arguments and returned data objects.

Fixes a few minor bugs in PatchTST implementation.

Ensures more consistent output shapes with regression when an output_distribution is chosen (in both forward and generate methods).

Fixes slow tests for PatchTST.

Comments:
Hi @wgifford, thanks for opening this PR! 

Let us know when it's ready for review! 

Hi @amyeroberts I just rebased on current main. I think it is ready for review.

thanks @amyeroberts I'll fix it up and push suggestions

> thanks @amyeroberts I'll fix it up and push suggestions

Hi Kashif, I think all the suggestions are good. Please reach out if any issues.

@kashif I see this is passing -- any other issues to address before we merge?

@amyeroberts Can this be merged? Some of these fixes are critical.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28083). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks for iterating on this - LGTM!

Thank you @amyeroberts!
Use `-v` for `pytest` on CircleCI 
Author: ydshieh
Pull request: yes

# What does this PR do?

So we get info about which process run which tests

> [gw4] [ 99%] PASSED tests/models/kosmos2/test_processor_kosmos2.py::Kosmos2ProcessorTest::test_model_input_names 
tests/models/kosmos2/test_processor_kosmos2.py::Kosmos2ProcessorTest::test_processor 
[gw3] [ 99%] PASSED tests/models/pix2struct/test_processor_pix2struct.py::Pix2StructProcessorTest::test_model_input_names 
tests/models/pix2struct/test_processor_pix2struct.py::Pix2StructProcessorTest::test_processor 
[gw0] [ 99%] PASSED tests/models/bloom/test_tokenization_bloom.py::BloomTokenizationTest::test_training_new_tokenizer 

Full CI will produce ~100K lines. The output is also saved and uploaded as artifact.

Comments:
Example run

https://app.circleci.com/pipelines/github/huggingface/transformers/83697/workflows/6b7bf7c2-cf98-474c-a3dd-3eea4cfa0cfa

For context for anyone coming cold to this PR - the motivation for introducing this is to enable retrieving which tests were run on which process to help when debugging.

Recently we had an issue with flaky tests, because some tests were affecting other tests. For example, the global logger state - which was resolved in #28638. Failures like these only happen sporadically because the interacting tests aren't always in the same process. 

Having the outputs means we can easily narrow down the culprit tests to a subset of size `1 / pytest_num_workers`. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28840). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I guess it's still not (super) easy to achieve re-run the subset of tests that were run in a process (grabbed from log). For example, we might get 20000 test names. It won't work if we concatenate those many test names and paste that long line to the terminal to run the test. 

We can have some utility to grab the folders (their name) seen by a process. This is more realistic, but not urgent task. I could work on this later.
Reduce GPU memory usage when using FSDP+PEFT
Author: pacman100
Pull request: yes

# What does this PR do?
1. For FSDP+PEFT, When using `use_orig_params=True` which is now the default in Accelerate, there is no memory savings when compared to FSDP full fine-tuning. We have to set `use_orig_params=False` to realize the memory savings which makes it difficult to be in line with Accelerate’s minimal API. However, this means that the model needs to be wrapped in FSDP unit before the creation of the optimizer. This PR wraps the model in FSDP before the creation of optimizer so that GPU memory savings are realized when using FSDP+PEFT.

![Screenshot 2023-12-27 at 8 48 12 PM (2)](https://github.com/huggingface/transformers/assets/13534540/4156e0b1-b84a-4a32-87ad-6d111fb2f81d)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28830). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] HfQuantizer
Author: stevhliu
Pull request: yes

A follow-up PR to tidy up some things for the `HfQuantizer`:

➕ `HfQuantizer` to the API docs and links to the relevant files for faster navigation for users
➖ I think all those emojis are a bit too much and distracting 😅 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28820). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Mark `test_encoder_decoder_model_generate` for `vision_encoder_deocder` as flaky
Author: amyeroberts
Pull request: yes

# What does this PR do?

Marks the tests as flaky to avoid blocking PRs. 

Reference failling CI run: https://app.circleci.com/pipelines/github/huggingface/transformers/83611/workflows/666b01c9-1be8-4daa-b85d-189e670fc168/jobs/1078635/tests#failed-test-0

Reference issue for tracking: #28841


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28842). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks, everyone has a nice Friday evening now :-) I hope