noamaanMulla-03/transformers-issues huggingface dataset:
Fine-tuning a pruned model
Author: Megh-Thakkar
Pull request: no
Labels: name: Feature request

### Feature request

I used [OWL](https://github.com/luuyin/OWL) to prune a Mistral-7b model and would like to further train this pruned model. Is there a way to pass a global mask (based on the pruned model) to ensure those weights don't get updated, particularly when using DDP.

**Note**:  We cannot update the parameter list of the optimizer as the pruning is not structured.

### Motivation

Pruning helps with creating more efficient models, and it would be interesting to analyze the effects of further training pruned models.

### Your contribution

We would be glad to submit a PR for this use case. 

Comments:
FYI @amyeroberts and @NielsRogge 

@ArthurZucker I think the OWL being referred to here is a library for pruning, rather than owl-vit 🦉 

@Megh-Thakkar I don't know - maybe @younesbelkada has an idea? 

Hi @Megh-Thakkar 
Thanks for the feature request! 
Currently indeed this is not supported out of the box, also I am not really familiar with OWL library. Would you like to sketch up a draft API of what you have in mind in order to support training pruned models? Can you also elaborate more on

> Note: We cannot update the parameter list of the optimizer as the pruning is not structured.
WandbCallback overwrite all log at summary step
Author: nmd2k
Pull request: no

### System Info

wandb = 0.16.3
transformers = 4.37.1

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When running fine-tuning (on multiple-gpus) using HF Trainer, I notice Wandb logger eventually overwrite all experiment's log at the final summary step, but it work normally while running.

=> While running:
![image](https://github.com/huggingface/transformers/assets/48562727/2893e3b0-482f-4f5d-b3de-8f34c5118f24)

=> At the end:
![image](https://github.com/huggingface/transformers/assets/48562727/975a07aa-d5fe-4e75-bdd3-51d0826fc521)


### Expected behavior

This might due to this line, define all other log to be summary (and overwrite) the log variables at the summary step: 

https://github.com/huggingface/transformers/blob/c38a12270a11e237cf65d085fbbaf0c4b7976b67/src/transformers/integrations/integration_utils.py#L751


Comments:
cc @muellerzr 

cc @parambharat since you've done a lot of work with our wandb plugin 🤗 

@nmd2k : Thanks for raising the issue. Can you check the next page in your train panel to see if you have the `train/loss` graph. Your metrics should still be present and not overwritten. Although, from the looks of it there are additional summaries `train/train_loss` etc that are being logged without the correct prefix handling. This needs further investigation. I'll investigate and update here soon.

@nmd2k : Please take a look at the PR and let us know if this fixes the issue you are facing.
CLI: detect and store weights as float16
Author: gante
Pull request: yes
Labels: name: WIP

# What does this PR do?

Updates the `pt-to-tf` CLI to detect and store weights as float16. Battle-tested with OPT.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17737). All of your documentation changes will be reflected on that endpoint.

> Will TensorFlow automatically load stored FP16 weights in FP32 like PyTorch does?

No, if we inspect the variables after loading, they are FP16. However, as soon as we pass some input, we can see that the outputs of the internal computations are FP32, despite the weights being FP16.

The model error (vs PT) is exactly the same, before and after adding these lines :)

The documentation is not clear, but I believe `tf.keras.backend.set_floatx` sets the precision of the internal computations. For instance, if we don't reset to float32 (the default) after storing as float16, the error is much much larger (~1e3 times larger).

Mmmm, in this case I would avoid storing the weights in FP16 on the Hub before adding something in `from_pretrained` that will convert them back to FP32 like it's done for PyTorch.

> Mmmm, in this case I would avoid storing the weights in FP16 on the Hub before adding something in from_pretrained that will convert them back to FP32 like it's done for PyTorch.

👍 I can give it a go (and leave this PR open meanwhile)
Examples: check `max_position_embeddings` in the translation example
Author: gante
Pull request: yes

# What does this PR do?

Fixes #28104 

When a model with absolute position embeddings (such as Marian) is used in the translation training example, we must confirm that the input sequence length argument is smaller than its maximum sequence length. Otherwise, the data gets prepared with a larger sequence length, resulting in indexing errors.

This PR adds an informative exception, explaining what can be fixed when that happens :)

[Note: models with relative position embeddings, like T5, don't need this check. They can handle arbitrary sequence lengths.]

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29600). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`Gemma`] Supports converting directly in half-precision
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, this PR make the Gemma conversion script more modulable to be able to load the model directly into the desired dtype without having to load it in fp32 all the time

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29529). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Is it a BUG when check all the imports in which may not be used？
Author: Tyx-main
Pull request: no

### System Info

transformers：4.38.1


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I load a model such as deepseek-moe as follow：

`
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
`

If there is a snippet like this in modeling_deepseek.py:

`
if is_flash_attn_2_available():
    from flash_attn import flash_attn_func, flash_attn_varlen_func
    from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input  # noqa
`

If I have not install flash_attn, there will be an ERROR in dynamic_module_utils.py, line 180, in check_imports:

`
ImportError: This modeling file requires the following packages that were not found in your environment: flash_attn. Run pip install flash_attn 
`

It seems not reasonable, it is a BUG when check all the imports in which may not be used？


### Expected behavior

The codes can run when I have not install flash_attn

Comments:
Hey! Sorry which modeling code are you referring too? 
Deep seek moe is not part of `transformers`

thanks, maybe I should ckeck out again
after batch_encode_plus generate returns sequence of EOS tokens 
Author: tempdeltavalue
Pull request: no

### System Info

Here's the screenshot which is show this issue
![Screenshot 2024-02-03 210015](https://github.com/huggingface/transformers/assets/36921178/2492512e-c555-4870-be15-f38fea07e516)


HF - discussion:
https://discuss.huggingface.co/t/gpt2-returns-sequence-of-endoftext-after-finetuning/70418

HF topic - discord:
https://discord.com/channels/879548962464493619/1201159949288488970

thank you..



### Who can help?

Would be great to get some advice about it
@gante @ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

To reproduce please check this notebook and repo with the code 
https://github.com/tempdeltavalue/temp_l/blob/main/finetune_seq2seq.ipynb


### Expected behavior

I want to encode list of strings to batch and pass it to generate and get output without EOS sequence 

Comments:
Most probably a padding side issue. If you skip the special tokens you should no longer see them when decoding. 
It is planned to get rid of this function anyway to keep only encode and decode for simplicity! 
I recommend to use encode

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Core: Fix copies on main
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fix fix copies on main after https://github.com/huggingface/transformers/pull/28295 being merged by myself 

cc @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29624). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Warn about tool use
Author: LysandreJik
Pull request: yes

This PR adds additional warnings when using tools through the `from_hub` and `load_tool` methods to explicitly mention that this loads tools defined remotely.

This is done in addition to:
- the existing disclaimers within the runtime mentioning that you're downloading new files that you really should inspect
- the python method and class names which are already indicative of remote downloads

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29628). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Whisper] Deprecate forced ids for v4.39
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Supersedes #29433 by deprecating forced decoder ids in preparation for the v4.39 release. The full Whisper CI was run with these changes, where it was confirmed all slow tests passed.


cc @zucchini-nlp @ArthurZucker (and also @gante for generation)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29485). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Indeed - Whisper was the only model to use this processor, so we should now deprecate it. Note that since the `ForceTokensLogitsProcessor` is importable from Transformers as a public class, and given there is currently no deprecation warning, we need to do this in 2-stages again: first with a `FutureWarning` explaining that this processor will be removed in v4.40, and second with a hard error.

```python
from transformers.generation.logits_process import ForceTokensLogitsProcessor

force_token_map = {1: 2, 3: 2}
logits_processor = ForceTokensLogitsProcessor(force_token_map)
```

_c.f._ #29487 for the first stage of deprecation

Great, thanks! I did not know that the it will be deprecated this soon 😅
[generate] deprecate forced ids processor
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

First stage of deprecation for the `ForceTokensLogitsProcessor`. This logits processor was exclusively used for Whisper. However, the PR #29485 removed the last calls to this processor from Whisper, in favour of `decoder_input_ids` set internally by generate, meaning this processor can now be removed.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29487). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix minor typo: infenrece => inference
Author: DriesVerachtert
Pull request: yes

No code changes: it just fixes a typo in the documentation

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29621). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Mask2Former slowdown starting from version 4.32.0
Author: matteot11
Pull request: no

### System Info

- System information: x86_64 GNU/Linux (with Titan RTX GPU)
- Ubuntu version: 18.04
- Python version: 3.8.12
- CUDA version: 11.1
- PyTorch version: 2.0.1
- transformers version: 4.32.0

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import Mask2FormerForUniversalSegmentation
import time

device = torch.device("cuda:0")
model = Mask2FormerForUniversalSegmentation.from_pretrained(
                "facebook/mask2former-swin-tiny-coco-instance",
                ).eval().to(device)
dummy_input = torch.randn((2,3,640,640)).to(device)

times = []
with torch.no_grad():
    for i in range(100):
        t1 = time.time()
        out = model(dummy_input)
        t2 = time.time()
        times.append(t2-t1)
print(sum(times)/len(times))
```
The code above computes average forward time for Mask2Former through 100 iterations.
The following average forward time is obtained with different ```transformers``` versions:
- ```transformers==4.31.0```-> ~0.133 s
- ```transformers==4.32.0``` -> ~0.405 s
- ```transformers==4.33.1``` -> 0.507 s

Versions 4.32.0 and 4.33.1 introduced, respectively:
- https://github.com/huggingface/transformers/pull/25297
- https://github.com/huggingface/transformers/pull/25741

I also report here my original torchscript export issue, form which the above PRs originated:
https://github.com/huggingface/transformers/issues/25261

### Expected behavior

I would expect similar (or slightly higher) inference times for Mask2Former after einsum removal and memory load reduction.

Comments:
cc @amyeroberts 

Hi @matteot11 - thanks for raising! Looking into it 🕵️ 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @matteot11 - I'm still looking into this. You're completely correct in your diagnosis of the offending PRs. The difficulty is that these were fixing other issues e.g. enabling tracing for the model. At the moment I'm looking into how to improve this e.g. profiling for other places to improve, rewriting the current logic or reenabling einsum (some tracing issues have been fixed in torch)
[`MaskFormer`, `Mask2Former`] Use einsum where possible
Author: amyeroberts
Pull request: yes

# What does this PR do?

einsum operations were removed from mask2former, oneformer and maskformer in #25297 to enable tracing on the models. This was with pytorch 2.0.1

This however, causes a large memory footprint, partially addressed in #25741. However, the models remain very slow. 

Testing on recent versions of python >= 2.1 tracing now works for einsum operations. 

This PR adds back the previous einsum operations, falling back to the slow but compatible versions when pytorch < 2.1 and the model is being traced. 

Fixes #27650


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29544). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Llama: allow custom 4d masks
Author: gante
Pull request: yes

# What does this PR do?

Fixes #29525 

Reintroduces the ability to pass custom 4D attention masks, which was removed in the static cache transition. The following tests are now passing
```
RUN_SLOW=1 python -m pytest -v ./tests/test_modeling_utils.py::Mask4DTestFP32
RUN_SLOW=1 python -m pytest -v ./tests/test_modeling_utils.py::Mask4DTestFP16
```

cc @ArthurZucker after you come back from holidays, have a look at this PR :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29618). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Only question before merge is how come this is only needed for the gemma and llama models?

@amyeroberts They are the only models that have received the static cache treatment. The static cache transition did not foresee this case in the original diff :) 

We are finalizing support on the `generate` side before we propagate this pattern across the library! (https://github.com/huggingface/transformers/pull/29374)
Bug:    TypeError: 'AcceleratorConfig' object does not support item assignment
Author: heyaudace
Pull request: no

### System Info

transformers: transformers-4.38.0
accelerate: accelerate-0.27.2
platform: linux
python: 3.10

### Who can help?

Training using accelerate (code section below in the trainer) fails using latest transformers (versions mentioned above)

```
        if is_accelerate_available():
            if not isinstance(self.accelerator_config, (AcceleratorConfig, dict)):
                if self.accelerator_config is None:
                    self.accelerator_config = AcceleratorConfig()
                else:
                    self.accelerator_config = AcceleratorConfig.from_json_file(self.accelerator_config)
            if self.dispatch_batches is not None:
                warnings.warn(
                    "Using `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use"
                    " `--accelerator_config {'dispatch_batches':VALUE} instead",
                    FutureWarning,
                )
>               self.accelerator_config["dispatch_batches"] = self.dispatch_batches
```

```
               TypeError: 'AcceleratorConfig' object does not support item assignment
```

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run training using accelerate with environment mentioned above. 

### Expected behavior

Not expected to crash. Issue doesn't arise with transformers version less than one above 

----
Additional information: 
- Seems to come from https://github.com/huggingface/transformers/blame/fc37f38915372c15992b540dfcbbe00a916d4fc6/src/transformers/training_args.py#L1751 
- PR https://github.com/huggingface/transformers/pull/28664 (cc @muellerz) 

Comments:
Cc @muellerzr and @ArthurZucker as it's a possible breaking change on the latest release

+1, fails from python launcher as well, not just accelerate launcher.

seems workaround is to replace `split_batches: True` in trainingargs with:

```
        accelerator_config={
            "dispatch_batches": True,
        },
``` 

guess deprecation just came a little early :)

Thanks for the fix @muellerzr 

Thx! But I‘m still confused about how can I fix it ,by upgrading transformers?

@Haruka1307 this isn't yet part of a version release. To have this recent fix, you'll need to install from source: `pip install git+https://github.com/huggingface/transformers`
Whisper remove forced_decoder_ids for longform generation
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #29287 . As I wrote in the comments to the issue, the error is related to using `forced-decoder-ids` processor in longform generation, which suppresses all logits to `-inf`. 

i see that `forced-decoder-ids` will be deprecated, but setting it to `None` will work for now.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sanchit-gandhi  @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29433). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I made the full updates in #29485 - feel free to take a look
[`Mask2Former`] Move normalization for numerical stability
Author: amyeroberts
Pull request: yes

# What does this PR do?

Moving the normalization before the matmul operation makes the calculation more stable and less likely to overflow. 

Same differences introduce in #26086 which was closed after becoming stale. 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29542). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.