noamaanMulla-03/transformers-issues huggingface dataset:
[DOCS] Fix typo for llava next docs
Author: aliencaocao
Pull request: yes

fix typo in `BitsandBytesConfig` import in the example code
cc @stevhliu @MKhalusova


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29829). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
What optimizations are available for AutoModelForVision2Seq
Author: FurkanGozukara
Pull request: no

Hello. I am using AutoModelForVision2Seq for Kosmos 2 model like below

```
    model_source = "microsoft/kosmos-2-patch14-224"

    model = AutoModelForVision2Seq.from_pretrained(model_source).to("cuda")
    processor = AutoProcessor.from_pretrained(model_source)
```

I checked this link but couldn't find anything regarding optimizations

Such as load as BF16?
Enable xformers?
Enable CPU offloading?
anything that can reduce VRAM usage, quantize or speed up inference?

Thank you

https://huggingface.co/docs/transformers/model_doc/auto

transformers==4.37.2

### Who can help?

text models: @ArthurZucker and @younesbelkada 
vision models: @amyeroberts
generate: @gante
pipelines: @Narsil
Big Model Inference: @SunMarc


Comments:
Hi @FurkanGozukara !
Thanks for the issue !

`AutoModelForVision2Seq` inherits from HF transformers' `PreTrainedModel`, therefore it benefits from most of the optimizations that are available, for instance you can load it in fp16, bf16 as usual through the `torch_dtype` argument, CPU offloading through device_map, and you can also load the model in 4-bit / 8-bit through the `load_in_kbit=True` argument (some models require to have the `accelerate` integration, so if it's not the case you can let us know). 

For more "advanced" optimisations such as SDPA or Flash Attention 2, it is on per-architecture basis, for your specific question you are asking for `microsoft/kosmos-2-patch14-224`. From what I can see kosmos2 does not support SDPA or FA2 so you can open a separate issue as a feature request so that we can work on adding that feature

> Hi @FurkanGozukara ! Thanks for the issue !
> 
> `AutoModelForVision2Seq` inherits from HF transformers' `PreTrainedModel`, therefore it benefits from most of the optimizations that are available, for instance you can load it in fp16, bf16 as usual through the `torch_dtype` argument, CPU offloading through device_map, and you can also load the model in 4-bit / 8-bit through the `load_in_kbit=True` argument (some models require to have the `accelerate` integration, so if it's not the case you can let us know).
> 
> For more "advanced" optimisations such as SDPA or Flash Attention 2, it is on per-architecture basis, for your specific question you are asking for `microsoft/kosmos-2-patch14-224`. From what I can see kosmos2 does not support SDPA or FA2 so you can open a separate issue as a feature request so that we can work on adding that feature

Awesome

this model is simply just amazing

Do you suggest FP16 or BF16?

I added 4, 8, 16 and 32 bit options
By default it was running at 32 bit

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`get_torch_version()` doesn't return same result as `import torch; torch.__version__` 
Author: relh
Pull request: no

### System Info

ubuntu linux with conda and pip

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`get_torch_version()` doesn't return same result as `import torch; torch.__version__` 

Hi, if you happen to have a version of torch installed with `pip` and one with `conda`, there can be an issue where the `import_utils` function doesn't return the same thing as actually importing.

This was recently an issue for me because I had a version `2.1.0` and a version `2.2.0` and the naming for `_torch_pytree.register_pytree_node` had been `_torch_pytree._register_pytree_node`. 

The solution is someone shouldn't have both installed (duh) but this problem could be avoided if line 311 in `generic.py` checked for the existence of a function and then fell back instead of parsing a version. 

### Expected behavior

1. have 2 versions of torch installed in an environment
2. use code that involves `generic.py` 
3. ???
4. profit 

Comments:
Hi @relh, thanks for raising this issue! 

Indeed, despite the promised profit, I don't think this is something we should be handling within `generic.py`. Assuming there's one version installed is a fair one, and being able to parse the torch version is something we do throughout the library e.g. [here](https://github.com/huggingface/transformers/blob/5f06053dd821c91f7bd697309109abaa3396b605/src/transformers/pytorch_utils.py#L31). It's better to have this failure hit with the parsed version not matching and raising and error because a function couldn't be found instead of trying to do some magic on the inspection. After all, it caught this problem :)!  

We should expect `get_torch_version` to match `import torch; torch.__version__` though. I don't think this is an easy fix, as the reason this is happening is `importlib` isn't reading the "correct" torch. The question is, how can we know which is the "correct" version without importing (which `_is_package_available` does)?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`dataset.map` for tokenization hangs at 60%
Author: sarahpannn
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", use_fast=True)
tokenizer.pad_token = tokenizer.eos_token
dataset = load_dataset("sarahpann/AMPS")

dataset = dataset.map(lambda x: tokenizer(x["problem"] + x["step_by_step"], truncation=True, max_length=2048))
```

### Expected behavior

The map freezes at 60%. I've also tried a similar thing with batched tokenization, but that freezes at around 93%. There don't seem to be issues with the dataset and weird Unicode characters. A simple for loop method also freezes at 60%. The specific example it freezes at varies by run.

Comments:
Hey, I ran this without issues, not stuck but just super slow. Might be a lot of things from the tokenizer to the dataset to your CPUs

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
About new ClearML Intergrations
Author: dimka11
Pull request: no
Labels: name: Feature request

name: Integrations

### Feature request

We need add set / filter metrics that displayed on ClearML plots. It's hard to use it with a tons of metrics that have diffrerent scale on the one plot.

### Motivation

This new implementation ClearML callback's output all metrics from log as multiple series at one plot, that's looks not readable. 
from https://github.com/huggingface/transformers/pull/28559
![image](https://github.com/huggingface/transformers/assets/6096108/614fabc2-0d1c-4320-b4cc-7ae61bbae88d)
For example: runtime and samples_per_seconds has different scale than other metrics.

Previous version report metrics as one metric - one plot.

Any ideas how can we set / filter metrics without code changes?
As temporarily solution I add my own callback for send some metrics to ClearML at other plots.

### Your contribution

Yes, I can.

Comments:
Hi @dimka11, thanks for raising this! 

We don't actively maintain integrations like ClearMI, instead they're managed by their original contributors. We obviously want the integrations to be as good as possible. If there's an improvement you have in mind, feel free to open a PR and ping the person who maintains the integration for first review
False-alarm when constructing a pipeline with PEFT models
Author: B-Step62
Pull request: no

### System Info

Python 3.9.8
transformers == 4.37.2
peft==0.8.2

### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Creating a pipeline object with PEFT model like this.
```
import transformers
from peft import LoraConfig, TaskType, get_peft_model

base_model_id = "Elron/bleurt-tiny-512"
base_model = transformers.AutoModelForSequenceClassification.from_pretrained(base_model_id)
tokenizer = transformers.AutoTokenizer.from_pretrained(base_model_id)

peft_config = LoraConfig(
    task_type=TaskType.SEQ_CLS, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1
)
peft_model = get_peft_model(base_model, peft_config)

pipeline = transformers.pipeline(task="text-classification", model=peft_model, tokenizer=tokenizer)
```

The last line triggers an error message:
```
The model 'PeftModelForSequenceClassification' is not supported for text-classification. Supported models are....
```

The error is raised from [here](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/base.py#L1041-L1045) but is a false alarm, because the pipeline just works fine (also for other PEFT models/pipelines I've tested). 

A similar issue was raised in PEFT repository (https://github.com/huggingface/peft/issues/218) but got auto-closed. There was one workaround suggested by the community, that is creating a pipeline with base model first and mutate `pipeline.model` to the PEFT model, but it is hacky and bypasses any other checks. 

### Expected behavior

Supposedly PEFT models don't change model input/output interface, so as long as the base model is the one supported by the pipeline, users should not see the error. Even if this assumption is not always correct, the message should be a soft warning like `PEFT model might not work correctly with pipeline xxx` rather than being raised by `logger.error`, otherwise it is misleading for many models that work fine.

Comments:
cc @Rocketknight1 ! 

@B-step62 I've opened a PR to fix this at #29517. Can you test that the problem is now fixed? You can install from the PR branch with `pip install --upgrade git+https://github.com/huggingface/transformers.git@fix_peft_model_in_pipelines`

As the PR was close, a potential solution was given [here](https://github.com/huggingface/transformers/pull/29517#issuecomment-1990988026). TLDR this is not planned solution:
> first load the peft adapter with AutoModelxxx interface then pass that model to the pipeline or pass a valid path to a Peft model!
Unable to trace DebertaForMaskedLM
Author: TXacs
Pull request: no
Labels: name: Feature request

### System Info

- `transformers` version: 4.37.1
- Platform: Linux-5.4.0-47-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.3.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

from config import load_config

from transformers import (
    DebertaForMaskedLM,
    DebertaConfig,
)
from transformers.utils.fx import symbolic_trace

def main():
    config_kwarg = load_config("deberta-large")
    config = DebertaConfig(
            **config_kwarg
        )

    # create model
    model = DebertaForMaskedLM(config)

    traced_model = symbolic_trace(model)

if __name__ == "__main__":
    main()

### Expected behavior

When I run the scirpt, get following error.
![image](https://github.com/huggingface/transformers/assets/60869411/2755e3d2-6733-40a6-bd58-2c6a9f960154)


Comments:
Hey, I don't think this model supports tracing! I'll mark this as a feature request. It's pretty much a duplicate of #20815

> Hey, I don't think this model supports tracing! I'll mark this as a feature request

Thanks apply. I see the _REGULAR_SUPPORTED_MODEL_NAMES_AND_TASKS has deberta, so I test it.
How to get input embeddings from PatchTST with (batch_size, sequence_length, hidden_size) dimensions
Author: nikhilajoshy
Pull request: no
Labels: name: Feature request

### System Info

-

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The following snippet outputs the last hidden state but it has (batch_size, num_channels, num_patches, d_model) dimensions
`inputs = encoder(
            past_values=series_list, output_hidden_states=True
        ).last_hidden_state`

Here, series_list has (batch_size, sequence_length, num_input_channels) shape.

To incorporate this with [EncoderDecoderModel](https://huggingface.co/docs/transformers/v4.37.2/en/model_doc/encoder-decoder#transformers.EncoderDecoderModel), I want the dimensions of the input embedding to be (batch_size, sequence_length, hidden_size). How do you get that?


### Expected behavior

-

Comments:
sorry for the delayed answer cc @NielsRogge 🤗 

Hi,

PatchTST is not supported to work out-of-the-box with the EncoderDecoderModel class, which supports text-only models. However, you could fork the Transformers library and make the necessary changes to use PatchTST as encoder. To get encoder hidden states of shape (batch_size, seq_len, hidden_size), you could reshape them, like so (cc @kashif):

```python
last_hidden_state = encoder( past_values=series_list, output_hidden_states=True ).last_hidden_state

batch_size, num_channels, num_patches, d_model = last_hidden_state.shape
last_hidden_state = last_hidden_state.permute(0, 2, 1, 3)
last_hidden_state = last_hidden_state.view(batch_size, num_patches, d_model*num_patches)
```

@nikhilajoshy, @NielsRogge is correct, the patch-tst tokenization does not lend itself to an enc-dec setting where the decoder is typically (causally masked), since the patching operation causes vectors with different times... 

but if you insist on using a decoder, then the hidden_size is the patch_size and so you need to just configure the decoder with the same config as the default encoder
Fix test failure on DeepSpeed
Author: muellerzr
Pull request: yes

# What does this PR do?

It looks like `grad_norm` might not always be a `torch.Tensor`, as that merged PR breaks all of our deepspeed trainer tests. In such cases, we can check for `item` before calling it. Follow-up to https://github.com/huggingface/transformers/pull/29212, linked to https://github.com/huggingface/transformers/pull/27326

Fixes failing DeepSpeed tests


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @pacman100 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29444). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Fixes the issue mentioned in #29207 , but I haven't checked whether #29212 fixed it or not for me.

A tensor, actually

When will update to pypi?

this one break training on deepspeed now.

This was released in https://github.com/huggingface/transformers/releases/tag/v4.39.0
[`Add Deci`] Llama with variable GQA per layer 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Add support for Deci. `# Ignore copy` makes it a lot easier @ydshieh 🪂 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27978). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Glad to know it doesn't cause (many) extra issues 😆 

Not at all! It's really nice for reviews, we are certain that only this part is what we need to look

Picking this back soon 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Starcoder2 model
Author: jlamypoirier
Pull request: yes

The Starcoder2 model, adapted from Mistral. All changes are done through options, so Mistral itself is still supported. Main changes:
* Use layer norm (RMS still available as option)
* Use standard MLP (gated still available as option)
* Add back biases (optional)
* Change (default?) tokenizer class
*Embedding and residual dropout

It does not support absolute embeddings, so can't support Santacoder or Starcoder

Todo:
* Forward changes from #27931, #29027 (and future changes from Feb. 19)
* Documentation
* Copyright
* Point to starcoder2 checkpoint
* Other minor things (see todos)

@younesbelkada 

Comments:
I re-created a PR here since Joel is on vacation: https://github.com/huggingface/transformers/pull/29215
Sorry for the inconvenience

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as #29215 was merged and starcoder 2 is officially supported
Automatic safetensors conversion when lacking these files
Author: LysandreJik
Pull request: yes

When a user calls the PyTorch `from_pretrained` on a repository that only contains PyTorch/Flax/TF files, start an auto conversion in the background so that it has a PR opened with `safetensors` files.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29390). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks both for the review!

I'll merge it now and will keep monitoring issues to ensure it doesn't break things in the wild.
Fix the behavior of collecting 'num_input_tokens_seen'
Author: youliangh
Pull request: yes

# What does this PR do?

The length of "inputs[main_input_name]" is not guaranteed to be the same when using DDP, which may make the training process hang. Besides, in a distributed setup, it costs a lot to gather the WHOLE input tensors on different workers. It is better to call .numel() first and then .gather().

Fixes https://github.com/huggingface/transformers/issues/28791

The modified code has already passed the relevant test ``tests/trainer/test_trainer_distributed.py`` with additional argument 'include_num_input_tokens_seen'.


## Who can review?

@pacman100 
@muellerzr


Comments:
Just got it rebased from the main. However something still went wrong in the test, and it seems irrevelant to my PR.
Could you help identify?
Much appreciate @muellerzr  

Done, and it has passed all the tests.
@muellerzr @ArthurZucker 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29099). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks for this fix