noamaanMulla-03/transformers-issues huggingface dataset:
HQQ
Author: blap
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.44.2
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.11
- Huggingface_hub version: 0.24.7
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.6.0.dev20240915+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce MX330


And: 
python.exe -m pip install --upgrade torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124
python.exe -m pip install --upgrade transformers
python.exe -m pip install --upgrade git+https://github.com/mobiusml/hqq.git
python.exe -m pip install --upgrade huggingface_hub


### Who can help?

@SunMarc

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

# from https://huggingface.co/docs/transformers/v4.44.2/quantization/hqq
```

from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig

# Method 1: all linear layers will use the same quantization config
quant_config  = HqqConfig(nbits=8, group_size=64, quant_zero=False, quant_scale=False, axis=0) #axis=0 is used by default

model = transformers.AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct", 
    torch_dtype=torch.float16, 
    device_map="cuda", 
    quantization_config=quant_config
)

```
Error:

Traceback (most recent call last):
  File "C:\Users\Admin\Desktop\hqq\hqq2.py", line 4, in <module>
    quant_config  = HqqConfig(nbits=8, group_size=64, quant_zero=False, quant_scale=False, axis=0) #axis=0 is used by default
  File "C:\Users\Admin\Desktop\hqq\venv\lib\site-packages\transformers\utils\quantization_config.py", line 228, in __init__
    from hqq.core.quantize import BaseQuantizeConfig as HQQBaseQuantizeConfig
  File "C:\Users\Admin\Desktop\hqq\hqq.py", line 3, in <module>
    from hqq.models.hf.base import AutoHQQHFModel
ModuleNotFoundError: No module named 'hqq.models'; 'hqq' is not a package



### Expected behavior

Quantized model.

Comments:
This may be due to your file being named hqq that is in the path `C:\Users\Admin\Desktop\hqq\hqq.py` . I'd suggest naming that something else and seeing if that resolves your issue. 

> This may be due to your file being named hqq that is in the path `C:\Users\Admin\Desktop\hqq\hqq.py` . I'd suggest naming that something else and seeing if that resolves your issue.

Solved. Thank you!
Fix: Qwen2-VL training on video datasets
Author: hiyouga
Pull request: yes

# What does this PR do?

We should clone the leaf tensor before doing the in-place operation, otherwise it raises exception in training.

```
File "/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1589, in forward
    inputs_embeds[video_mask] = video_embeds
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
```

Similar to

https://github.com/huggingface/transformers/blob/9230d78e76611cfa38c845213021aeb185362d10/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L1578-L1584

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@ArthurZucker @zucchini-nlp @simonJJJ 

Comments:
Oh yeah, and this also prevents compile. Can we use `torch.masked_scatter` for consistency among VLMs and readability (w/o any if/else)

@zucchini-nlp We have updated the implementation, how about the new one?

@hiyouga sorry, what do you mean by "the new one"? 

@zucchini-nlp sorry, i say the latest commit in this PR 96286c3747924c4c576cfba4ff91985f439e4f55

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33307). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker Hi, I think this slow test case already covers this patch, it may require non-trivial effort to implement a test case for training I think.

https://github.com/huggingface/transformers/blob/e48e5f1f13e05380e24f4f31f5fee07aa6f959eb/tests/models/qwen2_vl/test_modeling_qwen2_vl.py#L324-L359

Imo the fullgraph compile tests should capture the test, but those are part of GenerationTester and Qwen2VL yet doesn't have it. For that we need generation tests with image + text + (optional video) inputs, will add those soon. Tracker here https://github.com/huggingface/transformers/issues/33374

Okay all thanks ! Let's merge 🤗 We'll improve the ecosytem for video and related tests on the fly! 🚀
Fix parametrization-based weight norm
Author: ylacombe
Pull request: yes

# What does this PR do?

Supersedes #32194 and fixes #31970 and #26796!

While #32194 was already a great work, it wasn't compatible with versions of Torch that only had `nn.utils.weight_norm`.

I'll left a review to explain some choices and to highlight where I'm not quite sure of my solution!

cc @LysandreJik and @ArthurZucker !

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33275). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
warning about weight_g/weight_v missing on WeightNorm on PyTorch
Author: kamilakesbi
Pull request: yes

This PR addresses issue #26796: 

- On models using WeightNorm (such as Hubert or Wav2vec2), a warning message appears indicating that some weights have not been properly initialised (`weight_v` and `weight_g` parameters).

This is because Pytorch migrates the `weight_v` and `weight_g` params of WeightNorm to `original0` and `original1`.

The state dict is converted correctly thanks to PR #24030, but we still get a warning message when loading the model. 

We can solve this by replacing `weight_g` and `weight_v` keys with corresponding updated keys in `_load_pretrained_model` as done in this PR. 

# Who can review: 

cc @sanchit-gandhi 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32194). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sanchit-gandhi there are indeed a few models for which the new parametrizations format is missing: 

- Univnet
- sew_d
- sew
- Seamless_m4t
- Seamless_m4t v2
- fast_speech2_conformer
- Encodec

I've udpated the corresponding modeling_files. 

The problem may arise with future model integrations! I'm thinking of the recent [dac](https://github.com/huggingface/transformers/pull/31494/files) for example. 

We should probably add a warning suggesting to users to implement the new parameterisation format with weight_norm when adding a new model! 


cc @ylacombe 

Superseded by #33275
no_speech_probablity
Author: rizwanishaq
Pull request: no
Labels: name: Audio

              The `pipeline` is designed to be a high-level wrapper that goes from audio inputs -> text outputs. Anytime we want something more granular than that, it's best to use the `model` + `processor` API:
```python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from datasets import load_dataset
import torch

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")

librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = librispeech_dummy[0]["audio"]

input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt").input_features

outputs = model.generate(
    input_features, output_scores=True, return_dict_in_generate=True, max_new_tokens=128
)

transition_scores = model.compute_transition_scores(
    outputs.sequences, outputs.scores, normalize_logits=True
)

pred_text = processor.batch_decode(outputs.sequences, skip_special_tokens=True)
pred_language = processor.batch_decode(outputs.sequences[:, 1:2], skip_special_tokens=False)
lang_prob = torch.exp(transition_scores[:, 0])

print(pred_text)
print(pred_language)
print(lang_prob)
```
**Print Output:**
```
[' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.']
['
']
tensor([1.])
```

_Originally posted by @sanchit-gandhi in https://github.com/huggingface/transformers/issues/25138#issuecomment-1836554598_


How we can get the no_speech_probablity with this code?            

Comments:
cc @sanchit-gandhi @ylacombe @kamilakesbi 

Hey @rizwanishaq, you can simply add the[ `no_speech_threshold` argument](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperForConditionalGeneration.generate.no_speech_threshold) to the generate method:


```python
outputs = model.generate(
    input_features, output_scores=True, return_dict_in_generate=True, max_new_tokens=128, no_speech_threshold=0.2
)
```

Let me know if that works!


Hey @ylacombe I get this warning "Audio input consists of only 3000. Short-form transcription is activated.no_speech_threshold is set to 0.3, but will be ignored." 

Then it's related to a shortcoming of our Whisper implementation that we hope to fix soon: [we're not applying some of the features used for long-form generation to short audios](https://github.com/huggingface/transformers/issues/29508#issuecomment-2029435220).

We should be resolving this issue quite soon, I'll keep you up-to-date

This is a duplicate of #29595, I'll close this issue, let's talk in #29595 if you have further questions!

For reference, it was fixed by https://github.com/huggingface/transformers/issues/29508!
Also see [this comment ](https://github.com/huggingface/transformers/issues/29595#issuecomment-2354923459)
Enable finetuning with torchao quantized model 
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR enable training with torchao quantized model. @BenjaminBossan conducted a few experiements with torchao + peft and it works out of the box for int8 quantizer. More details [here](https://github.com/huggingface/transformers/pull/32306#issuecomment-2334296963). 

Comments:
Should the method also check if PEFT is installed? I guess it's not strictly necessary, so this is also fine, just wondering. Also, it would be nice to get confirmation if others also run into problems with int4 or if it's just me.

> Should the method also check if PEFT is installed? I guess it's not strictly necessary, so this is also fine, just wondering. Also, it would be nice to get confirmation if others also run into problems with int4 or if it's just me.

If one uses Trainer to train the quantized model, it will trigger an error in the trainer class if the user doesn't use peft. So I guess we don't need to add a warning there.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33361). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Catch ConfigError when attempting to set num_parameters in WANDB
Author: gheinrich
Pull request: yes

# What does this PR do?

The current version of the code catches AttributeError when attempting to set the model's number of parameters in the Weights & Biases config.

ConfigError is another exception that needs catching in the case when the current number of model parameters differs from that of the config. This happens, for example, on resuming from checkpoints if extra layers have been added.

Example error message:

```
wandb.sdk.lib.config_util.ConfigError: Attempted to change value of key "model/num_parameters" from 0 to 700416
```

Since the current code already includes support for ignoring AttributeError, it feels safe to add this extra catch.

I did not find a test suite for the WANDB integration.


Comments:
Hello, cc @parambharat since you wrote the code that catches the first exception. Thank you!
Update chameleon.md — fix runtime type error
Author: maxwbuckley
Pull request: yes

# What does this PR do?
Fix runtime type error when following the Chameleon documentation

```
RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same
```


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

Comments:
Generate: assistant should be greedy in assisted decoding
Author: gante
Pull request: yes

# What does this PR do?

Restores a previous behavior -- forces the assistant in assisted decoding to do greedy decoding.

While providing [this answer](https://github.com/huggingface/transformers/issues/30608#issuecomment-2107585243), I noticed that we dropped a property in the refactor that was present in the original implementation: the assistant should be greedy.

Why? We want the assistant to maximize the number of matches with the original model, to result in the maximum speedup. Remember: the main model always calls the shots, the output text is never changed regardless of the flags we set for the assistant. By forcing the assistant to be greedy, we pick its most likely tokens. If the assistant model has a distribution aligned with the main model, these tokens will also be the main model's most likely tokens, increasing the number of matches.

When running my original benchmarks (https://github.com/gante/huggingface-demos/tree/main/experiments/faster_generation), this resulted in a ~20% speedup @ temperature=0.5 over the current version of the code.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30778). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts 

> Will this result in different outputs for the users? 

Nope :) The only difference is execution time. (The output is entirely defined by the main model and its generation config, both untouched in this PR)

> Question for the future: would we ever want to enable a sampling strategy for the assistant?

Perhaps. A potential strategy for improvement is to sample several continuations and try them all as candidates. But that would require significant changes over the current code 👀 

> > Will this result in different outputs for the users?

> Nope 

@amyeroberts my answer did not age well (#32867 -- some randomness was lost as a result of this change)
chore: migrate coverage cfg to pyproject.toml
Author: SauravMaheshkar
Pull request: yes

# What does this PR do?

Migrate coverage configuration to `pyproject.toml`

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request)


Comments:
cc @ydshieh 

Hi @SauravMaheshkar Could you elaborate this a bit more? For example, where we can find the output coverage report file?

There should be no difference in where the output file is generated. It's just a simple change of where coverage is configured.

For the record:

https://coverage.readthedocs.io/en/latest/config.html

> Coverage.py will read from “pyproject.toml” if TOML support is available, either because you are running on Python 3.11 or later, or because you installed with the toml extra (pip install coverage[toml]
Create panoptic segmentation task guide
Author: NielsRogge
Pull request: no
Labels: name: Feature request

name: Vision

### Feature request

We currently already have a semantic segmentation task guide: https://huggingface.co/docs/transformers/tasks/semantic_segmentation. It would be great to create a similar one for panoptic segmentation with models like MaskFormer, Mask2Former and OneFormer.

### Motivation

Panoptic segmentation deserves its own task guide as it's different from semantic segmentation.

Lots of people have reported issues, including the following:

- [ ] computing mAP during MaskFormer training: https://github.com/NielsRogge/Transformers-Tutorials/issues/373
- [ ] evaluation of DETR and friends on a panoptic segmentation dataset: https://github.com/NielsRogge/Transformers-Tutorials/issues/320
- [ ] there was an effort to add the panoptic quality (PQ) metric to Evaluate: https://github.com/huggingface/evaluate/pull/408
- [ ] various people want to use a COCO-formatted dataset for fine-tuning, but there's no guide yet regarding how to do this: https://github.com/NielsRogge/Transformers-Tutorials/issues/296

### Your contribution

Can be based off the notebooks provided here: https://github.com/NielsRogge/Transformers-Tutorials/tree/master/MaskFormer

cc @qubvel 

Comments:
@NielsRogge Thanks for creating this issue! I believe if you add the `feature request` tag this will prevent the issue from being marked as stale. It's great to add the `vision` tag to these too to make them easier to filter and spot for those interested in the listed issues 

Hey @NielsRogge, I’d love to give it a try if that's fine with you.

Hi @A-Duss, great! 🤗 
You can mention me for a PR review or in case any help is needed!

My understanding is that examples of inference for semantic, instance and panoptic segmentation are already present in the docs (see [Types of segmentation](https://huggingface.co/docs/transformers/tasks/semantic_segmentation#types-of-segmentation)). However, a complete task guide (with data preprocessing, finetuning, evaluation and inference) only exists for semantic segmentation. 

Given the issues tagged, you'd want:
- Instance Segmentation evaluation code example (ideally with evaluate using mask mAP in coco format): https://github.com/NielsRogge/Transformers-Tutorials/issues/373
- Panoptic Segmentation evaluation code (ideally on coco): https://github.com/NielsRogge/Transformers-Tutorials/issues/320
- For this PR https://github.com/huggingface/evaluate/pull/408, I assume the idea for tagging it was to use the proposed panoptic quality metric in the example code. But since the PR still hasn't been merged yet I am not sure about what to do.
- Finally, pretty clear, make it with COCO: https://github.com/NielsRogge/Transformers-Tutorials/issues/296

Is my understanding correct ? @NielsRogge @qubvel 
Do not hesitate to tell me exactly how you were thinking it, or if I'm off-track.

Sounds good to me, you can also take a look at instance-segmentation examples in the repo regarding evaluation with `Trainer` and `torchmetrics`.
 - https://github.com/huggingface/transformers/pull/31084
Add support for GGUF Phi-3
Author: dabs9
Pull request: no
Labels: name: New model

name: Good First Issue

name: GGUF

### Model description

Model is Phi-3 saved as a GGUF.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

It would be the analog of this recent PR but for Phi 3: https://github.com/huggingface/transformers/commit/e4628434d854ddfb5c002a6cc00b4eb4f22b7df2#diff-55e3debce4586d986257d1b1acb48c8708504f4b3b63e413e89f27ac2cc738b3

Comments:
cc @SunMarc 

Hey @dabs9, thanks for showing interest in GGUF integration with Phi-3 model ! I will try to tackle this when I get the time but I highly encourage you to do the integration yourself if you have the time ! As you saw from the Qwen integration, this should be fairly straightforward. I will mark it as a first good issue in case a community member is interested. 

Hello @SunMarc @amyeroberts - if nobody started on this, i would like to contribute to this by adding the GGUF for phi3, is following Qwen integration as a guide good enough to get started?

Hi @a8nova, feel free to add the gguf path for phi3 ! Following Qwen integration should be good enough indeed ! I'll try to help if you need any help ! If the integration is harder than expected, we might write a guide on how to add gguf path. Let us know how it goes ! Qwen PR [here](https://github.com/huggingface/transformers/pull/31175). 

Perfect, I created an initial PR: https://github.com/huggingface/transformers/pull/31844. 

how can i contribute to this?
fix to jamba config, asserting attention and expert offset
Author: ErezSC42
Pull request: yes

# What does this PR do?

Ensures instanced models are supported by adding assertions to attention offset and expert offset values


Comments:
I think it would be more appropriate to move this to a separate (private) function and instead of an assertion error, raise a `ValueError`. 

Ah and maybe add a simple test to check if the expected errors are raised 👀 

Jamba, cc @ArthurZucker 

I changed t_ to a be more descriptive, but since the  _check_supported_offset function is called inside JambaConfig's __init__, it cannot be a method defined inside the class

Thank you! 

For `_check_supported_offset` it should be possible, see for reference:
https://github.com/huggingface/transformers/blob/f38590dade57c1f8cf8a67e9409dae8935f8c478/src/transformers/models/olmo/configuration_olmo.py#L150-L181
At least that's what I had in mind.

my bad, it should be working now

You can run `make style` which should fix the remaining issues. Thx for bearing with me :)

Hey, all the tests passed, is this PR good for merging?

It needs a review from a core maintainer, so we can just wait. 

Maybe cc @amyeroberts