noamaanMulla-03/transformers-issues huggingface dataset:
Fix mixtral ONNX Exporter Issue.
Author: AdamLouly
Pull request: yes

This PR addresses issue [#29857 ](https://github.com/huggingface/transformers/issues/29857) 

by removing the list conversion of top_x and idx in the Mixtral model code, which currently obstructs the ONNX exporter's functionality. 

By removing this conversion, the Mixtral model can be exported successfully using the ONNX exporter, resolving the compatibility issue.



Comments:
@thiagocrepaldi FYI.

Hey @ArthurZucker  any updates on this?

@AdamLouly 

I think what @ArthurZucker 's comment is about

> would you mind just running some benchmarks in generation?

on your own side to see if the PR and `main` have difference in the speed of generation for this model

On point @ydshieh . This is very core, and we added this change for a reason, we need a good reason to update, and make sure we are not affecting performances 

@ArthurZucker, I actually found it faster when we use the changes in the PR.

Environment specs:
V100
Mixtral-8x7B-v0.1
2 Hidden layers
Loaded using torch_dtype=torch.float16

With the PR changes:
{'train_runtime': 182.1736, 'train_samples_per_second': 8.783, 'train_steps_per_second': 0.274, 'train_loss': 1.3467385864257813, 'epoch': 5.71}                                 


Without PR changes:
{'train_runtime': 200.6888, 'train_samples_per_second': 7.973, 'train_steps_per_second': 0.249, 'train_loss': 1.3390922546386719, 'epoch': 5.71}                                 

Thank you



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29858). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I could not find anything like that on qwen models, are you sure it uses the same thing?

This here! : https://github.com/huggingface/transformers/blob/8e51901872894c3bf9666daf485bb4c267980cad/src/transformers/models/qwen2_moe/modeling_qwen2_moe.py#L847

qwen2 sorry
Missing preprocessor_config.json file after training segformer model 
Author: evanwong1020
Pull request: no

### System Info

- `transformers` version: 4.37.2
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.11.5
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1) Follow the Image Classification example in the HuggingFace docs: https://huggingface.co/docs/transformers/tasks/semantic_segmentation

2) Upon reaching the code segment

![image](https://github.com/huggingface/transformers/assets/76148061/115dc7e9-3ed2-48ef-9dd0-67e5b90d8118)

You should end up with an error that says
`OSError: segformer-b0-scene-parse-150 does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co/segformer-b0-scene-parse-150/None' for available files.`

### Expected behavior

I'd assume a preprocessor_config.json file should be generated in the training process.

Comments:
Hi, thanks for your interest in SegFormer.

This is because when training the model, one could also provide the image processor to the Trainer class, so that it gets saved along the model weights to the Trainer's `output_dir`. At the moment, one needs to do `Trainer(..., tokenizer=image_processor)` when instantiating the trainer, but we'll update this as the Trainer was initially designed only for text-only models.

The pipeline expects a path to a folder containing both the modeling files (model weights, a config.json) and the image preprocessor file (preprocessor_config.json).

When passing the tokenizer=image_processor parameter into the trainer, I get an attribute error:

`AttributeError: 'SegformerImageProcessor' object has no attribute 'pad'`

Ok thanks for reporting. Noticed that currently, when you pass `tokenizer=image_processor`, then the `DataCollatorWithPadding` is used as seen [here](https://github.com/huggingface/transformers/blob/f01e1609bf4dba146d1347c1368c8c49df8636f6/src/transformers/trainer.py#L284-L287). Hence a current workaround is to do the following:

```python
from transformers import default_data_collator

trainer = Trainer(...
            tokenizer=image_processor,
            data_collator=default_data_collator
)
```
However, this should be fixed:

* the "tokenizer" argument would need to extended, or the "image_processor" argument would need to be added to the Trainer (to allow for passing something different besides a tokenizer)
* the default data collator must be used in case an image processor is passed.
[bnb] Fix offload test
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR fixes a test in BNB that started failing because of this [PR](https://github.com/huggingface/transformers/pull/29958). This test was impacted by the mentioned PR because we were replacing the layers incorrectly in the first place, hence the output changed after the PR was merged. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30039). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
"addmm_impl_cpu_" not implemented for 'Half'
Author: ch1ld3r1c0
Pull request: no

### System Info

- `transformers` version: 4.38.2
- Platform: Linux-5.14.0-284.11.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.9.14
- Huggingface_hub version: 0.21.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

from spacy_llm.util import assemble
nlp = assemble("spacy_few_shots.cfg")

text="""
example tex
"""

doc = nlp(text)
for e in doc.ents:
    print(e.label_, e.text, e.start, e.end)


_______________________________
spacy_few_shots.cfg
[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = label(example)

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "spacy_few_shots.yml"

[components.llm.model]
@llm_models = "spacy.OpenLLaMA.v1"
name = "open_llama_3b"

### Expected behavior

Model should generate output.

Comments:
The issue you are facing right now is not from the hugging face/transformers team and that's happening cause you are using `model = model.half()` in CPU, and matmul op for CPU is not implemented in PyTorch

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as @erfanzar's answer resolves this issue
Update quantizer_bnb_4bit.py: In the ValueError string there should be "....you need to set `llm_int8_enable_fp32_cpu_offload=True`...." instead of "`load_in_8bit_fp32_cpu_offload=True`".
Author: miRx923
Pull request: yes

# What does this PR do?
There is an mistake in ValueError on line 86 of quantizer_bnb_4bit.py. In the error string there should be "....you need to set `llm_int8_enable_fp32_cpu_offload=True`...." instead of "`load_in_8bit_fp32_cpu_offload=True`". I think you updated the BitsAndBytesConfig() arguments, but forgot to change the ValueError in quantizer_bnb_4bit.py.

I also changed it in quantizer_bnb_8bit.py.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
I also fixed the 8 bit file.
[test fetcher] Always include the directly related test files
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently, if a module has many impacted test files, those test files are filtered by `IMPORTANT_MODELS`.
The file `models/whisper/modeling_whisper.py` is in this case, but `whisper` is not in `IMPORTANT_MODELS`.
So if `modeling_whisper.py` is modified, `test_modeling_whisper.py` is not included in the testing.

Such situations makes the CI miss the tests that we 100% need to test.

This PR changes the logic to always include the directly related test files.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30050). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Refactor daily CI workflow
Author: ydshieh
Pull request: yes

# What does this PR do?

This is a re-work of  #29197. The main difference is: in this new work, we 
  - keep the definition of all jobs in a single workflow file (`self-scheduled.yml`)
    - this is changed to be a **re-usable** workflow
  - create a new workflow file (`self-scheduled-caller.yml`) as the caller to the above one (`self-scheduled.yml`)
    - it pass some inputs to the `self-scheduled.yml`

The reasons of the re-work:
  - **avoid too many workflow files** for a single CI. (As we also have `push-ci`, `amd-ci` x 2, `nightly-ci`, `past-ci` etc)

Once approved and merged, the same refactorization will be applied to other workflow files.

I have made sure it runs successfully as expected: https://github.com/huggingface/transformers/actions/runs/8524250043
(please don't look at the diagram - look the left side instead)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30012). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Make final commit(s) to clean up the channels stuff - all channels are created and used in the new workflow file.

Will merge once CircleCI  is done
[WIP] Add musecoco
Author: Sonata165
Pull request: yes
Labels: name: New model

# What does this PR do?
Add the attribute-to-music generation model in [MuseCoco](https://arxiv.org/abs/2306.00110). The model undergoes large-scale pre-training on 1M multitrack MIDI dataset, can be useful for various tasks by fine-tuning.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Models:

- text models: @ArthurZucker and @younesbelkada



Comments:
FYI @ylacombe 
Thanks @Sonata165 for the pR! Could you link the original github, and the link to the checkpoints? 
Also for new models we usually recommend to start by publishing them on the hub following [this](https://huggingface.co/docs/transformers/en/custom_models) tutorial 

Here is the original github repo: https://github.com/microsoft/muzic/tree/main/musecoco
And here is the checkpoint of the 200M model in the paper: https://drive.google.com/file/d/16ETj2kRavkYf4NAAcjiLQxq9xuzqhS12/view?usp=sharing
Thanks for the tutorial link!
Add docstrings and types for MambaCache
Author: koayon
Pull request: yes

# What does this PR do?

Adds some typing to the MambaCache and some docstrings for the IDE.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Just a small developer Quality of Life one, would love it if you could take a look @ArthurZucker and @younesbelkada! 


Comments:
A test is currently failing referencing the `generate_whisper` file which as far as I can tell has nothing to do with this PR? 

<img width="1120" alt="image" src="https://github.com/huggingface/transformers/assets/58336584/6269cecd-32c8-4538-bc81-17f66701db65">


Could you run `make fixup` for the remaining test? 


feel free to also rebase on main 

Yeah good idea, have rebased! 

@ArthurZucker the tests are now passing and it's good to merge - thanks! 🙏 

Please be a tad bit patient 🤗
Exporting Mixtral using ONNX Exporter Fails Due to Python List Conversion
Author: AdamLouly
Pull request: no

### System Info

nightly pytorch
nightly transformers
stable latest onnx

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When attempting to use the ONNX exporter to export the Mixtral model I faced an exporting error, and digging further, I found out that a tensor to list conversation that is made in the MixtralSparseMoeBlock is what raises this error.

in this line:
https://github.com/huggingface/transformers/blob/fadb053379b3ef24c4ec8e6d7d58555af21f58db/src/transformers/models/mixtral/modeling_mixtral.py#L869

`top_x_list = top_x.tolist()
idx_list = idx.tolist()
`

The top_x and idx are converted to a list which ONNX exporter and potentially other exporters cannot handle.

To solve it we need to remove this list converstion.

------------

Question to @ArthurZucker , why do we need this conversion, how much perf gain does it bring?

### Expected behavior

Failed in export.

Comments:
Adding to @AdamLouly's point, th `tolist()` is an issue for PyTorch 2.X code in general (aka Dynamo). In Dynamo, pytorch tensors/operators are tracked to build a static computational graph. With operations like `numpy.array`, `torch_tensor.numpy()` or, as in this case, `torch_tensor.tolist()`, a pytorch tensor becomes a python scalar, which cannot be properly tracked by Dynamo, causing all sorts of issues

One solution is to convert the model code to use as much PyTorch API as possible. In this case, it means keeping the tensor as a tensor, not a list.

By the way, why do we need a `.tolist()` here? The code seems to work the same way without it. That would help us understand the underlying design decision for this and improve the Pytorch 2.x support

It works for sure, but as the comment points out we found that it was faster to index with list rather than with tensor (that is from profiling). I'll review!
Fix auto tests
Author: ydshieh
Pull request: yes

# What does this PR do?

Some failures after #29112 due to the wrong Hub repo. being used

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30067). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker 

it exists, it's not the problem of the existing or not. It's we use

>        model_name = "google-bert/bert-base-uncased"
but the check after it 
>        self.assertIsInstance(config, GPT2Config)
claim it is `GPT2Config`. And it fails as the claim is false obviously

It's just #29112 updated `model_name ` while not update `GPT2Config` etc (or use the repo. corresponding to the original previous repo)
Missing information in tutorial supporting broad range of models
Author: kirk86
Pull request: no

### System Info

transformers version: 4.39.2
python 3.12
platform linux

### Who can help?

@stevhliu @MKhalusova 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The [example here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) states that among the supported architectures for summmarization are `Bart` and `Pegasus`.

A couple of clarifying questions.

1. ~~Does the data need to the have the same format as for `T5`, i.e. using format such as `prefix + actual text data`?~~ It seems the answer is no. At least that's what I get from the examples.
2. ~~The models such as `Bart` and `Pegasus` do not implement `ignore_index` in cross entropy to ignore padding `-100` that is being added automatically by the `DataCollatorForSeq2Seq` in contrary to the `T5` model which does.~~ Seems like when calling cross entropy loss from pytorch comes with default value of `ignore_index=-100` already predefined. I would prefer a more explicit behaviour to make sure the user understands what's going on in the code instead of second guessing default values which might change in the future

Are we sure that if we follow the example linked above and make a simple MWE setting using `ForSeq2Seq` trainer, args, models, collator, that we won't get any errors? I mean the model might end up training fine but it won't ignore the padding token in the labels which might impair learning? 

### Expected behavior

A bit more clear and concise context on what exact format preprocessing expects each model for each equivalent task.
And, why some models don't implement `ignore_index` while other do, since to my naive understanding if one uses `DataCollatorForSeq2Seq` then it will pad the labels automatically with `-100` if they are different seq len.

Am I missing something here?

Comments:
Fix whisper kwargs and generation config
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Whisper code was changed recently to use `prepare_generation_config` from the GenerationMixin [here](https://github.com/huggingface/transformers/blob/81642d2b51de9d5e5aee1768abdc744d90f7f52d/src/transformers/models/whisper/generation_whisper.py#L477), which started causing some bugs. See failing test in [this PR](https://github.com/huggingface/transformers/pull/30016).

Before, Whisper generation config did not do any `generation_config.update` and manually updated kwargs along the way. After the above change, kwargs cannot be used anymore because it contains unused kwargs from generation config.

This PR replaces everywhere we had `kwargs.pop()` with `generaion_config`. All tests are passing, except for some slow tests. But the slow tests are not passing in `main` also, so should be unrelated.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30018). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

likely test fetcher didn't find it ...

well, i see , whisper is not in `IMPORTANT_MODELS`

IMPORTANT_MODELS = [
    "auto",
    # Most downloaded models
    "bert",
    "clip",
    "t5",
    "xlm-roberta",
    "gpt2",
    "bart",
    "mpnet",
    "gpt-j",
    "wav2vec2",
    "deberta-v2",
    "layoutlm",
    "llama",
    "opt",
    "longformer",
    "vit",
    # Pipeline-specific model (to be sure each pipeline has one model in this list)
    "tapas",
    "vilt",
    "clap",
    "detr",
    "owlvit",
    "dpt",
    "videomae",
]

should we put it ... 😄 @sanchit-gandhi @ArthurZucker ?

I would be in favour of adding Whisper and possible Wav2Vec2 as well to this list as the two most used audio models @ydshieh 

@sanchit-gandhi `wav2vec2` is already in that list.

Done!

@zucchini-nlp Do you have the merge permission (i.e. click the `squash and merge` button above)?

yep, merged!