noamaanMulla-03/transformers-issues huggingface dataset:
Fix copies between DETR and DETA
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes failing quality checks on main: 
https://app.circleci.com/pipelines/github/huggingface/transformers/84538/workflows/0d4691b0-4988-4040-a6bf-bd1ad90f523b/jobs/1093017?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29037). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FIX: Fix error with `logger.warning` + inline with recent refactor
Author: younesbelkada
Pull request: yes

# What does this PR do?

In fact current on transformers main some legacy setup that call `model._is_quantized_training_enabled` throw an error:

```bash
Arguments: (<class 'FutureWarning'>,)
--- Logging error ---
Traceback (most recent call last):
  File "/home/younes_huggingface_co/miniconda3/envs/fix-test/lib/python3.9/logging/__init__.py", line 1083, in emit
    msg = self.format(record)
  File "/home/younes_huggingface_co/miniconda3/envs/fix-test/lib/python3.9/logging/__init__.py", line 927, in format
    return fmt.format(record)
  File "/home/younes_huggingface_co/miniconda3/envs/fix-test/lib/python3.9/logging/__init__.py", line 663, in format
    record.message = record.getMessage()
  File "/home/younes_huggingface_co/miniconda3/envs/fix-test/lib/python3.9/logging/__init__.py", line 367, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
```

Because `logger.warning` does take a second positional argument

I also realised we should use `warnings.warn` to be in line with deprecation warning guildelines presented here: https://github.com/huggingface/transformers/pull/26527 🤯 

cc @amyeroberts 


Comments:
Thanks ! That's on me as well :D 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29039). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix a tiny typo in `generation/utils.py::GenerateEncoderDecoderOutput`'s docstring
Author: sadra-barikbin
Pull request: yes

Hi there!

To fix a tiny typo in `generation/utils.py::GenerateEncoderDecoderOutput`'s docstring

@gante 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29044). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts any idea why CI isn't running in this PR? 👀 

(this PR fixes a typo, it's ready to be merged)

@gante No :(  I can merge it as it's just a typo. cc @ydshieh for reference

Thanks for fixing @sadra-barikbin!
Patch to skip failing `test_save_load_low_cpu_mem_usage` tests
Author: amyeroberts
Pull request: yes

# What does this PR do?

A handful of tests started failing after the merging in of #28948. Tests didn't fail on PR or initial main commit, but now failing. Looks like might be relevant tests not fetched for the runners. 

This PR skips the tests for now.

cc @ylacombe As you might want to enable this feature for Music Gen

cc @ArthurZucker As this touches a few language models. Not sure it's worth digging in and fixing it for these as they have low-ish usage. 

cc @ydshieh For when you're back in case there's anything else I should address here 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29043). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for catching these, and sorry I missed some of them (I didn't know how to run the tests across all models at the time).

I've also worked to fill up all the failing tests:  https://github.com/huggingface/transformers/pull/29024

Do you mind taking a quick look? I'm hoping to get that in so it doesn't affect other people's workflows. Thanks!


> Thanks for catching these, and sorry I missed some of them (I didn't know how to run the tests across all models at the time).

Oh, no need to apologise, you shouldn't need to manually do it, there's something wrong on our end as they should have been run automatically. 

> Do you mind taking a quick look? I'm hoping to get that in so it doesn't affect other people's workflows. Thanks!

Looking now 🤗
Update all references to canonical models
Author: LysandreJik
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29001). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The failing tests are also failing on `main` and due to the static cache PR. The `tests_pr_documentation_tests` unfortunately cannot run as it exceeds 10 minutes
Update Mixtral modeling
Author: imoneoi
Pull request: yes

# What does this PR do?

The [Mixtral technical report](https://arxiv.org/pdf/2401.04088.pdf) was published recently, showing that Mixtral routing weights are calculated in the top-K before softmax order.

This PR updates the Mixtral model implementation accordingly.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @younesbelkada

Comments:
It worries me a little bit that Mistral has different implementations for this specific process in the model.

They submitted a PR to vLLM which is aligned with the current Huggingface implementation:

https://github.com/mistralai/vllm-release/commit/3e21dacb79471ebf946e72e67a5ca14ebcc598c1#diff-74473eb619768ed055303d29d74c47b1753615e8323e69467506b2b42d3f9898R341-R344

Yep, we tried to follow this instead of the paper as they shared that code. Thanks for submitting a pr anyways 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update important model list
Author: LysandreJik
Pull request: yes

Adds LLaMa to the important models to test on each commit

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29019). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I hesitated but Mistral and Llama are very close in terms of architecture and support so I didn't find it necessary; do you think it's different enough to warrant being added?

mistral uses sliding window attention which is not the case in llama, for that I thought maybe better to test it but I think all good to leave it as is as well
Fix - don't return pixel mask for yolos
Author: amyeroberts
Pull request: yes

# What does this PR do?

#28363 introduced a bug where the pixel mask was now being returned for YOLOS. `pixel_mask` isn't a valid YOLOS input, and so this breaks this. 

The PR fixes that. 

Weirdly, this wasn't caught on the original PR, but was triggered in #28312 

cc @ydshieh for reference - we can try and debug together why we're not catching all the necessary tests when you're back :) 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29048). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing as change was added in #28312
Report inconsistent output length from decoder-only model generate with input_ids and inputs_embeds
Author: Hannibal046
Pull request: no

### System Info

- `transformers` version: 4.35.0.dev0
- Platform: Linux-5.15.0-1050-azure-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@gante 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When using `max_length` in `generate`. The output length varies between `input_ids` and `inputs_embeds`

```python
from transformers import AutoTokenizer, LlamaForCausalLM

model = LlamaForCausalLM.from_pretrained("huggyllama/llama-7b",low_cpu_mem_usage=True).cuda()
tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")

prompt = "Hey, are you conscious? Can you talk to me?"
inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs.input_ids.cuda()
inputs_embeds = model.get_input_embeddings()(input_ids)

generation_kwargs = {
    # "max_new_tokens":20,
    "max_length":20,
}
generate_ids = model.generate(input_ids= input_ids, **generation_kwargs)
output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
print("generate with input_ids:",output)
print("**"*40)

generate_ids = model.generate(input_ids= input_ids, inputs_embeds = inputs_embeds, **generation_kwargs)
output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
print("generate with input_ids+input_embeds:",output)
print("**"*40)

generate_ids = model.generate(inputs_embeds = inputs_embeds, **generation_kwargs)
output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
print("generate with input_embeds:",output)
print("**"*40)
```

![image](https://github.com/huggingface/transformers/assets/38466901/5104a4f8-cd79-4350-bc12-0e8236a6fa31)


However, using `max_new_tokens`, it would generate identical results.

### Expected behavior

The output length should be the same whether the input to `generate` is `ids` or `embeds`.

Comments:
cc @zucchini-nlp 

Hey, @Hannibal046 . Thanks for pointing this out! I added a PR to fix this behavior. When it gets merged, you can install `transformers` from `main` to get the correct generation.
fix failing trainer ds tests
Author: pacman100
Pull request: yes

# What does this PR do?
1. After PR https://github.com/huggingface/transformers/pull/27568, when resuming from ckpt, it first loads the `trainer_state.json` file. As such, when bogus ckpt folder is passed it will throw file not found error. Earlier, the code would throw different invalid ckpt error in the function call `deepspeed_load_checkpoint`. As such, `test_can_resume_training_errors` tests were failing. This PR fixes the tests by removing the exact check on the error message when resuming from bogus ckpt.   

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29057). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Request for Flash Attention 2.0 Support in GPNRoFormerForMaskedLM
Author: YBoulaimen
Pull request: no
Labels: name: Feature request

name: Flash Attention

Hello,

I trust this message finds you well. I am currently attempting to run the GPN-MSA model, which utilizes AutoModelForMaskedLM, and I am keen on parallelizing the computation across multiple GPUs. To optimize the model's performance, I would like to request the integration of Flash Attention 2.0 support into GPNRoFormerForMaskedLM.

As I explore this avenue for parallelization, I envision that many others within the community could benefit from this enhancement.

Thank you for your time and consideration.

Best regards.

Comments:
Hi @YBoulaimen, thanks for opening this request! 

The model is defined and maintained under this repo: https://github.com/songlab-cal/gpn/blob/main/gpn/model.py

I suggest opening a request there.
Transformers trainer: All checkpoint restarts now FAILING
Author: whr778
Pull request: no

### System Info

 @muellerzr and @pacman100

In the trainer.py
Trainer code now requires trainer_state.json for checkpoint restarts
trainer.py does NOT save trainer_state.json in def _save_optimizer_and_scheduler(self, output_dir):
Recommend either removing the trainer_state.json dependency for checkpoint restarts 
or 
Adding at line 2497
```
       elif self.args.should_save:
            # deepspeed.save_checkpoint above saves model/optim/sched
            torch.save(self.optimizer.state_dict(), os.path.join(output_dir, OPTIMIZER_NAME))
            self.state.save_to_json(os.path.join(output_dir, TRAINER_STATE_NAME)) ## <= HERE
```

```
  File "/devel/venv/whr778/py3.11-st.11/lib/python3.11/site-packages/transformers/trainer.py", line 1513, in train
    state = TrainerState.load_from_json(os.path.join(resume_from_checkpoint, TRAINER_STATE_NAME))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

### Who can help?

 @muellerzr and @pacman100

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Trainer a Language Model
kill it 
try to do a checkpoint restart
Ran in a debugger all appropriate flags are set... file does not exist in the checkpoint as there is no code to save the file
Checkpoint restart requires trainer_state.json

### Expected behavior

Checkpoint restart should work as they used to

Comments:
Forgot the version info... sorry about that guys
transformers             4.37.2

Hello,

`trainer_state.json` is saved when calling `_save_checkpoint` at the given line below which is after the model, optimizer and schedulers are saved.

https://github.com/huggingface/transformers/blob/b2628086565e0eedf33f238fe2146f11087c0301/src/transformers/trainer.py#L2496

So, it should be saved properly and allow for reloading from the checkpoint as usual. You can see the following resume tests pass for 9 hours ago: https://github.com/huggingface/transformers/actions/runs/7925195377/job/21638061721

![Screenshot 2024-02-16 at 6 08 31 PM](https://github.com/huggingface/transformers/assets/13534540/ec2b1744-ca46-46bf-9a3b-e9a0e646d872)


Just confirmed... my bad had two training arguments stepping on each other... ran the debugger through TrainerControl to verify.   This can be closed.
Option to set the tracking URI for MLflowCallback.
Author: seanswyi
Pull request: no
Labels: name: Feature request

### Feature request

Option to set not only the experiment name but also the tracking URI for MLflow.

### Motivation

My company and I use our own MLflow URI and all of our code has `mlflow.set_tracking_uri($URI)` inside. I'm not seeing such an option for the MLflowCallback and am only seeing an option to set the experiment.

The `run_name` seems to be using the `run_name` value in TrainingArguments, so there's not really any problem with that.

### Your contribution

It seems like adding a few lines of code to the `setup` method would do the trick.

https://github.com/huggingface/transformers/blob/345b9b1a6a308a1fa6559251eb33ead2211240ac/src/transformers/integrations/integration_utils.py#L952

Comments:
Hi @seanswyi, thanks for opening this feature request! 

The integrations are maintained by third party contributors, rather than the transformers team. If you or anyone else in the community would like to open a PR to add this we'd be happy to review! 

Thanks for the heads up @amyeroberts! I'll submit a PR when I can.
Fix trainer test wrt DeepSpeed + auto_find_bs
Author: muellerzr
Pull request: yes

# What does this PR do?

Follow-up to https://github.com/huggingface/transformers/pull/29057, changes the test to ensure it raises a not-implemented-error


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Will verify on our CI before merging (as local deepspeed won't install for me).

Can confirm passes: https://github.com/huggingface/accelerate/actions/runs/7932048373

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29061). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @amyeroberts good for review now that I verified tests pass :)
Add chat support to text generation pipeline
Author: Rocketknight1
Pull request: yes

This PR modifies the text generation pipeline to support chats. It does this by inspecting the inputs - if they look like strings, it uses the original causal LM pipeline, and if they look like lists of message dicts, it applies a chat template instead before proceeding with generation.

Most changes are in the preprocessing/postprocessing - the actual generation itself is largely unchanged.

TODO:
- [x] Expand tests to cover other edge cases
- [x] Confirm the return format we want for this - just the model response, or the entire chat?
- [x] ~Add KV cache support, as this is important for performant multi-turn chat~
- [x] Deprecate `ConversationalPipeline` and update the chat template docs to refer to this instead?

cc @ArthurZucker @gante @LysandreJik 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28945). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(and yes we should remove the old `ConversationalPipeline` sooner rather than later given it already doesn't work anymore due to `conversational` pipeline-type being removed from the Hub, IIUC)

@julien-c Done! This PR now adds a `DeprecationWarning` to `ConversationalPipeline`. I also updated the chat template docs for the new pipeline.

very nice!

One question for people, maybe @gante: Are you okay with the return format I'm using? Right now, if you pass a chat like this:

```python
[ 
    {"role": "system", "content": "This is a system message."},
    {"role": "user", "content": "This is a test"},
]
```

You get a response that's the same chat, continued:

```python
[
    {"role": "system", "content": "This is a system message."},
    {"role": "user", "content": "This is a test"},
    {"role": "assistant", "content": "This is a reply"},
]
```

I think this is the right thing to do, because it matches the behaviour of the existing `text-generation` pipeline (it returns the prompt at the start of the generated string). Let me know if you have a different opinion, though!

IMO it looks good to me

Cool!

In that case, I think we're ready for final review (cc @amyeroberts)  - I'm leaving the KV cache to another PR.

cc @LysandreJik @julien-c as well if there's anything else you want me to add before we merge this!
Correct wav2vec2-bert inputs_to_logits_ratio
Author: ylacombe
Pull request: yes

# What does this PR do?

`chunk_length_s` parameter doesn’t yet work for w2v2-bert, when using the ASR pipeline.


```python
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="ylacombe/wav2vec2-bert-CV16-en-libri-cv", device=0)
pipe("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac", chunk_length_s=20.0, batch_size=16)
```

Turns out the `inputs_to_logits_ratio` wasn't working properly. I'm not totally sure that I've chosen the right input ratio so I'd like @Narsil's opinion on this.

The original W2V2 uses a [learned feature extractor](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/models/wav2vec2_conformer/modeling_wav2vec2_conformer.py#L486) to downsample the raw audio waveform to a more reasonable dimension, using a series of convolution layer.
However, w2v2-bert does that feature reduction thanks to a mel-spectrogram transformation;

If I understood `inputs_to_logits_ratio`, it only computes the hidden size after this feature reduction, which in the case of w2v2-bert doesn't exist. So I've put `inputs_to_logits_ratio=1`. I could very be wrong on this, WDYT @Narsil ?

cc @sanchit-gandhi 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28821). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

inputs_to_logit_ratio is:

-> Inputs (raw audio) -> [32000; u32] (4s audio at 8kHz for instance)
-> Logits [2000; u32] (Same 4s audio, just the convolutions has a reduction factor of 16 for instance, I don't remember the actual w2v2 ratio).


Usually mel spectrogram also has a downsampling factor, but I don't remember on the top of my head the name of the controlling factor. Mel Spectro is basically some convolutions on the raw audio too. They just happen to be FFT.


Thanks @Narsil, the ratio is wrong then, I'll correct it

Thanks for the review! All slow tests for the ASR pipeline pass! BTW, I'll make a subsequent PR to correct the fact that the long transcription script for s2s models (beside Whisper) is never called

Very good refactoring 😉