noamaanMulla-03/transformers-issues huggingface dataset:
Issue with finetuning Mixtral w/ deepspeed after new release
Author: sam-h-bean
Pull request: no

### System Info

transformers: latest
env: ray + deepspeed on k8s

There seems to be an issue with Mixtral on the latest transformers release that manifests like 

```
RuntimeError: Detected mismatch between collectives on ranks. Rank 4 is running collective:
CollectiveFingerPrint(SequenceNumber=724883, OpType=_ALLGATHER_BASE, TensorShape=[4194305], TensorDtypes=BFloat16, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))),
but Rank 0 is running collective:
CollectiveFingerPrint(SequenceNumber=724883, OpType=_ALLGATHER_BASE, TensorShape=[699051], TensorDtypes=BFloat16, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))).Collectives differ in the following aspects:
Tensor Tensor shapes: 4194305vs 699051
```

When I pin transformers to 4.36.2 the issue goes away.


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Finetune mixtral w/ deepspeed + accelerate my ds_config is like so
```json
{
    "fp16": {
        "enabled": false
    },
    "bf16": {
        "enabled": true
    },
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": false
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "gather_16bit_weights_on_model_save": true,
        "round_robin_gradients": true
    },
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "steps_per_print": 10,
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false,
    "comms_logger": {
        "enabled": true,
        "verbose": true,
        "prof_all": true,
        "debug": true
    }
}
```
2. 

### Expected behavior

The tensors are the correct shape

Comments:
Hi @sam-h-bean, thanks for raising an issue! 

So that we can best help you, can you make sure to follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and include all important information such as: 
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output. For example 'latest' isn't descriptive - it could mean dev branch or 4.37
* A minimal code snippet we can run to reproduce the error
* Full details of the error encountered, including full traceback

cc @ArthurZucker @pacman100 

Have same issue with latest transformers, how did you resolve the issue
Add JAX implementation of Phi 
Author: kanpuriyanawab
Pull request: no
Labels: name: New model

### Model description

Transformers already has Phi #26110 . I want to contribute the JAX/Flax implementation of the same.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
@shivance This issue is marked as completed but I can neither find the implementation in the [code tree](https://github.com/huggingface/transformers/tree/main/src/transformers/models/phi), nor load the model via `FlaxAutoModel`. Can you please point me to the implementation? 

@dfdx The Flax implementation of Phi hasn't been added yet. If you or anyone else in the community would like to contribute, we'd be happy to review a PR! 

FYI: I'm working on Flax implementation of Phi [here](https://github.com/dfdx/transformers/blob/dfdx/phi-2-flax/src/transformers/models/phi/modeling_flax_phi.py) (based on Flax Llama code). Attention is implemented, the rest should be pretty straightforward, so I expect a PR to be ready in a couple of weeks.
FIx Bark batching feature
Author: ylacombe
Pull request: yes

# What does this PR do?

This PR aims to fix batching with Bark:

> Currently, when several samples are transmitted to Bark at the same time, i.e. during batch generation, the generated audios all have the same duration, which extends the length of the shortest audios to the longest audio in the batch.
> This problem can be solved by keeping track of sample lengths as Bark.generate is called up, and then outputting the audio lengths with the generated audios at the end of the call.


This fix can be enabled with an additional bolean parameter (`return_output_lengths`, same naming than in #25943) so I've kept backward compatibility.
I also made sure that generated batched outputs were almost the same than without batching and listened to generated audio qualitatively!


cc @sanchit-gandhi and @amyeroberts, could you take a look ? Many thanks!

Fixes #25861 and #26673


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for your reviews here! merging! 

This was very helpful. Thanks!
Llava-demo-4bit.ipynb Error (KeyError: 'llava')
Author: RonanKMcGovern
Pull request: no

### System Info

See [this notebook from HuggingFace](https://colab.research.google.com/drive/1qsl6cd2c8gGtEW1xV5io7S8NHh-Cp1TV?usp=sharing#scrollTo=DFVZgElEQk3x)

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the notebook through to get this error:
```
KeyError                                  Traceback (most recent call last)
[<ipython-input-15-2c1efb550f3d>](https://localhost:8080/#) in <cell line: 6>()
      4 
      5 model_id = "llava-hf/llava-1.5-7b-hf"
----> 6 pipe = pipeline("image-to-text", model=model_id)
      7 url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"
      8 

2 frames
[/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py](https://localhost:8080/#) in __getitem__(self, key)
    759     [
    760         ("openai-gpt", "openai"),
--> 761         ("data2vec-audio", "data2vec"),
    762         ("data2vec-text", "data2vec"),
    763         ("data2vec-vision", "data2vec"),

KeyError: 'llava'
```

### Expected behavior

I would expect the notebook to run through the image + text example.

Comments:
You are probably running this on a local host, which does not have the latest version of `transformers` make sure to update with `pip install -U transformers`. You should have `import transformers;print(transformers.__version__)` >= transformers==4.37.2

Thanks @ArthurZucker , I was running the colab notebook in colab, which specifies that version.
```
!pip install -q -U transformers==4.37.2
```

I ran again today and it worked, so I'm not quite sure what the issue was and it seems there was no bug.
[`Doc`] update contribution guidelines
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes a misunderstanding from #28767 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28858). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix(tokenization): `encode` should remove leading batch axis for all types of single batch to keep consistent.
Author: scruel
Pull request: yes

# What does this PR do?
`encode` should remove leading batch axis for all types to keep consistent with `decode` method.

Fixes #28635


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker


Comments:
Hi @ArthurZucker, as we discussed in issue #28635, instead of let `encode` directly support multiple batches, this PR is main to fix the error of existing behaviour for a single batch. Let's call this as the first step for achieving your plan.

I may have few questions for the next step of your plan to deprecate some of the APIs to let `decode` supports both single and batch processing, along with some questions about this repo's code style, which place should I post them to ask? Or may I reach your team on your public discord server? Thanks.

We can keep the discussion on the related issue 🤗 

Ok, this PR can oly solve partial sections of the related issue and have some mistakes, so I closed it, will create another after I done.
WhisperForAudioClassification throws errors while using use_weighted_layer_sum
Author: chercheurkg
Pull request: no

### System Info

- `transformers` version: 4.36.0.dev0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.9.9
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1+cu118 (True)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sanchit-gandhi 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

For a classification task, I tried to fine-tune **whisper-small** pre-trained model using WhisperForAudioClassification and setting use_weighted_layer_sum equal to true. It threw the following error. 

```
  File "some_path\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "some_path\site-packages\transformers\models\whisper\modeling_whisper.py", line 2418, in forward
    hidden_states = torch.stack(encoder_outputs, dim=1)
TypeError: stack(): argument 'tensors' (position 1) must be tuple of Tensors, not BaseModelOutput
  0%|          | 0/2085 [00:52<?, ?it/s]
```

1. Using **whisper-small** pretrained model and setting use_weighted_layer_sum equal to true

```
        config = AutoConfig.from_pretrained(
                'openai/whisper-small',
                ..........
            )
       
        config.use_weighted_layer_sum = True
```
            

2. start training it using a label dataset

### Expected behavior

It should not throw the above error as it should work for both `use_weighted_layer_sum = True and use_weighted_layer_sum = False`

Comments:
Hi @chercheurkg, thanks for raising an issue!

This was fixed in #28563 and is part of the most recent release. Could you try installing v4.37 to verify this resolves your issue? 

`pip install -U transformers`

@amyeroberts ,
Thanks for your reply! After upgrading to v4.37, the issue seems to be resolved. 

Great to hear that @chercheurkg - closing as complete!
Image Feature Extraction pipeline
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds an ImageFeatureExtractor pipeline, which uses vision models for feature extraction. This is based off the `FeatureExtractionPipeline`, but is compatible with models which use image processors. 

```py
from transformers import pipeline

extractor = pipeline(model="google/vit-base-patch16-224", task="image-feature-extraction")
result = extractor("https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png", return_tensors=True)
```

This was added instead of adapting the current feature-extractor pipeline because of the conditional handling required for the tokenizers versus image processors


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28216). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

We will need to make some changes in 

https://github.com/amyeroberts/transformers/blob/d048cee154ddb36d4eaf0a072f5ffabd9c0bef2d/tests/models/align/test_modeling_align.py#L43

to make the test of `"image-feature-extraction"` works.

> Remark (just FYI): the attribute pipeline_model_mapping is generated via a script. And usually we don't modify them manually. It's fine here in this PR. But next time when I need to run the script again, it will add back "feature-extraction" anyway (despite the corresponding test will be skip).

@ydshieh OK, thanks for flagging. Is there anything else I need to do to make sure that `"image-feature-extraction"` will be automatically added / not deleted when running this script? 

> > Remark (just FYI): the attribute pipeline_model_mapping is generated via a script. And usually we don't modify them manually. It's fine here in this PR. But next time when I need to run the script again, it will add back "feature-extraction" anyway (despite the corresponding test will be skip).
> 
> @ydshieh OK, thanks for flagging. Is there anything else I need to do to make sure that `"image-feature-extraction"` will be automatically added / not deleted when running this script?

No, no action required from your side :-) in this PR


FYI @Rocketknight1
Unable to load models from HF Hub repos without main
Author: versipellis
Pull request: no

### System Info

transformers 4.37.2 on Python 3.11.6. 

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Issue lies with loading a model using `AutoModelForCausalLM.from_pretrained(...)`. If the HF Hub model resides in a branch and not in `main`, then it errors upon loading. E.g. for `someorg`'s fine-tuned Yi-34B model `somemodel` with nothing besides `.gitattributes` in branch `main` and with the `adapter_config.json` and `adapter_model.safetensors` files in `somebranch`, we get this error:

`OSError: someorg/somemodel does not appear to have a file named config.json. Checkout 'https://huggingface.co/someorg/somemodel/somebranch' for available files.`

I don't know if it's an incorrect error message, but I would've expected that link to be `https://huggingface.co/someorg/somemodel/tree/somebranch` to properly point at the branch.

If I copy over `adapter_config.json` into `main`, and re-run that command, I get this:

`OSError: somebranch is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at 'https://huggingface.co/01-ai/Yi-34B-Chat' for available revisions.`

I notice here that it points me to the original model's repo rather than my own org's fine-tuned model repo.

### Expected behavior

Expected model to load.

Comments:
Hi @versipellis 
You are trying to load adapter weights from a different branch than main, for that you need to pass the `revision` argument through `adapter_kwargs` as follows:
```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(model_id, adapter_kwargs={"revision": branch_name})
```

That explains why loading the model and setting the adapter separately works. Thanks a lot!
Support `logging_ratio`, `save_ratio`, and `eval_ratio` (like for `warmup_ratio`)
Author: konstantinjdobler
Pull request: no

### Feature request

I would love if `TrainingArguments` and the Huggingface `Trainer` would support `logging_ratio`, `save_ratio`, and `eval_ratio` arguments (complementing `logging_steps`, `save_steps`, and `eval_steps`). If the `*_ratio` argument is set to e.g. `0.1`, logging/saving/eval would be done every `0.1 * total_training_steps`. This is already done for `warmup_ratio` and `warmup_steps`.

### Motivation

When dealing with many different tasks and datasets, it can be frustrating to have to calculate different appropriate `logging_steps` etc. for each individual dataset. This proposal would enable a unified, simple and concise way to solve this problem.

### Your contribution

I realize this might not be trivial to fully integrate, but hopefully, we can take `warmup_steps` and `warmup_ratio` as a reference. Depending on how deep the required changes are, I can also submit a PR (with some pointers on what to look out for). 

Comments:
We already have 96 training arguments though, and that would make three more of all users to learn :-/

Another option would be to allow `float` inputs for `logging_steps`, `save_steps`, `eval_steps` and interpret all inputs `<1.0` as a ratio (and throw an error if inputs `>1.0` are not integers). 

But yes there is a tradeoff with too much complexity

I would prefer that solution actually, even if the naming is not perfect.

Going over the code a bit, it seems like we would have to wait until after this if statement in `_inner_training_loop` to set the correct values based on the `max_steps` that gets calculated there + some additional guards in the `__post_init__` of the `TrainingArguments`. 
https://github.com/huggingface/transformers/blob/04ab5605fbb4ef207b10bf2772d88c53fc242e83/src/transformers/trainer.py#L1683-L1711
At first glance, it looks like the setting of when things get logged/saved/evaluated in `DefaultFlowCallback` should work out-of-the-box with this change.
I'm willing to contribute the changes once I find some time, does the general plan sound reasonable?

Yes it does. I'll be looking forward to your PR!

Can someone clarify if this ratio means the % of training steps **per epoch** or the total training steps? If the latter, how do we know preemptively the total number of epochs (or total number of training steps) that the model is going to train for?
Do not use mtime for checkpoint rotation.
Author: xkszltl
Pull request: yes

mtime is not reliable across all filesystems.
Some fs do not have mtime support, may fake mtime, or use mount time as mtime.

Resolve https://github.com/huggingface/transformers/issues/26961

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28862). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Raise error when using `save_only_model` with `load_best_model_at_end` for DeepSpeed/FSDP
Author: pacman100
Pull request: yes

### What does this PR do?
1. `save_only_model` can't be used with DeepSpeed/FSDP along with `load_best_model_at_end`. This is because the models are wrapped in DeepSpeed Engine or FSDP units and require the ckpts in their respective formats which isn't the case when saving only the model as it is in transformers format. 
2. Fixes #https://github.com/huggingface/transformers/issues/27751 by explicitly raising an error when such config params are passed.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28866). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you! Addressed the comment in the latest commit.