noamaanMulla-03/transformers-issues huggingface dataset:
Rename artifact name `prev_ci_results` to `ci_results`
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently, when a (daily CI) workflow run finished, we save some prepared results to a directory named `prev_ci_results` and upload it as an artifact.

This is not a good name (sorry) as it is actually the current CI result. To avoid further confusion (well, I am confused by my own naming), this PR just rename it to `ci_results`.

This is also helpful as we will start to upload more results (like results of other job, like pipeline job, deepspeed job etc.).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30697). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Feature Request] add layer-wise optimizers
Author: peterjc123
Pull request: no
Labels: name: Feature request

name: optimization

### Feature request

Context: https://github.com/huggingface/transformers/pull/29588#discussion_r1523510004

### Motivation

The layer-wise optimizers is not GaLore-specific. We could apply it to generic optimizers to save memory. For example, the 8bit Adam optimizer paired with the layer-wise optimization sounds like a pretty good option for me.

### Your contribution

Have a try when it is supported

Comments:
cc @younesbelkada 

Just putting myself out here--I'd be happy to hear requests/requirements for design/support from the PyTorch side!

Hi! 
Thanks so much @janeyx99 ! 🙏 
Currently our approach is to use `post_accumulate_gradient_hook` approach: https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1300-L1309 by using dummy optimizers that return no-ops during `step()` (idea borrowed from @hiyouga), per my understanding this do not support well some training schemes such as distributed training, we would probably need some help on PT if possible to support that (cc also @jiaweizzhao as we've been discussing this as well)

@younesbelkada Yes, it'll be interesting to discuss the current known pain points/requirements. Some questions that have already appeared are around DDP (which saves around buckets of gradients to accumulate once all the data's been processed) and gradient accumulation (which saves the gradients across iterations of fwd-bwd until an optim step). In both these cases, layer-wise optimizers will not save memory. Though a side note for GaLore: since the gradients should be smaller, the buffers holding the previously fullsized grads should be enabled to be smaller too.

Beyond those two instances, we should be able to theoretically compose this technology with FSDP/all other use cases where the gradients are allowed to be freed right after the optimizer update. I'm curious where people have run into problems with:
- checkpointing (saving + loading)
- general composition with existing trainers + configs.

@janeyx99 @peterjc123  I guess this is similar to what Badam is doing: #30308
Construct a Marian tokenizer. Based on huggingface tokenizers
Author: RRaphaell
Pull request: no

I have a Hugging Face tokenizer with three files: tokenizer.json, tokenizer_config.json, and vocab.txt. However, according to the documentation, the Marian tokenizer requires files in the SentencePiece format (.model and .vocab files). Is there a way to construct a Marian tokenizer without these specific file formats?

I have already tried converting the tokenizer into a .model file and then constructing the Marian tokenizer, but it raised new issues. Are there any alternative approaches or workarounds to use the existing tokenizer files with the Marian tokenizer?

Comments:
Hi @RRaphaell, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

got it, closing the issue
Pin deepspeed
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR pins the deepspeed version to the same one as `accelerate`

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30701). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix layerwise GaLore optimizer hard to converge with warmup scheduler 
Author: hiyouga
Pull request: yes

# What does this PR do?

Fixes #30371 , a detailed explanation has been provided in the issue.

After this fix, the loss can converge similarly as the non-warmup scheduler

<img src="https://github.com/huggingface/transformers/assets/16256802/9d6b890a-eaf6-49bb-a311-28cdfd92bed7" alt="loss" width="400" />

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@muellerzr and @younesbelkada 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30372). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
CLIP Training Example Bug - Overfitting
Author: humanely
Pull request: no

### System Info

- `transformers` version: 4.40.1
- Platform: Linux-5.15.0-1053-gcp-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Accelerate config: 	- compute_environment: LOCAL_MACHINE
	- distributed_type: MULTI_GPU
	- mixed_precision: fp16
	- use_cpu: False
	- debug: False
	- num_processes: 4
	- machine_rank: 0
	- num_machines: 1
	- gpu_ids: all
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): 2.13.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help?

@patil-suraj I am training a CLIP for sanskrit. Although, it keeps diverging. I have tried changing weight-decay 0.1 to 0.4. Also tried adding early stopping:
`
early_stop = EarlyStoppingCallback(3)
`
 With standard params(lr 5e-5, weight_decay 0.1, no max_grad_norm)
![image](https://github.com/huggingface/transformers/assets/6862868/b2e59472-9792-4450-951e-91e04a65fefc)

2nd Run: Change Weight decay to 0.2, max_grad_norm 0.9)
![image](https://github.com/huggingface/transformers/assets/6862868/db5209bb-0fc6-47f5-a48e-70371ca14703)

I tried numerous other trials, and everything diverges.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I initialized the clip with this script: https://gist.github.com/humanely/b73cb9e53c3879c3bc50890f6318cb7b
The tokenizer built using this: https://gist.github.com/humanely/fdf2f46d37cc9de8d84e1579006a7828



### Expected behavior

Expected a normal convergence of eval_loss for CLIP to under 0.1.

Other discussions: 
[https://github.com/huggingface/diffusers/issues/7836](https://github.com/huggingface/diffusers/issues/7836)
[https://github.com/huggingface/diffusers/discussions/7835](https://github.com/huggingface/diffusers/discussions/7835)


Comments:
Hi @humanely, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.
top-k instead of top-p in MixtralConfig docstring
Author: sorgfresser
Pull request: yes

# What does this PR do?
The current docstring for the `num_experts_per_tok` is quite misleading - mixtral uses `TopK` to formulate their experts per token in their paper. While I think top-p would be interesting to see, the experts are currently routed using top-k. This PR addresses this.
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30687). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[pipeline] allow fp16/bf16 inputs on cpu
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #29475: rather than forcing the inputs to float32 when on cpu, we leave them in their respective dtypes.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29486). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Interpolate pos encode for altclip
Author: bhuvanmdev
Pull request: yes

# What does this PR do?

Adds **interpolate_pos_encode** and **a test case** for the said feature for the model altclip.

Fixes #30579 (altclip)

## Who can review?

@amyeroberts 

Comments:
This is my first open-source contribution, so any type of suggestions are appreciated :)

Hi @bhuvanmdev, thanks for working on this! 

I must apologies for the making the issue clear when I first created it. Models which are linked through `# Copied from` statements should be modified together i.e. CLIP should changed and then realted models like altclip included in this change. 

As there's other PRs open to add this feature to CLIP, those will take precedence over this PR. Apologies for the time that you've already spent adding it to this model. 

I did get to understand the architecture behind CLIP like models a bit better :), so it was'nt a waste of time. 
Then in that case, should I just close this PR?  

> Then in that case, should I just close this PR?

Yes, let's close this PR. Thanks again for working on this and contributing to improving the library
[BitsandBytes] Verify if GPU is available
Author: NielsRogge
Pull request: yes

# What does this PR do?

When installing Accelerate and BitsAndBytes in a CPU environment (e.g. Google Colab), one gets the following error:

```
ImportError: Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes: `pip install -i https://pypi.org/simple/ bitsandbytes`
```
This is confusing for users as they have installed the packages. The true error is that no GPU is available, causing `is_bitsandbytes_available` to return `False`.

This PR fixes that by first checking whether a GPU is available.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30533). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Llava: remove dummy labels
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

I think we should remove dummy initialized labels in Llava, as it can be a bit misleading from user's perspective to get a loss when no labels are passed

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30706). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add installation of examples requirements in CI
Author: qubvel
Pull request: yes

# What does this PR do?

Fixes failed examples tests

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30708). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Improve/simplify object detection task guide
Author: NielsRogge
Pull request: no
Labels: name: Feature request

name: Good Difficult Issue

### Feature request

The object detection task guide has a bunch for issues so it'd be great to solve them.

### Motivation

Several people have reported the following issues regarding the object detection task guide:

- [ ] the model is not predicting any boxes at inference time when you follow the guide for training: https://discuss.huggingface.co/t/example-detr-object-detectors-not-predicting-after-fine-tuning/41824. This may be explained by the slow convergence of DETR. The guide uses 10 epochs, but that may not be enough (cc @MKhalusova who wrote the guide might have some more insights). There's a [model on the hub](devonho/detr-resnet-50_finetuned_cppe5") with relatively good metrics which is trained for 100 epochs, so we might need to update this.
- [ ] unnecessary complicated evaluation: https://discord.com/channels/879548962464493619/897391062975385640/1207009510171091025
- [ ] people need to overwrite the Trainer in order to leverage `compute_metrics`: https://discuss.huggingface.co/t/possible-fix-for-trainer-evaluation-with-object-detection/72307

Moreover, when inspecting the notebook, I saw:

- [x] there's no GPU leveraged during evaluation
- [x] the timm dependency is no longer required as you can now pass `revision="no_timm"`. Moreover, Accelerate is a dependency but is not included in the guide
- [x] the `post_process` method needs to be updated to `post_process_object_detection`, passing a threshold of 0 for computing metrics

### Your contribution

I could work on this but would be great if someone else could take this up

Comments:
Hi @NielsRogge , do you still need help on this?

@rootonchair yes, the first 3 points still need to be addressed. We would need to simplify the evaluation of the model during training

Sure, let me work on this

Also related: https://github.com/huggingface/transformers/issues/28893

![image](https://github.com/huggingface/transformers/assets/17937555/c9e3f9f2-3be5-4894-912d-bc2a164aba3a)
Here is a training run that I did a while back using the [original repo](https://github.com/facebookresearch/detr), 10 epochs is roughly around 200k steps in this figure and I doubt the model would give any meaningful predictions at that point. I haven't ran the training through HF's example yet. Will try it to understand the problem better. The problem is just that this model is awfully slow to train even with A100s...

Closing as this is now fixed by https://github.com/huggingface/transformers/pull/30683
Refusal rejection removal as a feature
Author: KnutJaegersberg
Pull request: no
Labels: name: Feature request

### Feature request

It seems to be possible to add or remove 'intentions' or directions in trains of 'thought' in LLMs via a relatively easy method. They suggest both an on the fly measure and to simply change the weights. It's not perfect, but likely a useful tool. 
You could for example try to get rid of certain stereotypes by that method or make a model permanently 'smarter', i.e. reasoning more like an expert. 

### Motivation



[Refusal in LLMs is mediated by a single direction](https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction)

### Your contribution

Hope you have a nice day

Comments:
Cache: models return input cache type
Author: gante
Pull request: yes

# What does this PR do?

Changes the `Cache` compatible models to return the same type of cache as the cache passed through `past_key_values`. This was not happening in the case a `DynamicCache` was explicitly passed as input, which was breaking a test :)

Comments:
@zucchini-nlp this was the issue you pointed me to yesterday 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30716). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> I think we deprecated the legacy cache so at some point we can just return whatever was given!

@ArthurZucker `generate` still defaults to the legacy format -- I'm going to rectify this 😉
questions on checkpoint and 'training_args.bin' in run_lm_finetuning.py
Author: molsheim
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
2 questions:

1. there is a **checkpoint** save logical, but don't see any logical to load this checkpoint. nothe load method in code
1. there is '**training_args.bin**' has been store with checkpoint together. no load code to support it. 

could you please tell how to use these checkpoint and 'training_args.bin' to **continue** training?
thanks.



Comments:
The model saved can be loaded by using the `model.from_pretrained(directory)` method. The training arguments are saved so that they can be re-used later. You can load them using the `torch.load(directory/training_args.bin)` method.

thanks for reply. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> The model saved can be loaded by using the `model.from_pretrained(directory)` method. The training arguments are saved so that they can be re-used later. You can load them using the `torch.load(directory/training_args.bin)` method.

if i'm using a pretraining model (TrOCR) , what are the necessary and enought file to use it in production ?, the folder have like 6 files 8