noamaanMulla-03/transformers-issues huggingface dataset:
Update README.md
Author: NinoRisteski
Pull request: yes

broken link on documentation tests, now fixed.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25851). All of your documentation changes will be reflected on that endpoint.

Hi @NinoRisteski Sorry we forgot to merge this PR, and it ends up being done in another PR #26213.

We really appreciate your contribution and I want to apologize for what happens here 🙏 .
Support gradient checkpointing for ESM models
Author: mahdip72
Pull request: no
Labels: name: Good Second Issue

name: Feature request

Would you please add `gradient_checkpointing_enable()` feature for ESM models?
These models currently are the best available pre-trained protein language models for researchers.
Many thanks.

Comments:
cc @Rocketknight1 

Any updates?

It's on the to-do list, but I'm afraid there are competing priorities at the moment!

Let's open it up for anyone in the community who might want to tackle it :) 

Hi @amyeroberts @Rocketknight1 I would like to work on this 

@sanjeevk-os Great! Once you have the code ready, open a PR and ping both @Rocketknight1 and me. Looking forward to reviewing! 

Hi @sanjeevk-os, I actually took a look at the ESM code - it actually looks like some of the supports for gradient checkpointing are already there, in which case you just need to make a one-line change to set `supports_gradient_checkpointing = True`

Hi @Rocketknight1 Thank you for taking a look. I also noticed that the ESM model has the _create_custom_forward_ passed to torch checkpoint function. I will do some more checks and will raise a PR soon.

Hi @sanjeevk-os - we're getting even more requests for this, so we'd like to try to add it soon! If you're having trouble, just let us know. We can take over the PR internally to try to get it through, and we appreciate your effort regardless.

This issue has now been resolved - thank you to @sanjeevk-os for the very clean PR!
How to freeze a few layers in t5 model  during fine-tuning
Author: Aniruddha-JU
Pull request: no


Comments:
In PyTorch, this is very easy, like so:

```
from transformers import T5ForConditionalGeneration

model = T5ForConditionalGeneration.from_pretrained("t5-base")
for name, param in model.named_parameters():
     if name == "...":
        param.requires_grad = False
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Can you tell me in which file I add this code? or how to use this code you mentioned @NielsRogge
Multi-objective HP Optimization Not Possible
Author: marcusinthesky
Pull request: no

https://github.com/huggingface/transformers/blob/41aef33758ae166291d72bc381477f2db84159cf/src/transformers/integrations.py#L208C10-L208C10


Because the current implementation always passes in a direction, an error is thrown when a user tries to pass in Optuna's `directions` argument for multi-objective optimization. To solve this issue, either:
1. HuggingFace's `direction` argument must allow a list to be passes, or
2. HuggingFace's default direction must not be passes to Optuna if a `directions` kwargs is passed. 

Comments:
cc @sywangyi 😉 

could we set direction value to None if kwargs contains directions to solve the issue? see https://github.com/optuna/optuna/blob/7b3824aaab38843fdfbfc67ea93d6da446dc1548/optuna/study/study.py#L1125
https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2514
if direction is default value, and directions is passed through kwargs. directions will be used.

Feel free to open a PR and ping me ! 🤗 

@ArthurZucker do we decide to support multi-objectives HPO in transformers? I check the code, not only the input to optuna create_study need to be addressed, but also output in https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/integration_utils.py#L211 should be changed as well. Currently for single objective, only one best trial will be returned. However, for multi-objectives, the best trails in pareto front need to be all returned, something like the list of BestRun. Also the value in BestRun should be switched to the values to indicate the values in multi-directions.

You can do pretty much whatever you want there, since you are the author, if you think this is a good addition I'll trust you 😉 

I am not the author, I just add HPO DDP feature, I think author is from https://github.com/huggingface/transformers/pull/6576

Oups, but anyway as a main contributor, feel free to add it if you feel like it! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as #25969 fixed it.
[InternLM] Add support for InternLM
Author: Rocketknight1
Pull request: yes

InternLM is based on the LLaMA code but adds a `config.bias` parameter. We can support those models by adding `config.bias` to LLaMA, and preserve backward compatibility by defaulting it to `False`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ArthurZucker Also, to answer the question, there is no `Dropout` anywhere in the LLaMA code or the InternLM code.
Registering Models in MLflow Callback
Author: Tolga-Karahan
Pull request: no
Labels: name: Feature request

### Feature request

We can add registering models functionality to MLflow callback so that we can use MLflow model registry with 🤗 models. To do that we can introduce an optional _registered_model_name field to the callback, and can register the model in case these field is not None.


```
class MLflowCallback(TrainerCallback):
    #...
    def setup(self, args, state, model):
        #...
        self._log_artifacts = os.getenv("HF_MLFLOW_LOG_ARTIFACTS", "FALSE").upper() in ENV_VARS_TRUE_VALUES
        self._registered_model_name = os.getenv("HF_REGISTERED_MODEL_NAME") # Suggested Change
        #...
        
    def on_save(self, args, state, control, **kwargs):
        if self._initialized and state.is_world_process_zero and self._log_artifacts:
          ckpt_dir = f"checkpoint-{state.global_step}"
          artifact_path = os.path.join(args.output_dir, ckpt_dir)
          logger.info(f"Logging checkpoint artifacts in {ckpt_dir}. This may take time.")
          self._ml_flow.pyfunc.log_model(
              ckpt_dir,
              artifacts={"model_path": artifact_path},
              python_model=self._ml_flow.pyfunc.PythonModel(),
              registered_model_name=self._registered_model_name or None  # Suggested Change
          )
```




### Motivation

Model registry is one of the most useful features of MLflow, but current callback doesn't support it. It forces users to make a custom implementation to use this functionality. Instead, we can extend 🤗 MLflow callback to provide this feature.

### Your contribution

I can implement this extension and create a PR.

Comments:
fyi @pacman100 and @muellerzr 

A PR would be most welcome! Feel free to open one and try :)
Add torch `RMSProp` optimizer
Author: natolambert
Pull request: yes

# What does this PR do?

Add torch `RMSProp` for easy use in TRL library, particularly to match the default use case of Direct Preference Optimization.
Script is [here](https://github.com/huggingface/trl/blob/main/examples/dpo.py).

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@younesbelkada and I discussed this.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks, I can't merge it as I don't have write access, so @ArthurZucker or @younesbelkada can do so!
save_pretrained 4-bit models with bitsandbytes
Author: westn
Pull request: no

With the latest version of bitsandbytes (0.39.0) library, isn't it possible to serialize 4-bit models then?

Thus this section should be updated to allow the user to save these models.
https://github.com/huggingface/transformers/blob/68d53bc7178866821282f45732c1e465f5160fa6/src/transformers/modeling_utils.py#LL1704C36-L1704C36

I'm mainly looking at this article to see what got produced lately in the area: https://huggingface.co/blog/4bit-transformers-bitsandbytes

I aim to quantize a model to 4-bit, enabling it to be used in e.g., GPT4all and other CPU platforms.

Comments:
cc @younesbelkada 

Hi @westn 
Thanks for the issue, I don't think 4bit models are serializable yet. Let me double check that with the author of bitsandbytes and get back to you

Also note that 4bit / 8bit is also applicable to GPU / CUDA devices, you cannot run quantized models with bitsandbytes on a CPU device

Hi @westn 
Currently it is not possible to save 4bit models but this is in the roadmap of bitsandbytes for the next releases. We will keep you posted!

Hi, @younesbelkada , Whether 4bits/8bits models can be saved now?

Hi @jameswu2014 
Thanks for the heads up, currently 4bit saving is not possible, however, 8bit saving is possible, check: https://huggingface.co/docs/transformers/main_classes/quantization#push-quantized-models-on-the-hub

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

See linked bnb issue: https://github.com/TimDettmers/bitsandbytes/issues/695

Bumping this since this is a top Google result for this topic, and I haven't found an answer elsewhere - is there any way to "de-quantize" a model which was trained in 4-bit or 8-bit (without a peft adapter) so it can be saved and loaded in bfloat16 or some other type? (This is currently a blocker to training checkpointing, which also means things like early stopping are not possible).

I tried basic things like `model.to(torch.blfoat16)` but it throws an error stating that bitsandbytes models can't be cast with `to`.

Is there any update on this? @nathan-az @younesbelkada 

hi @kbulutozler 
Yes, @poedator is working on it in https://github.com/huggingface/transformers/pull/26037 and the corresponding bnb PR is here: https://github.com/TimDettmers/bitsandbytes/pull/753
IDEFICS Processor will not respect `padding_side="right"` when `padding="longest"`
Author: xhluca
Pull request: no

### System Info

- `transformers` version: 4.33.2
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.34
- Python version: 3.8.16
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@VictorSanh 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics-9b", padding_side="right")

sents = [["hello world"], [" this is a longer sentence to testing padding"]]

# This is the correct behavior:
a = processor(sents, padding="max_length", truncation=True, max_length=20)
print(processor.tokenizer.decode(a['input_ids'][0]))
# => 
 hello world<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>

# This is the incorrect behavior:
b = processor(sents, padding="longest", truncation=True, max_length=30)

print(processor.tokenizer.decode(b['input_ids'][0]))
# => <unk><unk><unk><unk><unk><unk>
 hello world
```

### Expected behavior

It should pad on right side

Comments:
@VictorSanh @ArthurZucker Not sure, but maybe this is because idefics processor forces the tokenizer to take the prompts one by one and for `padding="longest"` to work it has to take all the prompts at once.

https://github.com/huggingface/transformers/blob/main/src/transformers/models/idefics/processing_idefics.py#L285-L333

*a potential fix would be to get the text encoding part of it out of the loop and some changes to prompt preprocessing, it should be a fairly quick fix (if there are no border conditions following this)*


This indeed looks like a bug cc @leot13 @ArthurZucker 

This is definitely a bug. with a quick dive into the code, i think what @shauray8  makes sense.
It's mostly about moving things around: unindenting the block 324-330, let the tokenizer handle the padding logic (block 346-352).
(i might be wrong, just having a quick look)
If you feel like it, feel free to open a PR @shauray8 / @xhluca 

On it!
[`FA` / `tests`] Add use_cache tests for FA models
Author: younesbelkada
Pull request: yes

# What does this PR do?

While testing out https://github.com/huggingface/transformers/pull/26414 I realised the current tests silently pass as we only check for the immediate next predicted token  
For small models FA2 is quite flaky so I have decided to not increase `max_new_tokens` in `test_flash_attn_2_generate_padding_right` and `test_flash_attn_2_generate_left_padding` and rather have a separate test that will run `generate` with `use_cache` with a relatively large `max_new_tokens` to catch issues with caching when porting models to FA

cc @LysandreJik 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
testing doc-builder
Author: mishig25
Pull request: yes

testing https://github.com/huggingface/doc-builder/pull/418


Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26445). All of your documentation changes will be reflected on that endpoint.
[`PEFT`] Fix PEFT multi adapters support
Author: younesbelkada
Pull request: yes

# What does this PR do?

With https://github.com/huggingface/peft/pull/905 being merged, it added the support for multi-adapter inference (combining multiple adapters at inference). 

The PR has introduced some changes that are not compatible anymore with peft integration of transformers, for example `active_adapter` is now a property method and returns a list. This PR adds the corresponding fixes to make everything compatible with the newest features of PEFT, while preserving backward compatiblity

cc @BenjaminBossan @pacman100 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for all the reviews, I think I have addressed most of them now, I propose to handle the multi-adapter saving in a follow up PR #26411 to be consistent with PEFT API

Thanks ! For that we could maybe sync the next PEFT release before / at the same time than the transformers release

As discussed offline I would propose the following:

- 1.) We either deprecate or fully remove `active_adapter(...)` and replace it with `active_adapters(...)` which **always** returns a list. `active_adapter(...)` function has not been in a release yet (right @younesbelkada ? ), so I would deprecate it for two minor versions and then remove it. It's cleaner to always return a list here.
- 2.) Instead of "magically" only saving the first adapter when having multiple active adapters, let's teach the user the difference between multi and single active adapters here by raising a ValueError when trying to save multi adapters. It's really not yet necessary to be able to save multiple adapters at once

I am ok to deprecate the `active_adapter` method ! The plan sounds great 

> active_adapter(...) function has not been in a release yet 

I meant that `active_adapters` was not released yet :D In any case I think it is ok here 

Agreed also on your second point! let's keep it simple



Agree with Patrick!

If we choose to go with
> 2.) Instead of "magically" only saving the first adapter when having multiple active adapters, let's teach the user the difference between multi and single active adapters here by raising a ValueError when trying to save multi adapters. It's really not yet necessary to be able to save multiple adapters at once

let's maybe put this PR https://github.com/huggingface/transformers/pull/26411 to the side for now and focus on merging this PR with single-adapter saving first? WDYT @younesbelkada ?

Sounds great @LysandreJik !

I should have addressed the final comments @patrickvonplaten , @LysandreJik , thanks for your time reviewing the PR !

Failing test seems unrelated - merging !
Fix failing doctest
Author: LysandreJik
Pull request: yes

Fixes a failing doctest by putting the file in the `not_doctested.txt` file for now.

Comments:
It's the same failure as other LLaMa-like models:

```
NameError: name 'PATH_TO_CONVERTED_WEIGHTS' is not defined
/home/circleci/transformers/src/transformers/models/mistral/modeling_mistral.py:806: UnexpectedException


FAILED src/transformers/models/mistral/modeling_mistral.py::transformers.models.mistral.modeling_mistral.MistralForCausalLM.forward
```


I'm adding the `modeling_mistral` file as well to make sure.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26450). All of your documentation changes will be reflected on that endpoint.
Update `runs-on` in workflow files
Author: ydshieh
Pull request: yes

# What does this PR do?

I also updated the labels on runners to have `t4` and `daily-ci`, `push-ci` etc.

Comments:
Also is there a difference between the `past-ci` and `daily-ci`? Or is it just that you'd like to have runners dedicated to these tasks?

> Also is there a difference between the `past-ci` and `daily-ci`? Or is it just that you'd like to have runners dedicated to these tasks?

It's a typo for `push-ci` no?

I see runners with the following tags on nvidia's side: `push-ci`, `daily-ci`, `past-ci`, `doctest-ci`

_The documentation is not available anymore as the PR was closed or merged._

> Also is there a difference between the `past-ci` and `daily-ci`? Or is it just that you'd like to have runners dedicated to these tasks?

So far, each CI workflow is running in a dedicated runner by using tags/labels like `single-gpu-docker` (daily CI has it) or `docker-gpu` (push CI has it). However I find this is difficult to maintain (they are all single gpu and docker but with differnt kind of label names to mean it).

Using explicit `daily-ci`, `push-ci`, `past-ci` etc.  makes things clear and easy to understand IMO. But the machines are all of the same.

Talking to @glegendre01 we decide to keep these extra labels (`push-ci` etc.) at this moment, and will remove them just before we switch to AWS with the auto-scaling runners setting.

Yes, i'm not ready to deploy new runners on transformers now (focusing on 2 others topics). But will be able to handle it next week. at this moment we will remove extra labels
Added Image Processor for ProPainter
Author: mahimairaja
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Added Image Processor Class for ProPainter


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)  - https://github.com/huggingface/transformers/issues/26360


- [ ] Did you make sure to update the documentation with your changes? 
- [ ] Did you write any new necessary tests?


## Who can review?

@rafaelpadilla @LysandreJik 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments: