noamaanMulla-03/transformers-issues huggingface dataset:
Docs request: installation for various hardware acceleration
Author: jamesbraza
Pull request: no

### Feature request

It would be nice to have a centralized and standardized page on how to `pip install transformers` with hardware acceleration for various systems (e.g. Metal, cuBLAS, OpenBLAS, etc).

Here are some examples:
- https://github.com/abetlen/llama-cpp-python/tree/v0.2.11#installation-with-hardware-acceleration
- https://github.com/ggerganov/llama.cpp/tree/b1329#cublas

Currently, what is mainly mentioned is installing `torch` for Metal, or `accelerate` package.  Would be nice if there was a centralized and enhanced documentation page on this.

### Motivation

Making use of hardware for additional inference speed, as well as saving on carbon emissions

### Your contribution

TBD

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @MKhalusova 

Thank you for the suggestion! I'll look into this. 

Hi @jamesbraza ! I looked into this topic. For Apple’s Metal, we already have documentation on how to [use Trainer for accelerated PyTorch Training on Mac](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#using-trainer-for-accelerated-pytorch-training-on-mac).

When it comes to other hardware acceleration libraries, there are no Transformers-specific setup steps. 
If you can use environment variables to make PyTorch use an alternative BLAS implementation, then this is the way to go. 
If you have to rebuild PyTorch from sources to link to `flexiblas` during PyTorch's compile time, for example, again, you can do this. The steps will be PyTorch-specific. 

With that in mind, it doesn’t make sense to include this in the Transformers docs as this is not Transformers-specific functionality. 

Some additional blog posts on CPU scaling that you might find interesting: 
* Part 1 (hardware part): [https://huggingface.co/blog/bert-cpu-scaling-part-1](https://huggingface.co/blog/bert-cpu-scaling-part-1)
* Part 2 (software part): [https://huggingface.co/blog/bert-cpu-scaling-part-2](https://huggingface.co/blog/bert-cpu-scaling-part-2)


Thanks for the response!  Appreciate you investigating.

It seems like your comments, mainly that optimization is done outside of `transformers`, are worth adding to the README somewhere or to a new page in `docs/`. If you feel it's not worthwhile, feel free to close this out.  Thanks again!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix documentation docstring 
Author: not-lain
Pull request: no

### System Info

the following the deocumentation in huggingface about [ConversationalPipeline](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.ConversationalPipeline.example) the example shown in here seems to be broken

![image](https://github.com/huggingface/transformers/assets/70411813/2959e117-790f-4fbf-9a3d-90988200fe0d)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

checkout the link above

### Expected behavior

a parsable example

Comments:
Update split string in doctest to reflect #28087
Author: amyeroberts
Pull request: yes

# What does this PR do?

Resolves current failing test `tests/utils/test_doc_samples.py::TestDocLists::test_sdpa_support_list`  on main because the string used to split the doc string wasn't updated in line with #28087 

cc @stevhliu 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28135). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update modeling_utils.py
Author: mzelling
Pull request: yes

In the docstring for PreTrainedModel.resize_token_embeddings, correct the definition of the new_num_tokens parameter to read "the new number of tokens" (meaning the new size of the vocab) rather than "the number of new tokens" (meaning the number of newly added tokens only). This is in agreement with what the code does (see source and docstring of function PreTrainedModel._get_resized_embeddings).

@stevhliu @MKhalusova

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
[docs] Fix mistral link in mixtral.md
Author: aaronjimv
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fix the mistral link in **`Mixtral`** docs page.

The link in this section generate a 404 error:

> The following implementation details are shared with Mistral AI’s first model [mistral](https://huggingface.co/docs/transformers/main/en/model_doc/~models/doc/mistral):

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->
@stevhliu

Comments:
Happy to help 🤗
Finetune ClipSeg model 
Author: sleeping4cat
Pull request: no
Labels: name: Good Second Issue

name: Vision

### Feature request

Quite recently, I was exploring zero-shot classification to segment medical images. And it looks quite promising. I stumbled upon ```ClipSeg``` a few days ago and it looked wonderful and just well-suited for my work. Unfortunately, I couldn't find any tutorials or notebooks that showed how to perform fine-tuning on ClipSeg model. 

I am assuming, we have to train the decoder with a dataset containing binary classification images of cells and their corresponding masks and a text description. Unfortunately, a bit confused. is there any tutorials/resources anyone could suggest on this topic? Cuz I couldn't none. 

### Motivation

```ClipSeg``` shows a lot of potential than SAM (Segment Anything Model). Unfortunately, there's no fine-tuning script neither instructions on **How to prepare the dataset?** which is very frustrating. Will love some help from the community. 

And another point, Zero shot classification looks a way lot better option with fine-tuning than training a model like ```U-Net```, ```R-CNN``` and others from scratch while you have very few images and don't have much room to play around with.

### Your contribution

I could provide a PR on my LinkedIn, where I have a lot of AI experts as my connections and then I contribute in the programming as well. 

Comments:
cc @alaradirik for information.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@sgugger Can you provide some insights/help/update to my requested feature?

cc @amyeroberts and @rafaelpadilla
move code to Trainer.evaluate to enable use of that function with multiple datasets
Author: peter-sk
Pull request: yes

# What does this PR do?

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@lvwerra
@younesbelkada 



Comments:
@younesbelkada
@lvwerra 
Let me know if you would like a test for this.

@lvwerra
@younesbelkada 
I added a test.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27844). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Makes sense! @peter-sk would you be happy to address:

> let's go ahead and adjust the type for the docstring and tell users how they can pass in multiple eval datasets as a <Tip> in the docstring to evaluate please! 

This would require slightly changing the docstring of the evaluate method in the PR

Finally got around to changing the doc string and adding a tup. Any further requests for changes? Or is this good-to-go?


Upon CI passing, that should be good to go I think
Misc updates to CPU Dockerfiles
Author: ashahba
Pull request: yes

# What does this PR do?
This PR fixes #28082 

Currently `cpu` containers and possibly GPU ones as well under `docker` folder are out of date.The reason being [ubuntu:18.04](https://hub.docker.com/_/ubuntu/tags?page=1&name=18.04) base image was updated over 6 months ago and most likely there won't be anymore updates for that base and that's causing containers to fail during the build.

I also noticed even after updating to newer base images, Torch CPU containers install `GPU` distribution as well which is not only unnecessary but also leads to large final containers.

```
$ docker images | grep transformers-pytorch-cpu
transformers-pytorch-cpu-pr                                    latest                                                   9e377aa5efd9   20 minutes ago   867MB
transformers-pytorch-cpu-main                                  latest                                                   9a88121eeb33   20 hours ago     1.61GB
```

This PR:
- Sets the default base to [ubuntu:22.04](https://hub.docker.com/_/ubuntu/tags?page=1&name=22.04) which should be supported for a couple of years
- Adds appropriate license headers to the files
- Removes unnecessary `Torch GPU` bits from `CPU` containers

Comments:
@dmsuehir feel free to review too.
Thanks.

cc @ydshieh for first review 

Hi @ashahba 

Thank you for the issue and this PR 🤗 .

These files are not used at all in our CI. We should probably them. So we won't merge this fix, sorry.



Thanks @ydshieh
Do you want me to close the PR and submit one that deletes them?

> Thanks @ydshieh Do you want me to close the PR and submit one that deletes them?

Hi @ashahba I will discuss with team members and come back to you :-) Thanks.

Hello again @ashahba Yes, we can remove them, and if you want to open a PR for this, go ahead 🤗 Thanks a lot.

Thanks @ydshieh 
The issue and the PR to delete those Dockerfiles are here:
Issue: https://github.com/huggingface/transformers/issues/28148
PR: https://github.com/huggingface/transformers/pull/28149

Once this one is merged, I'll close the original PR and both issues can be marked as done as well.

Thanks.
Dockerfiles under "docker" folder specially CPU ones fail to build.
Author: ashahba
Pull request: no

### System Info

- tip of `main` hence `v4.36.1`
- Runing Docker version 24.0.7 on Linux Ubuntu 22.04 

### Reproduction

Running:
```
docker build -f docker/transformers-pytorch-cpu/Dockerfile . --tag transformers-pytorch-cpu
```
from tip of `main` branch results in:
```
 => [5/6] COPY . transformers/                                                                                                   1.5s 
 => ERROR [6/6] RUN cd transformers/ &&     python3 -m pip install --no-cache-dir .                                              6.8s 
------                                                                                                                                
 > [6/6] RUN cd transformers/ &&     python3 -m pip install --no-cache-dir .:                                                         
0.939 Processing /workspace/transformers                                                                                              
0.942   Installing build dependencies: started                                                                                        
3.357   Installing build dependencies: finished with status 'done'                                                                    
3.358   Getting requirements to build wheel: started                                                                                  
4.225   Getting requirements to build wheel: finished with status 'done'
4.228   Preparing metadata (pyproject.toml): started
5.367   Preparing metadata (pyproject.toml): finished with status 'done'
6.195 Collecting pyyaml>=5.1
6.237   Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB)
6.365 Collecting requests
6.375   Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
6.462 ERROR: Could not find a version that satisfies the requirement huggingface-hub<1.0,>=0.19.3 (from transformers) (from versions: 0.0.1, 0.0.2, 0.0.3rc1, 0.0.3rc2, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.0.11, 0.0.12, 0.0.13, 0.0.14, 0.0.15, 0.0.16, 0.0.17, 0.0.18, 0.0.19, 0.1.0, 0.1.1, 0.1.2, 0.2.0, 0.2.1, 0.4.0)
6.462 ERROR: No matching distribution found for huggingface-hub<1.0,>=0.19.3
------
Dockerfile:22
--------------------
  21 |     COPY . transformers/
  22 | >>> RUN cd transformers/ && \
  23 | >>>     python3 -m pip install --no-cache-dir .
  24 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c cd transformers/ &&     python3 -m pip install --no-cache-dir ." did not complete successfully: exit code: 1
```


### Expected behavior

Container builds specially on recent Docker versions should be successful.

I have fixes I can supply for the build issues and I'm going to submit the first set in a subsequent PR for your review along with looking into the root cause.

Thanks.

Comments:
cc @ydshieh
[Flash Attention 2] Performance improvement
Author: li-plus
Pull request: no
Labels: name: Good Difficult Issue

name: Flash Attention

### Feature request

The current flash attention 2 integration is sub-optimal in performance because it requires unpadding and padding the activations on **each** layer. For example in llama implementation:

https://github.com/huggingface/transformers/blob/769a9542de4e8b23f0a551738e18760621f463e8/src/transformers/models/llama/modeling_llama.py#L591-L612

These small kernels for unpad/pad keep gpu waiting for cpu, as shown in the visible gaps between kernels in cuda stream.

![image](https://github.com/huggingface/transformers/assets/39846316/f8bfa837-3ddd-447f-a6dd-de4883db63e6)

I'll suggest unpadding the activations at the very beginning (right after word embeddings) and padding it back at the end (maybe before lm_head), and the gap should disappear.


### Motivation

To eliminate performance overhead of flash attention 2.

### Your contribution

I can write the code when I'm not busy. Maybe not now.

Comments:
cc @ArthurZucker @younesbelkada 

Hi @li-plus 
Thanks a lot for the suggestion ! 
@fxmarty tried the approach of pad / unpadd at the beginning of the model forward call here: https://github.com/younesbelkada/transformers/pull/5 but the implementation ended up bloating the modeling code, therefore it has been decided to not move forward for that approach maybe we can revisit this cc @ArthurZucker 

I think this could be revisited given that we have more flexibility with the cache and the attention layer as well, not bandwidth on my side but ready to review a PR so will label it as a good difficult issue!
Fix weights not properly initialized due to shape mismatch
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently, if there is some weight shape mismatched between the model and the checkpoint, and if ignore_mismatched_sizes=True, that/those weight(s) won't get initialized by the model's `_init_weights` method, and could get crazy values like 1e37.

This could make the training gets `nan loss value` from the beginning, (then `Trainer` will change this to `0.0`) and the training won't have any progress (loss always 0.0).

One example is by running `src/transformers/modeling_utils.py` (add `ignore_mismatched_sizes=True`).

We usually set `ignore_mismatched_sizes=True` when we want to perform classification tasks using an existing model but to another task having different number of targets.

This PR aims to fix this issue.


Comments:
I will explain it a bit more tomorrow.

> only the weight matrices that need to be resized are randomly initialized.

That is a very good point, and I did think of it once. Here we have keys in checkpoint and keys in the model's `state_dcit()`.
They are not always identical, as could be seen [here](https://github.com/huggingface/transformers/blob/5aec50ecaf9c1c039cde85881f0586110f845859/src/transformers/modeling_utils.py#L4007-L4021).

Those conversions are only inside `modeling_utils.py` and not exposed at all. So I don't have the way to do the suggested testing.

In most cases, I would say they are identical. I can still implement it, but I am a bit worried we also have edge cases to skip.

WDYT?


--------------------------------------------------

But the change in this PR adds initialization **before any actual weight loading**. So for those weights intended to be loaded,  they are loaded in the same way as before this PR. So if there is any problem, it must be something wrong before this PR.

 

@ydshieh Would it be possible to create a small dummy model with e.g. two linear layers - one we set to explicit values and the other we resize to test this? So we don't have to try to handle complex layer naming in the state dict? 

> @ydshieh Would it be possible to create a small dummy model with e.g. two linear layers - one we set to explicit values and the other we resize to test this? So we don't have to try to handle complex layer naming in the state dict?

Yes, that could be done, similar to [this](https://github.com/huggingface/transformers/blob/main/tests/test_modeling_common.py#L434-L436)

I will do it. But TBH, the above test and the new one to be added are model class independent - and we should probably move them into a new test class rather than being inside `ModelTesterMixin`.

Let me update this PR with your suggestion, and leave the moving of test methods in a follow up PR.

SGTM - thanks! 

A new test `def test_matched_shapes_have_loaded_weights_when_some_mismatched_shapes_exist(self)` is added

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28122). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
remove cpu dockerfiles to fix #28148
Author: evelynmitchell
Pull request: yes

# What does this PR do?

Removes unneeded cpu Dockerfiles.


Fixes ##28148


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
https://github.com/huggingface/transformers/issues/28148

- [x ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? - not needed removed unnecessary item.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
Thank you @evelynmitchell for this PR.

However, the issue author @ashahba had opened a PR #28148 earlier .

Thanks!