noamaanMulla-03/transformers-issues huggingface dataset:
How to install transformers==4.45, two or three days I can install successfully, but today cannot.
Author: HyacinthJingjing
Pull request: no
Labels: name: Installation

name: bug

### System Info

torch2.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

pip install git+https://github.com/huggingface/transformers.git

### Expected behavior

How to install the latest transformers

Comments:
Maybe try using upgrade/reinstall flags, i.e. `pip install --upgrade git+https://github.com/huggingface/transformers.git` or `pip install --force-reinstall git+https://github.com/huggingface/transformers.git`. 

Could you share what error messages/stack trace you get in case it doesn't work? Maybe you don't have git installed on your system?

Thanks @vasqu for the help! @HyacinthJingjing what is the error you're running into?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Revert `accelerate` error caused by `46d09af`
Author: steveepreston
Pull request: yes

# What does this PR do?

Fixes #34176 

Starting from `v4.45.0` Trainer was not able to train on TPU VMs.
so i tested commits between `v4.44.2` and `v4.45.0` one by one, then found this 1 line change commit [46d09af](https://github.com/huggingface/transformers/commit/46d09af4fc62db59cb836099d17a2c6102b39835) causes this error.
You can see full test result on #34176.


## Before submitting
- [x] This PR fixes a bug on code, not a typo
- [x] Read Pull Request Guideline
- [x] Discussed before on github issue.
- [x] No Document update is required. 1 line change.
- [x] Did you write any new necessary tests?


## Reviewers
Trainer: @muellerzr and @SunMarc
Deepspeed: HF Trainer/Accelerate: @muellerzr

Comments:
@SunMarc Thank you too!
@muellerzr Please review this if possible, Thanks!
Fix wrong name for llava onevision and qwen2_vl in tokenization auto
Author: yonigozlan
Pull request: yes

# What does this PR do?
`llava-onevision` -> `llava_onevision`
qwen2_vl is also missing in tokenization_auto
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34177). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Load partial model from pretrained.
Author: johnsutor
Pull request: no
Labels: name: Feature request

### Feature request

Ability to load only portions of models.

### Motivation

I'd like to only use a portion of a model, that has it's own defined class, without having to load in the entire model. Speecifically, I'm attempting to load in the [Seamless Code HiFi Gan](https://huggingface.co/docs/transformers/model_doc/seamless_m4t#transformers.SeamlessM4TCodeHifiGan) without loading in the entire Seamless model. 

### Your contribution

I'd be willing to assist in this effort if the HuggingFace team views this as something they'd like to have in the library. 

Comments:
Hey @johnsutor, we don't support that through `transformers` as of now as it is quite niche usage.

However, the underlying serialization mechanism for models with `transformers` is `safetensors`. With `safetensors`, you can load individual layers individually, without loading the entire checkpoint in memory.

See for example the getting started guide: https://github.com/huggingface/safetensors?tab=readme-ov-file#getting-started

You should be able to load only the layers you want by leveraging this

Awesome, thank you!
phi3 model is not running in cpu
Author: devang-choudhary
Pull request: no

1) microsoft/Phi-3-small-128k-instruct is not running on cpu  (ICELAKE or Graviton) 
the script which i was using : 

![Screenshot 2024-07-26 165001](https://github.com/user-attachments/assets/e0b2ee77-1f03-4d83-9865-78f39f5044f3)


Error I got for Graviton:-
![Screenshot 2024-07-26 162942](https://github.com/user-attachments/assets/040d5bd1-1c10-438d-b885-d62371beecbf)


Error for ICELAKE :-

![Screenshot 2024-07-26 163015](https://github.com/user-attachments/assets/efd77b24-7849-48f9-9bce-28d65f9cd234)


 
2) microsoft/Phi-3-mini-128k-instruct and other mini models are running on cpu but showing warning that You are not running the flash-attention implementation, expect numerical differences.
![Screenshot 2024-07-26 170522](https://github.com/user-attachments/assets/f9a2bcc5-a18b-4c72-94e8-2828e4f02426)

but the implementation of Flash attention for cpu is available in pytorch aten native is there any flag I need to enable in order to use the flash attention for cpu.


Comments:
Probably, similar to (the same error regarding Flash Attention)
 - https://github.com/huggingface/transformers/issues/32201

@ArthurZucker @zucchini-nlp @fxmarty @amyeroberts can you please look into this issue and share your comments on it.

Hey @devang-choudhary ! 

Similar issue was reported at https://github.com/huggingface/transformers/issues/32365. I just digged a bit, Phi3-small models are not natively supported in Transformers because their implementation is slightly different from mini/medium series. So the error you're seeing is related to code in the `hub` and seems like you are unable to `import flash_attn`. FA2 requires CUDA and particular hardware to be installed/run properly (see [here](https://huggingface.co/microsoft/Phi-3-small-128k-instruct#hardware) and [here](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features))

Also, imo we should add native support for Phi3-small but not sure if anyone is already working on it. Nice to see if we can make it work without relying on FA2. cc @ArthurZucker for that

I think it is supported, just their checkpoints are not in the correct format. I don't remember because they are a bit messy and don't seem willing to integrate it natively 😓 

For phi-small it is also the code, because they have some interleaving of two dense/sparse attention and changes the activation function. Sad to hear they don;t want to contribute, should we work on integration ourselves then?

seems to be asked a bit, but I am not entirely sure, we can open an issue for community contribution! 

Agreed, it is too much for the community. I meant we can work on if it's asked a lot, but we'll will be slow

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
PreTrainedModel.from_pretrained(path, from_flax=True) fails for sharded Flax checkpoints
Author: KathyHaem
Pull request: no
Labels: name: Flax

### System Info

Python==3.11
transformers==4.41.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Background: Some code I'm running is trying to load a flax model into a standard PreTrained(X)Model in order to save it to pytorch format. This works fine as long as the flax weights are in a single file.

When there is more than one flax weights file, this fails with `OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory <dir>.`

### Expected behavior

The fallback to looking at the weights index file is implemented in `modeling_flax_utils` for flax, but as far as I can tell all the other types of weights also have this handled within `modeling_utils` , so the behaviour in the general PreTrainedModel should match for flax.

As a workaround will it also work to save FlaxPretrainedModel.save_pretrained() as pytorch? @sanchit-gandhi 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Device Mismatch Error when Exporting Hugging Face's BERTModel to TorchScript
Author: lsrock1
Pull request: no
Labels: name: Compilation

### System Info

- `transformers` version: 4.37.1
- Platform: Linux-4.18.0-477.27.1.el8_8.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU): 2.1.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. export in cpu
```
from transformers import BertModel, BertTokenizer
import torch

model = BertModel.from_pretrained('bert-base-multilingual-uncased', torchscript=True)
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-uncased')

input_test = torch.ones([1, 20]).long()

script_model = torch.jit.trace(model, [input_test])
torch.jit.save(script_model, 'test.pt')
```

2. load and inference in gpu
```
import torch

input_test = torch.ones([1, 20]).long()
script_model = torch.jit.load('test.pt', map_location='cuda')
script_model(input_test.cuda())
```

3. error
```
class BertSelfAttention(Module):
  File "code/__torch__/transformers/models/bert/modeling_bert.py", line 175, in forward
    attention_scores = torch.matmul(query_layer, torch.transpose(key_layer, -1, -2))
    attention_scores0 = torch.div(attention_scores, CONSTANTS.c2)
    input = torch.add(attention_scores0, attention_mask)
            ~~~~~~~~~ <--- HERE
    input2 = torch.softmax(input, -1)
    context_layer = torch.matmul((dropout).forward(input2, ), value_layer)
....
....
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
```

The device variable creating tensors remains fixed during tracing, necessitating dynamic adjustment for flexibility.
https://github.com/huggingface/transformers/blob/2a9b1f80c45cab19b542bc7cc004937d39d6f6fb/src/transformers/models/bert/modeling_bert.py#L968-L974


### Expected behavior

Ensuring traced models operate on different devices(cpu -> gpu, gpu -> cpu)






Comments:
cc @fxmarty maybe optimum supports it?

@ArthurZucker can I send a PR?

@lsrock1 for sure

Gentle ping @fxmarty 

cc @IlyasMoutawwakil 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Any attempt to export any model to onnx returns an ATOL value of nan.
Author: Jcwscience
Pull request: no

`ValueError: Outputs values doesn't match between reference model and ONNX exported model: Got max absolute difference of: nan`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I got the same error, could anyone help with this issue?

transformers[onnx] == 4.25.1
onnx == 1.13.0

trying to convert huggingface bert-based model
[Request] Guidance of modifying the structure of a model load by trust_remote_code?
Author: URRealHero
Pull request: no

I found it easier to directly modify the model structure by override the method if the model is set in the transformers package. However, when I load by AutoModelForCausalLM.from_pretrained(.. trust_remote_code=True), the modeling file is now under the hub dir, instead of the transformers package, how can I override the classes or methods this time without modifying the source code but rewrite to override?

Comments:
Update `trainer._get_eval_sampler()` to support `group_by_length` arg
Author: larin92
Pull request: yes

Trainer didn't support grouping batches by length for evaluation, which made evaluation slow with `eval_batch_size`>1.

Updated `trainer._get_eval_sampler()` method was based off of `trainer._get_train_sampler()`.

Comments:
Thanks @larin92 for the PR! Pinging @muellerzr and @SunMarc for review :)

checked failed test, it's not related to this PR

@SunMarc , rebased on top of 0f49deacbff3e57cde45222842c0db6375e4fa43 (as latest commit with all checks passed)
Add Log-Softmax Temperature Option to transformers.Seq2SeqTrainingArguments for CE Loss
Author: oswaldoludwig
Pull request: no
Labels: name: Feature request

### Feature request

I would like to propose an enhancement to the transformers.Seq2SeqTrainingArguments class. Specifically, I suggest including an option to adjust the log-softmax temperature for the cross-entropy (CE) loss function. 

### Motivation

This feature would be particularly beneficial for training models with large vocabularies, as highlighted [here](https://medium.com/@oswaldoludwig/unveiling-a-pitfall-in-cross-entropy-loss-for-large-vocabularies-ef388d846bc1)

### Your contribution

For Whisper fine-tuning, I worked around this issue by using the following class:

```
class CustomSeq2SeqTrainer(Seq2SeqTrainer):
    def __init__(self, *args, temp=1.0, **kwargs):
        super().__init__(*args, **kwargs)
        self.temp = temp

    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.get("labels")
        # Forward pass
        outputs = model(**inputs)
        logits = outputs.get("logits")

        # Apply temperature scaling
        logits = logits / self.temp

        # Compute loss
        loss_fct = torch.nn.CrossEntropyLoss()
        loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1))

        return (loss, outputs) if return_outputs else loss

```

Comments:
Hey @oswaldoludwig, the trainer implementation is made to be extensible as it cannot ever satisfy every request; the way you did it is great!

If we see significant interest in this issue we can see about upstreaming it; if a reader is interested in having this in transformers as well, please comment here and we'll consider it.
Fix warning message for fp32_cpu_offloading in bitsandbytes configs
Author: amosyou
Pull request: yes

# What does this PR do?

The bitsandbytes config was introduced in #21579, with the `llm_int8_enable_fp32_cpu_offload` argument to enable cpu offloading. However, the environment validation function for bitsandbytes configs has a warning message that suggests using the argument `load_in_8bit_fp32_cpu_offload`, which is incorrect. This PR fixes this inconsistency.

cc @SunMarc

Comments:
ready for another review

@SunMarc bump in case you forgot! should be ok to merge

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34079). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Ping team members for new failed tests in daily CI
Author: ydshieh
Pull request: yes

# What does this PR do?

Use `git bisect run` to find the commit for a failed test, and ping team member using Slack's keyword mention (`GH_ydshieh` for example)

The scope is limitied

- Currently only focus on the `single-gpu` tests
- During each `git bisect run`, we only check a single (failed) test like 
    > tests/models/vit/test_modeling_vit.py::ViTModelTest::test_foo`
   
   (but some failures only occur in a test suite of that model being run)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34171). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

You mean the (new) GitHub Action yml file, right?

I have been quite careful recently and this new file has no (actual) new stuff than other existing workflow files :-)
Generate: move `logits` to same device as `input_ids`
Author: gante
Pull request: yes

# What does this PR do?

In `generate`, we create many auxiliary tensors for internal operations. Prior to this commit, the assumption was that the model's output (`logits`) and these auxiliary tensors were on the same device. In practice, this is the same as assuming that `input_ids` (or other main input, depending on the model) would be on the same device as `lm_head`.

This assumption doesn't always hold, even if the user follows our suggested examples. For instance, if we do `model_inputs = model_inputs.to(model.device)`, as we suggest in many examples, we move `model_inputs` to the same device as in the first set of parameters in `model.parameters()`, which is far from guaranteed to be the same device as `lm_head`. ([PreTrainedModel.device source code](https://github.com/huggingface/transformers/blob/24b82f3cd56d5eeb26e649207aac3ce8a7d75bdc/src/transformers/modeling_utils.py#L1072), [.parameters() docs](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.parameters))

To make the assumption hold, this PR moves the `logits` returned by the model to the same device as `input_ids`, which should fix most (if not all) of device issues in multi-device settings.

Example of command fixed by this PR: `CUDA_VISIBLE_DEVICES=0,1 RUN_SLOW=1 python -m pytest tests/quantization/bnb/test_mixed_int8.py -k test_generate_quality_config -vv` ✅ 

_________________________________

### ⚠️ caveats

The following script now runs. 

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2", device_map="auto")

inputs = tokenizer(["The quick brown"], return_tensors="pt")
gen_out = model.generate(**inputs, do_sample=False)
```

Note that `input_ids` is on the CPU, so some internal ops in `generate` will run on the CPU (while the forward pass happens on GPU). We throw a [loud warning](https://github.com/huggingface/transformers/blob/24b82f3cd56d5eeb26e649207aac3ce8a7d75bdc/src/transformers/generation/utils.py#L2090), but this may still be a source of issues (`generate` running slower than expected because there are CPU ops and device transfers)

Comments:
fix(Wav2Vec2ForCTC): torch export
Author: chrsmcgrr
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fixes #34022 by implementing the masking of the hidden states using an elementwise multiplication rather than indexing with assignment.

The torch.export functionality seems to mark the tensor as frozen even though the update is legal.

This change is a workaround for now to allow the export of the model as a FxGraph. Further investigation is required to find the real solution in pytorch.

Tagging:
@ylacombe, @eustlb

Please let me know if someone else is more appropriate to review this PR. 


Comments:
Seems okey to me, compile stuff usually complain on in-place modification to tensor

Thanks @zucchini-nlp, requesting a core maintainer review then !

@ylacombe @zucchini-nlp these errors seem unrelated to the slow tests. Is this expected?

LGTM 👍 

(I've rerun CI in case it was a transient error)